Robots.txt: What It Controls and What It Does Not

What robots.txt controls

The file lives at the origin root, such as https://www.example.com/robots.txt, and applies only to that protocol and hostname. A crawler identifies a matching User-agent group, then evaluates its path rules according to the crawler's documented implementation.

Disallow asks a participating crawler not to request matching paths. Allow can create a narrower exception where supported. A Sitemap line can point to an absolute sitemap URL. Comments begin with #.

What robots.txt does not control

• It is not authentication and must not protect confidential information.
• It does not reliably remove an already known URL from search results.
• It does not replace a page-level or HTTP noindex directive.
• It does not set the canonical URL or redirect duplicate variants.
• It does not force a crawler to request, index, rank, retrieve, or cite an allowed page.
• It does not automatically configure a CDN or web application firewall.

If a crawler is blocked from a page, it generally cannot read a noindex directive on that page. For intended removal, allow the relevant crawler to access the directive and use authentication when content is private.

Safe syntax examples

This simple example allows crawling and identifies a sitemap:

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

This example blocks a private-looking administrative path from participating crawlers but does not make that path secure:

User-agent: *
Disallow: /administrator/
Disallow: /internal-search/

Sitemap: https://www.example.com/sitemap.xml

Replace example paths only after confirming the real CMS and route behavior. A broad rule such as Disallow: / blocks the entire origin for the matching agent. On a production site, that line deserves immediate investigation.

Do not block essential rendering assets

Search systems may need CSS, JavaScript, images, fonts, and API responses to understand the page as a user sees it. Historic rules that block entire template, component, plugin, or asset directories can hide essential output. Test representative pages after changing a rule and compare initial HTML with the rendered DOM.

Missing is different from invalid or unintended

A site without a robots.txt file generally has no file-based crawl restrictions for participating agents. That can be acceptable for a fully public site, although a valid file is useful for an explicit policy and sitemap reference. The dangerous conditions are an unintended broad block, a file that serves an error or HTML page, conflicting versions by hostname, invalid encoding, or a policy that does not match the delivery infrastructure.

Do not publish a generic file only to clear a warning. Inventory the paths and agents first, preserve necessary administrative security through authentication, and verify that the new response does not expose or block routes unexpectedly.

When multiple hostnames are public, check each origin separately; one correct file does not govern another hostname, protocol, or port. Keep the production and protected-staging policies deliberately separate, current, tested, and documented.

Confirm the finding

1. Request the exact root robots URL for every public protocol and hostname.
2. Record the final status, redirects, content type, body, and cache behavior.
3. Identify the exact user agent and path being tested.
4. Evaluate the matching group and the most specific applicable rule using that crawler's official syntax guidance.
5. Confirm the sitemap URL resolves and uses the preferred hostname.
6. Check CDN, firewall, and bot-manager logs for blocks that robots.txt does not reveal.

Fix the policy

1. Define intent: decide which public content each documented crawler may access.
2. Remove accidental sitewide blocks: correct broad rules only after confirming they are unintended.
3. Use narrow paths: block specific duplicate, utility, or administrative areas instead of large CMS directories.
4. Separate agent groups: use exact platform tokens when search and training crawlers require different policies.
5. Preserve resources: allow files necessary to render important public pages.
6. Publish one valid root file: avoid conflicting application, CDN, and server versions.
7. Test externally: fetch the public file and representative URLs after caches update.

Joomla and WordPress notes

Joomla

Review the root file, server rules, hosting controls, and any extension that edits robots policy. Do not block /media/, template assets, component resources, or other paths needed to render public pages without testing. Joomla's article robots metadata is separate from robots.txt.

WordPress

WordPress can generate a virtual robots response when no physical file exists. SEO plugins, security plugins, hosting tools, CDN rules, and a physical file may compete. Confirm the public response rather than assuming the dashboard editor is authoritative.

Pass and fail criteria

PASS
- /robots.txt returns one intended, readable policy
- Important public pages and required assets are allowed
- Agent-specific groups use current documented tokens
- Sitemap line resolves to the preferred sitemap
- CDN and firewall behavior matches the policy

FAIL
- Unintended Disallow: /
- Preferred pages or rendering assets are blocked
- File redirects, errors, serves HTML, or conflicts by hostname
- Policy relies on robots.txt as privacy or noindex
- Bot management silently blocks an intentionally allowed crawler

Verify and rescan

Fetch the final file without authentication, test the intended agent and paths, request representative public pages, and review infrastructure logs where available. Rescan the site to confirm the observable robots result changed. Last platform-policy review: 30 July 2026.

Sources

• Google Search Central: Introduction to robots.txt
• Google Search Central: noindex guidance
• OpenAI crawler documentation