AI Crawler Access: OAI-SearchBot, PerplexityBot, and Robots.txt

Separate search crawlers from training crawlers

An “AI bot” label is too broad for a sound policy. A company may operate one agent that supports search results, another that may be used to improve models, and another involved in a user-requested action. Site owners should identify the exact token, purpose, official documentation, and desired permission.

Names and purposes can change. Treat the official platform pages as the current source of truth, not an old third-party bot list.

Choose and document the policy

Decide whether public website content may be used for search retrieval, potential model improvement, and user-initiated access. The choices can differ. Record the business owner, decision date, exact token, allowed paths, blocked paths, reason, official source, and next review date.

Robots.txt is public and voluntary. It should not expose a secret path list or replace authentication. Private reports, account areas, administrative interfaces, and protected staging need real access controls.

A business may reasonably allow search retrieval while declining potential model training, allow both, or block both. The article does not choose that policy for the site owner. It provides the controls and verification steps needed to implement an informed decision.

Revisit the decision when the platform changes its documented purpose, the site's licensing policy changes, or logs show unintended infrastructure behavior. Keep the last-reviewed date, accountable owner, rationale, and official source with the policy record.

Illustrative robots.txt examples

This example permits OpenAI search crawling but declines GPTBot:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This example permits Perplexity search crawling:

User-agent: PerplexityBot
Allow: /

These snippets are examples, not a recommended universal policy. Check the complete file, existing wildcard groups, path rules, hosting stack, and current official tokens before deploying them.

Robots permission is not enough

A CDN, firewall, security plugin, rate limiter, geographic rule, origin allowlist, or bot-management service may reject the request after robots.txt allows it. Review response codes and logs. A 403, 429, challenge page, or intermittent 5xx can prevent access even when the policy file looks correct.

A self-declared user-agent string can be spoofed. When a platform publishes official IP ranges or verification guidance, use the current file and a secure infrastructure process. OpenAI and Perplexity publish machine-readable IP information. Do not hard-code copied ranges into permanent documentation without an update method.

Confirm the finding

1. Fetch the root robots.txt for the exact protocol and hostname.
2. Evaluate the matching group for the exact documented agent and requested path.
3. Check broader wildcard rules and more specific path rules for conflicts.
4. Request representative public pages and assets through the delivery stack.
5. Review CDN, firewall, and origin logs for 403, 429, challenge, timeout, and 5xx responses.
6. Verify infrastructure identity with the platform's current official method where needed.
7. Confirm the page itself is public, successful, canonical, useful, and internally linked.

Fix access without overexposing the site

1. Approve the purpose: document search, training, and user-triggered decisions separately.
2. Use exact tokens: add or adjust only the documented agent groups.
3. Scope paths carefully: allow public content and keep private or utility areas protected by proper controls.
4. Preserve rendering assets: ensure permitted agents can request resources needed to interpret the page.
5. Align infrastructure: update bot-management or firewall behavior using official verification data.
6. Test externally: confirm robots output and public responses after cache propagation.
7. Schedule review: recheck official documentation and access logs regularly.

Joomla and WordPress notes

Joomla

Check the physical root file, hosting controls, security extensions, CDN, and server rules. Do not change global Joomla SEO settings to solve an agent-specific access problem. Confirm public component and article routes expose the intended content and assets.

WordPress

Determine whether robots.txt is physical or virtual and which plugin, host, or CDN owns it. Security and caching plugins may override the intended response or challenge bots. Test the public file and route after clearing caches.

Pass and fail criteria

PASS
- Exact current crawler tokens have a documented policy
- Robots rules match the approved search and training decisions
- Public pages and essential resources return usable responses
- CDN and firewall behavior matches the robots policy
- Official infrastructure verification is used where appropriate
- No promise links permission to guaranteed citation

FAIL
- One vague “AI bots” rule substitutes for purpose-specific decisions
- Allowed crawler receives 403, 429, challenge, timeout, or repeated 5xx
- Copied or obsolete token/IP information controls access
- Private content relies on robots.txt for security
- Provider claims access guarantees inclusion, ranking, or citation

Verify and rescan

Fetch the final robots policy, test representative pages and resources, review infrastructure logs, and rescan the public site. Record the platform documentation date. Access can be verified; citation and recommendation remain platform decisions.

Sources

• OpenAI crawler documentation
• OpenAI publisher and developer FAQ
• Perplexity crawler documentation
• Google robots.txt guidance