Check AI crawler access

Most sites have never decided whether AI assistants may read them; the answer is whatever the robots.txt happened to say. Some crawlers train models, some power search answers that cite the site, and blocking the second kind by accident removes the site from a growing share of discovery. This playbook makes the current state explicit.

Sequel supplies the list of pages that matter. Reading robots.txt and the pages uses your own browsing.

Steps

1. Get the pages that matter. google_search_console.query with dimensions: ["page"], 90 days ending 3 days ago, rowLimit: 200. These are the pages whose access is worth checking.

2. Fetch and parse robots.txt. Read /robots.txt. For each user agent below, work out whether the root and the top page paths are allowed or disallowed, remembering that a group for * applies to any agent without its own group, and the most specific matching rule wins.

   • Answer and search crawlers, whose results cite the site: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot, Googlebot, DuckAssistBot.
   • Training crawlers, which do not send traffic: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent.

3. Check the pages themselves. Fetch five to ten of the top pages and record meta robots and X-Robots-Tag values, looking for noindex, nosnippet, max-snippet, noai and noimageai. Note whether the main content is present in the raw HTML; most AI crawlers do not execute JavaScript, so a page that renders client-side is effectively empty to them.

4. Explain the consequences. Say specifically what each block does. Blocking GPTBot keeps the site out of model training and has no effect on ChatGPT search; blocking OAI-SearchBot removes it from ChatGPT's search answers. Google-Extended controls Gemini training only; AI Overviews use Googlebot, and the only controls are nosnippet and max-snippet, which also affect the normal result. Apply the same reasoning to each agent.

5. Report. A matrix of agent against access (allowed, disallowed, partially disallowed with the paths), the meta robots findings per page, the JavaScript-rendering observation, and a recommendation in two parts: answer engines that cite the site are usually worth allowing; training crawlers are a policy decision for the user, listed with what each one feeds. Flag any block that looks accidental, such as a Disallow: / under * with no exceptions.

Gotchas

• robots.txt is a request, not a wall. Well-behaved crawlers honour it; others do not. It is still the right place to state the policy.
• The edge can block what robots.txt allows. A CDN or WAF bot rule can return 403 to these agents with nothing in robots.txt. If you can fetch with a custom user agent, test the top pages as GPTBot and PerplexityBot; if you cannot, say the edge was not tested.
• Agent names change. The list above is current at the time of writing. Check each vendor's documentation with your browsing if a name looks stale.
• Blocking is not neutral. A site invisible to answer engines loses citations to competitors that allowed them. Present that trade-off, not a recommendation to block by default.
• Continue with ai-referral-traffic to see what the current policy is producing, and llms-txt-audit-and-draft to give allowed crawlers a curated map.