A lot of sites quietly slammed the door on AI crawlers in late 2023 and 2024. It felt principled \u2014 \u201cdon\u2019t train on our content without paying us\u201d \u2014 and the one-line robots.txt edits were trivial. The bill came due the moment AI search took off. Pages that locked out GPTBot to protect themselves from training also vanished from the answer surfaces that now decide whether a brand gets cited. Most operators still haven\u2019t separated those two decisions, and it\u2019s costing them visibility in the only channel that\u2019s growing.<\/p>\n
The AI labs run two distinct classes of crawler, and the names sit close enough that people conflate them. On one side: the training crawlers \u2014 GPTBot (OpenAI), ClaudeBot (Anthropic, when used for training), Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, FacebookBot, anthropic-ai. These are the ones the publishing world reacted to, because their job is to vacuum content into model weights. On the other side: the answer-fetchers \u2014 OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User (Perplexity), ClaudeBot when invoked for live browsing, and \u2014 quietly the most important of all \u2014 regular Googlebot, which is what feeds Google AI Overviews.<\/p>\n
The training bots and the answer bots are different user agents. Blocking GPTBot does not opt you out of OpenAI\u2019s search products, and blocking Google-Extended does not opt you out of AI Overviews. But \u2014 and this is where a lot of sites got cooked \u2014 most \u201cblock all AI scrapers\u201d presets at the CDN layer (Cloudflare\u2019s super-bot toggle, several WAF rule packs, half the WordPress security plugins) lump training and search agents into one bucket. You think you opted out of training. You actually opted out of being cited.<\/p>\n
The stakes are now lopsided. AI Overviews queries that used to send organic traffic out are collapsing classic CTR by roughly 58\u201361% on informational intents, but the brands quoted inside the AI summary are seeing about a 35% lift in clicks on the cited surface. The pages that get cited are the pages the answer bots can fetch \u2014 full stop. If your robots.txt or your edge firewall says \u201cno\u201d to OAI-SearchBot, you are not in the running.<\/p>\n
Stop treating \u201cAI access\u201d as one toggle. Decide twice.<\/p>\n
Decision one \u2014 training. Are you OK with your content being used to train the next foundation model? Reasonable people land on either side. Block GPTBot, Google-Extended, anthropic-ai, CCBot, Bytespider if you want out of training. That\u2019s a values\/business-model call, not an SEO call.<\/p>\n
Decision two \u2014 answer retrieval. Do you want to show up when someone asks ChatGPT, Perplexity, Claude, or Google AI Overviews about your category? If yes \u2014 and for almost every B2B and lead-gen brand the answer is yes \u2014 you must allow the answer-side user agents. Allow OAI-SearchBot and ChatGPT-User. Allow PerplexityBot and Perplexity-User. Leave regular Googlebot alone (it is your AIO pipeline). Allow ClaudeBot for live retrieval if you want Claude\u2019s web mode to find you.<\/p>\n
1.<\/strong> Pull your current robots.txt and read it with these two columns in mind: which user agents are blocked, and which class each one belongs to. If anything on the answer-fetch list is denied, that\u2019s your bleeding wound.<\/p>\n 2.<\/strong> Audit the edge. Cloudflare\u2019s \u201cblock AI bots\u201d managed rule, Sucuri\u2019s AI scraper preset, similar WAF toggles \u2014 log into the dashboard and look at the actual UA list. The defaults change without notice and frequently bundle search bots into the training block.<\/p>\n 3.<\/strong> Check your server logs for the past 30 days, filtered to OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot. Zero requests is a red flag \u2014 either you blocked them, your CDN did, or your robots.txt is wrong. Don\u2019t trust the documentation; trust the logs.<\/p>\n