A lot of sites quietly slammed the door on AI crawlers in late 2023 and 2024. It felt principled — “don’t train on our content without paying us” — and the one-line robots.txt edits were trivial. The bill came due the moment AI search took off. Pages that locked out GPTBot to protect themselves from training also vanished from the answer surfaces that now decide whether a brand gets cited. Most operators still haven’t separated those two decisions, and it’s costing them visibility in the only channel that’s growing.
The mechanic: training bots and answer bots share a parking lot
The AI labs run two distinct classes of crawler, and the names sit close enough that people conflate them. On one side: the training crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic, when used for training), Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, FacebookBot, anthropic-ai. These are the ones the publishing world reacted to, because their job is to vacuum content into model weights. On the other side: the answer-fetchers — OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User (Perplexity), ClaudeBot when invoked for live browsing, and — quietly the most important of all — regular Googlebot, which is what feeds Google AI Overviews.
The training bots and the answer bots are different user agents. Blocking GPTBot does not opt you out of OpenAI’s search products, and blocking Google-Extended does not opt you out of AI Overviews. But — and this is where a lot of sites got cooked — most “block all AI scrapers” presets at the CDN layer (Cloudflare’s super-bot toggle, several WAF rule packs, half the WordPress security plugins) lump training and search agents into one bucket. You think you opted out of training. You actually opted out of being cited.
The stakes are now lopsided. AI Overviews queries that used to send organic traffic out are collapsing classic CTR by roughly 58–61% on informational intents, but the brands quoted inside the AI summary are seeing about a 35% lift in clicks on the cited surface. The pages that get cited are the pages the answer bots can fetch — full stop. If your robots.txt or your edge firewall says “no” to OAI-SearchBot, you are not in the running.
The two questions, decided separately
Stop treating “AI access” as one toggle. Decide twice.
Decision one — training. Are you OK with your content being used to train the next foundation model? Reasonable people land on either side. Block GPTBot, Google-Extended, anthropic-ai, CCBot, Bytespider if you want out of training. That’s a values/business-model call, not an SEO call.
Decision two — answer retrieval. Do you want to show up when someone asks ChatGPT, Perplexity, Claude, or Google AI Overviews about your category? If yes — and for almost every B2B and lead-gen brand the answer is yes — you must allow the answer-side user agents. Allow OAI-SearchBot and ChatGPT-User. Allow PerplexityBot and Perplexity-User. Leave regular Googlebot alone (it is your AIO pipeline). Allow ClaudeBot for live retrieval if you want Claude’s web mode to find you.
What to do this week
1. Pull your current robots.txt and read it with these two columns in mind: which user agents are blocked, and which class each one belongs to. If anything on the answer-fetch list is denied, that’s your bleeding wound.
2. Audit the edge. Cloudflare’s “block AI bots” managed rule, Sucuri’s AI scraper preset, similar WAF toggles — log into the dashboard and look at the actual UA list. The defaults change without notice and frequently bundle search bots into the training block.
3. Check your server logs for the past 30 days, filtered to OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot. Zero requests is a red flag — either you blocked them, your CDN did, or your robots.txt is wrong. Don’t trust the documentation; trust the logs.
4. If you want explicit training opt-out language without losing citations, put the training bots in robots.txt with `Disallow: /`, leave the answer bots with `Allow: /`, and document the decision next to the file so the next person doesn’t “simplify” it.
Paris Roussos has been doing SEO since 1996 (co-founded a Forbes Best of the Web–winning site back in the day) and now runs a white-label AI SEO practice for agencies and brands — flat-rate, $500–$1,500/mo per client. If your top-of-funnel traffic is leaking into ChatGPT and Perplexity and you want it back, email parisroussos@gmail.com.
The right answer isn’t “block everything” or “allow everything” — it’s knowing exactly which bot you just told to leave.