Short answer: decide per job, not per company. Allow the bots that put you in AI answers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, and above all Googlebot and Bingbot (those feed Google’s AI Overviews, Copilot and, indirectly, ChatGPT). Blocking a training bot (GPTBot, ClaudeBot, Google-Extended) is a separate and legitimate choice that does not remove you from AI search. And remember robots.txt is a polite request: user-triggered fetchers and scrapers (bots that harvest your site’s content) like Bytespider may ignore it, so for those you need server-level rules.
We run managed hosting for client sites and read these crawlers in our own access logs, so this is the cheat sheet we actually use — built on the platforms’ own documentation, linked throughout.
First, the only thing that matters: what is the bot for
Every AI crawler does one of four jobs. The job — not the brand — decides whether you should let it in.
- Search / answer indexing. Builds the index an AI answer is drawn from. Block it and you vanish from that engine’s answers. Allow these.
- Model training. Collects content to train future models. This is a content/IP decision, not an SEO one — blocking it does not affect whether you appear in AI search.
- User-triggered fetch. Grabs a single page because a user asked the assistant about it. These often ignore
robots.txt, because they act on a person’s behalf. - Ads / previews / other. Validates ad landing pages, builds link previews, and so on.
So “should I block AI bots?” has no single answer. You almost always want the search bots in, and the training bots are your call.
The cheat sheet
| Crawler | Owner (country) | What it’s for | Obeys robots.txt? | Our recommendation |
|---|---|---|---|---|
Googlebot | Google (US) | Search index — also feeds AI Overviews & AI Mode | Yes | Allow (critical) |
Bingbot | Microsoft (US) | Search index — feeds Copilot, and indirectly ChatGPT | Yes | Allow (critical) |
OAI-SearchBot | OpenAI (US) | ChatGPT search index | Yes | Allow |
Claude-SearchBot | Anthropic (US) | Claude’s search results | Yes | Allow |
Claude-User | Anthropic (US) | Fetches a page when a Claude user asks | Yes | Allow (don’t block) |
PerplexityBot | Perplexity (US) | Builds Perplexity’s answer index | Yes | Allow |
Meta-WebIndexer | Meta (US) | Meta AI search results | Yes | Allow for Meta AI visibility |
GPTBot | OpenAI (US) | Trains OpenAI models | Yes | Your call — block to opt out (no search impact) |
ClaudeBot | Anthropic (US) | Trains Anthropic models | Yes (+ Crawl-delay) | Your call — block to opt out |
Google-Extended | Google (US) | Control token for Gemini / Vertex AI training | Yes (it’s a control token) | Block to opt out of training — does not affect Search ranking |
Amazonbot | Amazon (US) | Improves Amazon products; may train Amazon AI | Yes | Your call |
Meta-ExternalAgent | Meta (US) | Trains Meta models / indexes content | Yes | Your call — block to opt out |
ChatGPT-User | OpenAI (US) | Fetches a page when a ChatGPT user asks | ”may not apply” | Can’t rely on robots.txt — block at server if needed |
Meta-ExternalFetcher | Meta (US) | Fetches links at a user’s request | May bypass | Server-level if needed |
facebookexternalhit | Meta (US) | Builds link previews on Meta apps | May bypass | Usually keep (link previews) |
Perplexity-User | Perplexity (US) | Real-time fetch on a user’s question | Generally ignores (user-triggered) | Can’t rely on robots.txt — block at server if needed |
GoogleOther | Google (US) | Generic Google crawl (R&D), not Search ranking | Yes | Neutral — allow |
OAI-AdsBot | OpenAI (US) | Validates submitted ad landing pages | Only visits ad pages | Leave alone |
Bytespider | ByteDance (China) | Bulk scraping, reportedly for training | No clear official commitment | Block at server if it hurts load |
Sources: OpenAI crawlers, Anthropic crawlers, Google crawlers, Amazonbot, Meta web crawlers, Perplexity crawlers.
Countries and firewalls. Almost every AI crawler is run from the US — Google, Microsoft, OpenAI, Anthropic, Amazon, Meta, Perplexity. The one major non-US crawler is Bytespider (ByteDance, China), and it’s frequently the most aggressive of the bunch. A caveat for ipset / GeoIP blocking: crawlers run on cloud IPs that don’t always match the company’s home country — so block by each provider’s official published IP ranges (Google, OpenAI, Perplexity and others publish them) and confirm with reverse DNS, not by country alone, or you’ll cut a search bot you wanted along with the whole country.
A robots.txt you can copy
If your robots.txt has no site-wide Disallow, the search bots are already allowed — the point is simply don’t add a Disallow for them. The block below is the part most people actually want: opting out of AI training while staying fully visible in AI search.
# Keep the AI search/answer bots allowed (do NOT disallow these):
# Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot,
# Claude-User, PerplexityBot
# OPTIONAL — opt out of AI *training* only.
# This does NOT remove you from AI search or AI answers.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Bytespider
Disallow: /Google-Extended is worth calling out: per Google it controls whether your content trains Gemini and Vertex AI models, and it does not affect your inclusion or ranking in Google Search — so AI Overviews still draw on you through the normal index.
robots.txt is a request, not a fence
Two groups of crawlers will not be stopped by robots.txt:
- User-triggered fetchers. When someone asks ChatGPT or Perplexity about your page, the assistant fetches it on the user’s behalf. OpenAI states that for
ChatGPT-User, “robots.txt rules may not apply,” and Perplexity saysPerplexity-User“generally ignores robots.txt rules.” (Claude-Useris the polite exception — Anthropic says it respectsrobots.txt.) If you genuinely need to stop these, do it at the server, not inrobots.txt. - Scrapers that just ignore it.
Bytespider(ByteDance) has no clear public commitment to the robots exclusion standard and is widely reported to crawl aggressively. If it’s eating bandwidth, block it at the web-server or firewall level — on nginx aUser-Agentmatch returning403, or a Fail2ban rule.
This is exactly where managed hosting earns its keep: the visibility bots stay allowed in robots.txt, and the abusive ones get cut at the edge so they don’t slow the site down.
The expensive mistake: blocking the bots you want
In practice, a site hides itself from AI search by accident far more often than it gets hammered by scrapers. Here’s how it usually happens:
- a blanket
Disallow: /left over from a staging site; - a WAF (a protective filter in front of your site) or “bot fight” mode that blocks unknown user-agents (the label a bot identifies itself by) — including
OAI-SearchBotandPerplexityBot; - blocking
GPTBotand assuming that also covers ChatGPT search (it doesn’t — that’sOAI-SearchBot).
If you’re not sure which bucket you’re in, start with our companion piece on the myths of AI search and what Google and OpenAI actually say — it covers the indexing and snippet basics that decide whether you can be cited at all.
FAQ
If I block GPTBot, do I disappear from ChatGPT? No. GPTBot is training-only. ChatGPT search runs on OAI-SearchBot. Blocking GPTBot just opts you out of training.
Does Google-Extended change my Google ranking? No. Per Google it only controls use of your content for Gemini and Vertex AI training; it does not affect inclusion or ranking in Search, and AI Overviews use the normal index.
Will robots.txt stop Bytespider? Maybe not. It has no clear official commitment to honor it and is reported to ignore it — block it at nginx, your firewall, or a WAF if it’s hurting load.
Can I block ChatGPT-User or Claude-User? Claude-User respects robots.txt; ChatGPT-User may not. To reliably stop user-triggered fetches, block at the server.
Should I block all AI training bots? That’s a content and IP decision, not an SEO one. Blocking training bots does not hurt your AI-search visibility — so only block them if you don’t want your content used to train models.
We set this up for clients as part of managed hosting and our SEO & GEO work — visibility bots allowed, abusive ones cut at the server, all based on the platform docs above. Want a quick check of what’s crawling your site and whether you’re blocking the wrong bots? Message us.