Robots.txt for AI Crawlers: GPTBot, ClaudeBot, PerplexityBot, Google-Extended

By Cory Maki · Sep 10, 2026

Robots.txt is the single most consequential file most SaaS founders never look at — and the one place a five-line mistake can quietly delete your brand from ChatGPT, Claude, and Perplexity answers. The problem is that the advice circulating online is mostly a stale copy-paste list that blocks every AI user-agent by default. That list was written for a training-data debate, not for an era where the same bots decide whether AI names you to a buyer.

This is a tactical guide to the syntax itself: the exact user-agent tokens for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, which of them actually respect the file, which ignore it, and the allow/disallow rules that match a GEO strategy instead of fighting it. If you want the broader picture of how AI engines choose sources, that lives in the GEO guide — this post stays narrow on the file.

Robots.txt is advisory, not enforceable — treat it as a policy signal, not a wall

The first thing to internalize: robots.txt is a request, not a firewall. Robots.txt is a standardized set of instructions for websites to indicate to visiting web crawlers what they can or can't access. While it can't enforce the rules and is built on trust, most legitimate tech companies respect this sort of "Code of Conduct" sign.

That distinction matters because two of the four crawlers in this post behave very differently. OpenAI, Anthropic, and Google document their tokens and honor them. Perplexity's compliance is contested. So the correct mental model is: use robots.txt to set your policy for compliant bots, and rely on a WAF or CDN rule for anything you genuinely need to block. Robots.txt is a voluntary protocol that ethical AI companies respect, but it cannot guarantee protection. Some systems may reach your content by other means. For sensitive content, add authentication and terms of service.

One more framing point before the syntax. Robots.txt is a blocking layer — it controls access. It is not a discovery layer. If you want AI crawlers to find and prioritize your best pages, that is what llms.txt is proposed to do — though the adoption data there is grim. Don't confuse the two files: one denies access, the other suggests reading order.

Each AI company now runs multiple bots — blocking the wrong one costs you citations

The most expensive robots.txt mistake in 2026 is treating each AI company as one bot. It isn't. OpenAI, Anthropic, and Perplexity each split their crawling into distinct jobs — training, search indexing, and live user fetches — and each job has its own user-agent token that you control independently.

For OpenAI: OpenAI uses three main crawlers — GPTBot for AI training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for direct user requests. Each serves a distinct purpose and can be controlled independently via robots.txt. The practical consequence: blocking GPTBot protects your training data without harming your visibility in ChatGPT's answers. If you want to be cited inside ChatGPT Search, the bot you must not block is OAI-SearchBot — and separately, your site has to be reachable at all, which is often a Bing-index problem.

Anthropic made the same split formal in early 2026. The page now lists ClaudeBot (training data collection), Claude-User (fetching pages when Claude users ask questions), and Claude-SearchBot (indexing content for search results) as separate bots, each with its own robots.txt user-agent string. Critically, the tokens don't inherit rules from each other: Each Anthropic bot requires its own directive. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User. Anthropic's own warning about Claude-SearchBot is blunt — blocking it "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results."

So the old advice — copy a list, block everything — is now actively harmful. The outdated strategy — copy a list of AI user-agents, block them all, and move on — now comes with measurable costs. Blocking the wrong bot can remove your content from AI-powered search answers, from user query results, and from citation pools that are growing as a referral traffic source.

The four crawlers, what they actually do, and whether they obey the file

Token	Company	Job	Honors robots.txt?	Block it if…
GPTBot	OpenAI	Training data for GPT models	Yes	You don't want your content training future GPT models
OAI-SearchBot	OpenAI	Indexing for ChatGPT Search	Yes	You don't want to appear in ChatGPT Search answers (rarely)
ClaudeBot	Anthropic	Training data for Claude	Yes	You don't want your content in Claude's training set
Claude-SearchBot	Anthropic	Indexing for Claude search	Yes	You don't want to be cited in Claude's answers (rarely)
PerplexityBot	Perplexity	Search index for cited answers	Documented yes; disputed in practice	You want out of Perplexity entirely — and pair with a WAF rule
Perplexity-User	Perplexity	Live user-initiated fetch	Perplexity says it needn't	You want to block real-time fetches (may not comply)
Google-Extended	Google	Gemini training + grounding policy token	Yes (policy gate)	You want out of Gemini training without touching Search

GPTBot: block it only if you object to training, not search

GPTBot is OpenAI's training crawler, and it respects the file. GPTBot is OpenAI's official web crawler designed to collect publicly available internet content for training and improving GPT models, including ChatGPT. Launched in August 2023, this bot respects robots.txt directives and provides website owners with full control over whether their content is used for AI training. The full block is exactly what you'd expect:

User-agent: GPTBot
Disallow: /

But for most SaaS founders that's the wrong default. Blocking GPTBot opts you out of the model's learned knowledge of your product. The reasonable posture is to allow GPTBot on public marketing and docs while disallowing private areas — the same logic you'd apply to any crawler: members areas, internal tools, customer dashboards. Public content — blog, glossary, product pages, press area — typically belongs in the allowed zone.

ClaudeBot: three tokens, three decisions

Anthropic honors robots.txt across all its bots — including the user-initiated one, which is the opposite of OpenAI's and Perplexity's stance. Anthropic says all three of its bots honor robots.txt, including Claude-User. OpenAI and Perplexity draw a sharper line for user-initiated fetchers, warning that robots.txt rules may not apply to ChatGPT-User and generally don't apply to Perplexity-User. To fully opt out of Anthropic, you must name each token:

User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /

Anthropic also supports a non-standard crawl-delay for rate control: To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt. An example of this might be: User-agent: ClaudeBot Crawl-delay: 1. One housekeeping note: before ClaudeBot, Anthropic operated under the Claude-Web and Anthropic-AI user agents, both now deprecated. Leaving old rules for them in place does no harm.

PerplexityBot: the one where robots.txt may not be enough

Perplexity documents two tokens — PerplexityBot (a traditional crawler for the search index) and Perplexity-User (a user-initiated fetcher). On paper, the crawler complies: PerplexityBot follows robots.txt directives, so allowing the PerplexityBot User-Agent makes the content discoverable through Perplexity responses. Since Reddit and Perplexity are tightly linked in how answers get sourced, most brands should allow PerplexityBot, not block it.

The complication is enforcement. In August 2025, Cloudflare published findings that Perplexity circumvented no-crawl directives. Cloudflare blocks Perplexity due to deceptive crawling behavior, including rotating IP addresses and ignoring robots.txt instructions. Cloudflare announced that they delisted Perplexity's crawler as a verified bot and are now actively blocking Perplexity and all of its stealth bots from crawling websites. Their reported behavior when blocked: ignoring robots.txt directives, rotating IP addresses and spoofing user agents to disguise its crawlers as legitimate human traffic. Perplexity disputes the framing, but the operational lesson is firm: robots.txt rules are not a reliable defense against Perplexity if Perplexity does not want to respect them. Blocking at the server or WAF level is the only real control. If you truly need Perplexity out, pair the robots.txt disallow with a Cloudflare AI-scraper rule.

Google-Extended: the token everyone misreads

Google-Extended is the most misunderstood entry on the list, because it isn't a crawler at all. Google-Extended is a standalone product token for robots.txt. It has no separate HTTP user-agent string and does not fetch pages itself. It's a policy gate applied after Googlebot crawls, deciding whether that content is eligible for Gemini uses.

The decisive fact: Google states that the token does not affect a site's inclusion or ranking in Google Search. And it doesn't touch AI Overviews either, because those run off the live Search index. The block stops Gemini training. It does not stop use in AI Overviews, which draw on the live search index. This makes Google-Extended a rare, clean lever — a rare case where you can opt out of AI training without paying a search-ranking penalty.

User-agent: Google-Extended
Disallow: /

The common failure is confusing Google-Extended with Googlebot. Googlebot and Google-Extended are two separate robots.txt product tokens with two different consequences, and confusing them is how a team blocks the index that gates two tracked platforms while believing it opted out of model training. If you block Googlebot, you leave Search — and take your AI Overview and Gemini grounding eligibility with it.

The recommended default for a SaaS brand that wants AI visibility

For most founders the goal is the opposite of the old block-everything list: you want to be crawled, learned, and cited. Allow the compliant search and training bots on public content, keep private areas closed, and reserve WAF-level blocks for genuine abuse. A sensible starting configuration:

# Search/answer bots — allow so AI can cite you
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /

# Training bots — allow if you want AI to learn your brand
User-agent: GPTBot
Disallow: /private/
User-agent: ClaudeBot
Disallow: /private/

# Gemini training policy (does not touch Search or AI Overviews)
User-agent: Google-Extended
Allow: /

# Everything else
User-agent: *
Disallow: /admin/
Disallow: /api/

Adjust the training lines to your own stance on model training. The point is that access is a strategic decision per bot, not a blanket one: Strategic robots.txt AI crawlers configuration requires balancing accessibility with protection. Consider which content adds value to AI responses and which should remain private. Your public-facing educational content, guides, and thought leadership pieces should typically be accessible, while proprietary methodologies, premium content, and internal documentation should be restricted.

Verify with server logs — don't trust the file blind

Robots.txt is a claim about what should happen; your logs tell you what actually did. After you ship a change, confirm it. Use server logs or Google Search Console to monitor AI bot activity on your site. This gives you real-world data about which crawlers are actually visiting and which pages they access.

Two checks matter most. First, confirm the compliant bots you allowed are actually reaching you — if PerplexityBot never shows up after two weeks, the block is usually happening at a CDN or WAF layer, not in robots.txt. Second, watch for the enforcement gap on Perplexity: because its stealth traffic can look like a human browser, some of it won't appear as a labeled bot at all. Treat robots.txt as your policy statement and your access logs — plus an ongoing AI visibility measurement habit — as your ground truth.

Getting the file right is table stakes, not a strategy. It decides whether AI can see you; it does nothing to make AI choose you. Once access is clean, the work shifts to making your content citable and building the off-site consensus AI actually quotes. If you'd like a read on where your brand stands across ChatGPT, Perplexity, Gemini, and Google AI Overviews, run the free AI visibility report.