AI crawlers and robots.txt: GPTBot, ClaudeBot, PerplexityBot
AI companies run separate agents for separate jobs, and robots.txt treats each one on its own. Training crawlers such as GPTBot and ClaudeBot collect pages for future models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes that AI answers cite. User-triggered agents such as ChatGPT-User fetch a page when someone asks a question that needs it. Blocking a training crawler tells that vendor to leave your content out of future training and leaves you citable; blocking a search crawler can take you out of that engine's answers. Two exceptions catch SaaS teams out. Google-Extended governs Gemini's training and its grounding in Gemini Apps together, and several user-triggered agents may not follow robots.txt at all. For a B2B SaaS marketing site the sound default is to allow every search and user agent, make the training decision deliberately, serve the facts in server-rendered HTML, and treat llms.txt as an optional extra rather than a control.
Every plan at The Rank Master includes a full technical audit in week one, with the fixes ranked by what they are worth. Book a 30-minute call and we will look at your category live and show you what the engines say when a buyer asks for a recommendation.
AI agents by vendor and job
The table draws on each vendor's own crawler documentation, read on 17 October 2026. "If you block it" gives what the vendor says the block does or, where the vendor says nothing, what follows from the agent's job.
| Vendor | robots.txt token | Job | Follows robots.txt? | If you block it |
|---|---|---|---|---|
| OpenAI | GPTBot | Collects content that may be used to train OpenAI's foundation models | Yes: OpenAI offers it as the training opt-out | Content is marked as not for training; ChatGPT search is a separate setting |
| OpenAI | OAI-SearchBot | Finds sites to show in ChatGPT's search features | Yes | Pages are left out of ChatGPT search answers, apart from navigational links |
| OpenAI | ChatGPT-User | Visits a page when a ChatGPT or custom GPT user's request calls for it | OpenAI says robots.txt rules may not apply | Where the rule is applied, ChatGPT cannot open the page for a user; search inclusion is unaffected |
| OpenAI | OAI-AdsBot | Reviews the pages that advertisers submit as ads in ChatGPT | Not stated | Matters only if you advertise in ChatGPT |
| Anthropic | ClaudeBot | Collects content that could contribute to training Anthropic's models | Yes | Future content is excluded from training datasets |
| Anthropic | Claude-SearchBot | Indexes content to improve the quality of Claude's search results | Yes | Content is not indexed for search, which may reduce your visibility in Claude's answers |
| Anthropic | Claude-User | Fetches pages when a Claude user's question needs them | Yes | Claude cannot retrieve your page for a user's question |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity's search results; not used for training | Perplexity recommends allowing it | You drop out of Perplexity's search results |
| Perplexity | Perplexity-User | Fetches a page a user's question calls for; no crawling, no training | Generally ignores robots.txt | Little, through robots.txt |
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | Yes, whenever it crawls automatically | Google cannot crawl your pages for Search, AI Overviews and AI Mode included | |
| Google-Extended | A control token, with no crawler of its own, for Gemini: training future models, and grounding answers in Gemini Apps and on Vertex AI | Yes | No effect on Google Search; your pages stop grounding Gemini Apps' answers and stay out of future Gemini training | |
| Google-Agent (no robots.txt token listed) | Agents on Google's infrastructure that browse and act at a user's request | Google's user-triggered fetchers generally ignore robots.txt | Little, through robots.txt | |
| Meta | Meta-WebIndexer | Indexes the web to improve the results of Meta AI's search | Meta ties allowing it to Meta AI citing and linking your content | Meta AI's answers become less likely to cite and link your pages |
| Meta | Meta-ExternalAgent | Crawls for uses that include training foundation AI models and indexing content to improve Meta's products | Yes: Meta gives the disallow rule | Those crawls of your site stop; Meta AI's search crawler is a separate agent |
| Meta | Meta-ExternalFetcher | Fetches single links when a user asks, including for agentic AI features | Meta says it may bypass robots.txt | Little, through robots.txt |
| Apple | Applebot-Extended | A control token for training Apple's foundation models; it does not crawl | Yes | Training opt-out only: Applebot still crawls for Siri, Spotlight and Safari |
| Common Crawl | CCBot | Crawls the web for Common Crawl's open archive | Yes: Common Crawl gives the rule to block it | Common Crawl stops crawling your site |
| xAI (Grok) | None documented | Grok's web search opens pages, but xAI names no agent | Not documented | Nothing to set |
Sources: OpenAI, Anthropic, Perplexity, Google's common crawlers and user-triggered fetchers, Meta, Apple, Common Crawl and xAI's full documentation, all read 17 October 2026.
Three patterns run through the table.
Training and search are separate switches almost everywhere. OpenAI and Anthropic each give training its own token; Meta keeps Meta AI's search crawler apart from Meta-ExternalAgent, which crawls for training among other uses; and Perplexity says its search crawler does not collect training data at all. Google is the exception. Googlebot covers Search and its AI features, and Google-Extended covers both Gemini's training and its grounding, so a Google-Extended block cannot keep you out of training while leaving you in Gemini Apps' grounded answers. Google's Search Console help adds that Google-Extended is also how a site limits training of the models behind AI Overviews and AI Mode, without changing whether its pages appear there. Our guide to getting cited in AI Overviews and AI Mode sets out the other controls Google offers for those two features.
User-triggered agents follow different rules. OpenAI, Perplexity, Google and Meta all describe their user-triggered agents as acting for a person, and say robots.txt may not apply, may be bypassed or generally does not apply. Anthropic is the outlier: it says all three of its bots honour robots.txt, and that disabling Claude-User stops Claude retrieving your content for a user's query. If you need a page kept away from these agents, robots.txt is the wrong tool; put it behind a login.
Grok has no documented agent. As of 17 October 2026 xAI's documentation names no crawler or user agent and never mentions robots.txt, though Grok's web search tool does open pages. Our guide to getting cited by Grok covers what that means in practice.
What to allow on a B2B SaaS site
A typical SaaS site has four kinds of content, and each deserves its own answer.
| Part of the site | Search and user agents | Training crawlers |
|---|---|---|
| Marketing pages, pricing, comparisons, blog | Allow: these pages exist to be found and quoted | Your call; there is little to protect in pages written to be repeated |
| Public documentation, help centre, integration pages | Allow: buyers ask engines about integrations, single sign-on and security | Your call |
| Paid research, proprietary data, gated assets | Allow only what you want cited; gate the rest | Disallow if the content is the product |
| App, account and admin paths | Disallow, and protect with authentication | Disallow |
A file that keeps every engine able to cite you while opting out of training could look like this:
# Training crawlers: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else, the search and user agents included
User-agent: *
Disallow: /app/
Disallow: /account/
Sitemap: https://www.example.com/sitemap.xml
Four things about it are deliberate.
- One group, several agents. RFC 9309 defines a group as one or more user-agent lines followed by rules, so the four training tokens can share a single Disallow.
- Google-Extended is left out. Adding it to the training group would also withdraw your pages from grounding in Gemini Apps. Add it only if that trade is acceptable to you.
- The search and user agents have no group of their own. OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Googlebot fall back to the
*group, private paths included. Give any of them a group of its own and it stops reading*, so the private paths must be copied into that group. Our Perplexity guide shows the pattern with PerplexityBot. - Applebot borrows Googlebot's rules. Apple says that when a file mentions Googlebot but not Applebot, Applebot follows the Googlebot instructions. A Googlebot-only group can therefore change Apple's crawling too.
For OpenAI's own example of the same split (OAI-SearchBot allowed, GPTBot disallowed) and the settings around it, see our guide to getting cited by ChatGPT.
How crawlers read the file
RFC 9309, the IETF's standards-track specification for robots.txt, published in September 2022, settles questions that guides often skip.
- Matching is by token, without regard to case. Two groups that name the same token are merged into one. An agent that finds no group naming it uses the
*group, and with no*group at all, no rules apply to it. - The longest matching path wins. If an Allow and a Disallow match equally, the Allow should win.
- The status code of robots.txt itself matters. A 4xx response tells a crawler there are no rules, so it may fetch anything. A 5xx response, from a server error or a misbehaving bot challenge in front of the file, tells it to assume the whole site is disallowed. After about 30 days of errors it may treat the file as unavailable or keep using a cached copy.
- Changes are not instant. Crawlers should not use a cached copy for more than 24 hours unless the file is unreachable. OpenAI says ChatGPT search takes about 24 hours to adjust after a robots.txt update, and Perplexity and Meta each allow up to 24 hours.
- Size has a floor. Any parsing limit a crawler sets must be at least 500 kibibytes, so only a very large file risks being cut short.
- It is not security. The RFC says the protocol is "not a substitute for valid content security measures", and listing a private path in robots.txt advertises it. Anything that must stay private needs authentication.
Each host also keeps its own file: the robots.txt on www does nothing for docs.example.com or app.example.com. Anthropic asks for the opt-out rule to be added for every subdomain you want covered.
Past robots.txt: firewalls, IP lists and impostors
A user-agent string costs nothing to fake. Google warns that the string can be spoofed, and Common Crawl says it knows of crawlers falsely presenting themselves as CCBot. Before you exempt a visitor from a firewall rule, or trust a log line, check it against what the vendor publishes:
| Vendor | How to verify its agents |
|---|---|
| OpenAI | IP lists per agent: searchbot.json, gptbot.json, chatgpt-user.json |
| Anthropic | The IP list at claude.com/crawling/bots.json |
| Perplexity | perplexitybot.json and perplexity-user.json |
| common-crawlers.json and reverse DNS under googlebot.com; user-triggered-agents.json for Google-Agent | |
| Apple | Reverse DNS under applebot.apple.com, or Apple's published CIDR list |
| Common Crawl | Reverse DNS under crawl.commoncrawl.org, or ccbot.json |
Two cautions go with it. Anthropic warns that blocking its IP addresses may not opt you out reliably, because the block also stops it reading your robots.txt; use robots.txt for opt-outs and the firewall for abuse. And if your bot protection challenges automated traffic, check what it does on /robots.txt itself: an error page served in its place reads, to a compliant crawler, as either "no rules" or "disallow everything", depending on the status code.
JavaScript: content many AI crawlers never see
Vercel published a study on 17 December 2024, with MERJ, based on traffic across Vercel's network and on sites built with different stacks. In that data none of the major AI crawlers rendered JavaScript: OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta's Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot. ChatGPT's and Claude's crawlers did download JavaScript files (11.50% and 23.84% of their requests) but did not run them. Gemini, which crawls with Googlebot's infrastructure, rendered pages fully, and so did Applebot. The same study found 34.82% of ChatGPT's fetches and 34.16% of Claude's landing on 404 pages, against 8.22% for Googlebot.
The study is almost two years old and names ClaudeBot only, not Claude-SearchBot or Claude-User, and crawlers change. None of the OpenAI, Anthropic or Perplexity crawler pages we read on 17 October 2026 says whether their agents run JavaScript. Until one does, assume they do not. Fetch your pricing page, a comparison page and an integration page without a browser (with curl, or by viewing the source) and search the raw HTML for the prices, plan limits and integration names. Anything you cannot find there may be invisible to these agents; server-side rendering or pre-rendering fixes it. The 404 figures point to the cheap fix Vercel itself recommends: keep redirects in place and the sitemap current.
llms.txt: a proposal, not a control
llms.txt is a Markdown file, placed at /llms.txt or under any path, that gives an agent a short guided route into a site: a title (the only required part), a short summary, and lists of links, ideally to Markdown versions of the key pages. It is set out at llmstxt.org, where it was first published on 3 September 2024; the current version 2 was last modified on 10 August 2026. The page calls itself a proposal and says the specification is open for community input. It is not a standard in the way RFC 9309 is, and it does not grant or refuse access. robots.txt says what a crawler may fetch; llms.txt is a reading guide an agent might consult while helping someone.
What the platforms say about it:
- Google: its guide to generative AI features in Search (updated 10 July 2026) says Google Search does not use llms.txt, and that keeping one neither helps nor harms a site in Search.
- OpenAI, Anthropic and Perplexity: none of their crawler pages asks site owners for one. Each does publish an llms.txt for its own developer documentation, as xAI does, which shows the format is used by documentation sites, not that it changes what their crawlers do.
Our view: an llms.txt is cheap to publish and most useful for developer documentation that coding agents read. Do not expect it to earn citations, and never use it in place of robots.txt.
Worked example: this site's robots.txt
This is the robots.txt that therankmaster.com publishes alongside this guide:
User-agent: *
Allow: /
# AI answer engines: explicitly welcome.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://therankmaster.com/sitemap.xml
Reading it the way a crawler does:
- The first group lets everything in. Any agent without a group of its own may fetch the whole site. The agents the file leaves unnamed include Googlebot, Bingbot, Applebot, Google-Agent and OAI-AdsBot.
- The nine named groups change nothing today. Lines 5 to 22 give GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Claude-User and Google-Extended a group each, and every one of them allows everything, exactly as the catch-all does. Under RFC 9309 each named agent reads only its own group, so what the groups add is a statement of intent: as the comment on line 4 says, AI answer engines are welcome, training crawlers included. The Perplexity-User group is more statement than control, since Perplexity says that agent generally ignores robots.txt.
- The Sitemap line points every crawler at the list of pages.
Two rules from RFC 9309 make any file shaped like this easy to break. Crawlers without a group of their own, Googlebot and Bingbot among them, follow the catch-all, so tightening a catch-all to Disallow: / shuts out every one of them, search crawlers included. And a named group replaces the catch-all for its agent, so a Disallow: /app/ added under User-agent: * alone never reaches the named agents. Anyone narrowing a catch-all should first give every crawler they rely on a group of its own, then repeat each rule in every named group.
Questions about AI crawlers
Which AI agents ignore robots.txt?
By their vendors' own accounts: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, Google says the same of its user-triggered fetchers, Google-Agent among them, and Meta says Meta-ExternalFetcher may bypass them. All four describe these agents as acting at a user's request. Anthropic says its bots, Claude-User included, honour robots.txt. xAI documents no agent at all.
Can you block AI training and still be cited in AI answers?
Yes. GPTBot, ClaudeBot and Applebot-Extended are training opt-outs that leave the same vendors' search agents alone, and blocking CCBot affects only Common Crawl's own crawling. Meta's search crawler, Meta-WebIndexer, is a separate agent from Meta-ExternalAgent, which crawls for training among other uses. Perplexity's search crawler does not collect training data in the first place. Google is the trade-off: Google-Extended blocks both Gemini's training use and its grounding in Gemini Apps, though it does not affect Google Search, AI Overviews and AI Mode included.
Method and sources
Each vendor's facts come from its own page, all read on 17 October 2026: OpenAI's crawler overview; Anthropic's help article on its bots (last updated 7 April 2026); Perplexity's crawler documentation; Google's lists of common crawlers and user-triggered fetchers, its Search generative AI control help and its guide to generative AI features; Meta's crawler page (updated 21 May 2026); Apple's Applebot page; Common Crawl's CCBot page; and xAI's full documentation file. The parsing rules come from RFC 9309, the llms.txt description from llmstxt.org and the JavaScript figures from Vercel's December 2024 study, which we cite with its date because crawlers change. The worked example is the robots.txt this site publishes with this guide.
The Rank Master gets B2B SaaS companies found in Google and recommended by the seven AI engines buyers now ask first: ChatGPT, Perplexity, Gemini, Claude, Grok, Google AI Overviews and Google AI Mode. Every plan includes a full technical audit in week one, and the work comes with published pricing, open measurement and no annual lock-in. Book a 30-minute call to see what the engines say about your category today and get a plain answer on whether we can help.