AI companies separate training crawlers from search crawlers and user-triggered fetchers, each with its own robots.txt token (Sitemaps and robots.txt). Blocking OpenAI 86 's or Anthropic's training bot keeps you in their AI search; blocking their search bot can remove you.
| robots.txt token | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training foundation models |
| OAI-SearchBot | OpenAI | ChatGPT search results |
| ChatGPT-User | OpenAI | Visits a user asked for |
| ClaudeBot | Anthropic | Training |
| Claude-SearchBot | Anthropic | Search result quality |
| Claude-User | Anthropic | Fetches for a user's question |
| PerplexityBot | Perplexity | Perplexity search; not training |
| Google-Extended | Gemini 1 training and grounding | |
| Applebot-Extended | Apple | Apple model training |
The two -Extended tokens only control use: Googlebot and Applebot fetch, and blocking them affects neither Google Search nor Apple's Spotlight, Siri and Safari 10 results. User-triggered fetchers such as ChatGPT-User and Perplexity-User may ignore robots.txt, and OpenAI and Perplexity need up to about 24 hours to apply changes.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /To leave Google's AI Overviews and AI Mode without leaving Search, use the Search generative AI setting that every Search Console 1 property has had since August 31, 2026, or page-level nosnippet rules (AI Overviews).
llms.txt, proposed by Jeremy Howard in September 2024 and revised as v2 in August 2026, is a Markdown file at /llms.txt (or /docs/llms.txt) with an H1 site name, a blockquote summary and H2 lists of links, ideally to .md versions of pages, for agents such as coding assistants reading documentation. Mintlify 35,007 , GitBook 7,219 , Yoast SEO 6,399 and Wix 630 generate one, and Lighthouse 30,824 checks only that fetching it causes no server error. Google Search ignores it, so it will "neither harm nor help" rankings: publish one for developer docs, not for search visibility.