AI crawlers list: which bots to allow in robots.txt
Every major AI crawler user agent, what each one does (training vs search vs live fetch), which ones you must allow to appear in ChatGPT, Claude and Gemini answers, and copy-paste robots.txt rules.
Updated 24 October 2026
AI assistants have separate crawlers for training, for their search index, and for live fetches when a user asks about a page. The search and live-fetch bots are the ones that decide whether you can appear in answers. Blocking a training bot does not hide you from AI search.
The full list
| User agent | Company | Used for | Recommendation |
|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT search index | Allow — needed to appear in ChatGPT search |
ChatGPT-User | OpenAI | Fetches a page live when a user asks | Allow |
GPTBot | OpenAI | Model training | Your choice — blocking doesn't affect ChatGPT search |
Claude-SearchBot | Anthropic | Claude search index | Allow |
Claude-User | Anthropic | Fetches a page live when a user asks | Allow |
ClaudeBot | Anthropic | Model training | Your choice |
Google-Extended | Control token: Gemini training and grounding | Allow if you want to appear in Gemini answers | |
Googlebot | Search, AI Overviews, AI Mode | Always allow | |
PerplexityBot | Perplexity | Perplexity search index | Allow |
Perplexity-User | Perplexity | Live fetch for users | Allow |
Applebot | Apple | Siri, Spotlight, Safari | Allow |
Applebot-Extended | Apple | Control token: Apple model training | Your choice |
Sources: OpenAI, Anthropic, Google, Perplexity, Apple.
robots.txt: allow AI search, block training
A common setup for businesses — appear in AI answers, but don't donate content to model training:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xmlrobots.txt: allow everything
User-agent: *
Allow: /
Sitemap: https://yourwebsite.com/sitemap.xmlCommon mistakes
- A blanket
Disallow: /underUser-agent: *left over from a staging site — this blocks every AI crawler. - Security plugins and CDNs that block “bots” — Cloudflare's AI-bot blocking toggle, WordPress firewall plugins. Check them, not just robots.txt.
- JavaScript-only pages. Allowed crawlers still see nothing if your content is rendered client-side.
Not sure what your site does? Run the free AI Visibility Checker — it reads your robots.txt and tells you which AI bots are blocked.
Frequently asked questions
Should I block GPTBot?
Blocking GPTBot only stops OpenAI from using your pages to train models. It does not remove you from ChatGPT search — that's OAI-SearchBot. Many businesses block GPTBot and allow OAI-SearchBot.
Does GPTBot respect robots.txt?
OpenAI states that GPTBot and OAI-SearchBot respect robots.txt. ChatGPT-User (fetches a page because a user asked) may not apply robots.txt the same way, per OpenAI's documentation.
Does blocking Google-Extended hurt my Google ranking?
No — it doesn't affect Google Search ranking. But Google-Extended also controls whether your content is used for grounding in Gemini, so blocking it can keep you out of Gemini answers.
Do AI crawlers execute JavaScript?
Mostly no. OpenAI's and Anthropic's crawlers and PerplexityBot read raw HTML. If your text only appears after JavaScript runs, they see an almost empty page.