AI Crawlers Explained: How to Optimize Your Site for GPTBot and Friends
AI crawlers fetch pages for training, search and user requests. See the real user agents, what each does, robots.txt examples and how to test your own site.
Key takeaways
- AI crawlers fall into three jobs, training, search indexing and user-triggered fetching, and vendors now publish a separate user agent for each.
- To appear in AI answers, allow the search and user-fetch agents even if you block training agents.
- Some user-triggered fetchers say they may ignore robots.txt, so robots.txt is not a complete control.
- Most AI crawlers fetch raw HTML, so key text must be in the initial response, and a CDN or firewall rule can block bots that robots.txt allows.
- Test with the AI crawlability checker, then confirm in your server logs.
AI crawlers are automated agents that AI companies use to fetch web pages, and they serve three different jobs: collecting content for model training, indexing pages for AI search, and fetching a page on demand when a user asks a question. Vendors now publish a separate user agent for each job, so you can allow search visibility while declining training. To optimize for them, allow the bots you want in robots.txt, make sure key text is in the initial HTML, and confirm nothing at the CDN blocks them.
Run your own site through the AI crawlability checker to see which of these bots your robots.txt currently lets in.
Which AI crawlers exist, and what does each do?
This table lists user agents documented by their operators, as of October 2026. Vendors change names and versions, so check the linked documentation before relying on a string.
| Operator | User agent | Purpose | Robots.txt behavior |
|---|---|---|---|
| OpenAI | GPTBot |
Crawls content that may be used to train generative AI models | Disallowing it signals content should not be used for training (OpenAI) |
| OpenAI | OAI-SearchBot |
Surfaces sites in ChatGPT search features | Opted-out sites are not shown in ChatGPT search answers |
| OpenAI | ChatGPT-User |
Visits pages when a user asks ChatGPT something, and GPT Actions | OpenAI says robots.txt rules may not apply because the user initiates the action |
| Anthropic | ClaudeBot |
Collects web content that could contribute to model training | Disabling it signals future material should be excluded from training (Anthropic) |
| Anthropic | Claude-SearchBot |
Indexes content to improve the quality of search results | Disabling it prevents indexing for search optimization |
| Anthropic | Claude-User |
Fetches pages when a person asks Claude a question | Disabling it prevents retrieval in response to a user query |
| Perplexity | PerplexityBot |
Surfaces and links sites in Perplexity search results; not used for foundation model training | Perplexity recommends allowing it (Perplexity) |
| Perplexity | Perplexity-User |
Fetches a page when a user asks a question | Perplexity says this fetcher generally ignores robots.txt |
Google-Extended |
A robots.txt token controlling use for Gemini training and grounding | Not a ranking signal and does not affect Search inclusion (Google) | |
| Apple | Applebot |
Powers Apple search features such as Spotlight, Siri and Safari | Respects directives aimed at it, and follows Googlebot rules if none mention it (Apple) |
| Apple | Applebot-Extended |
A control for how Applebot-crawled data is used, for example training | Does not crawl; disallowing it does not remove pages from search |
| Meta | meta-externalagent |
Crawls for uses such as training foundation models or indexing content (Meta) | Meta provides robots.txt blocking instructions |
| Meta | meta-externalfetcher |
Fetches individual links at a user’s request | Meta says it may bypass robots.txt |
| Common Crawl | CCBot |
Builds an open repository of web crawl data used widely for research and AI datasets (Common Crawl) | Opt out through robots.txt |
| ByteDance | Bytespider |
ByteDance’s crawler | ByteDance publishes little documentation, and independent reports dispute its robots.txt compliance |
Two notes on this list.
- Google’s AI features run on Googlebot, not on Google-Extended. Google says pages must be indexed and eligible to show with a snippet to appear in AI Overviews and AI Mode. Blocking Googlebot removes you from Search itself.
- Bytespider is the unverified one. Unlike the others, we could not confirm its behavior in current vendor documentation. If you need it blocked, enforce the block at the server or firewall and match on the user agent and, where possible, IP ranges.
Which bots should you allow?
Work from your goal.
| Goal | Allow | Consider blocking |
|---|---|---|
| Appear in AI answers, accept training use | All search, user and training agents | Nothing |
| Appear in AI answers, decline training use | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot |
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent |
| Stay out of AI products entirely | Googlebot and Bingbot for classic search | All of the above |
The middle row is the common choice for publishers who want citations without feeding training sets. It is a policy decision with trade-offs, not a recommendation: training exclusion does not guarantee anything about how models already treat your brand.
What do the robots.txt rules look like?
Robots.txt must sit at the root of the host it applies to. Google also documents that a crawler matches only one rule group, the most specific one that applies to it, so an agent-specific group overrides the wildcard.
Allow everything (default)
User-agent: *
Allow: /Appear in AI search, decline training
# Search and user-triggered agents: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Training and dataset agents: disallowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /
# Everyone else
User-agent: *
Allow: /Block part of a site
User-agent: GPTBot
Disallow: /members/
Disallow: /drafts/
Allow: /How do you make sure crawlers can actually read your pages?
Allowing a bot in robots.txt is step one. The bot also has to get the content.
- Put key text in the initial HTML. An analysis from Vercel in December 2024 found that none of the major AI crawlers it measured rendered JavaScript, with Gemini and Applebot as exceptions. That is a snapshot, so test the bots you care about with
curl. - Check your CDN and firewall. Bot protection can return 403s to user agents robots.txt allows. Cloudflare switched its default to blocking AI crawlers unless they pay as of July 2025, so review any managed rules.
- Return real status codes and keep pages fast. A bot that times out gets nothing.
- Keep a valid sitemap and strong internal links, so pages are discoverable.
- Avoid putting answers in images or interactions only.
The full diagnosis workflow is in why AI can’t see your content.
How do you verify who is actually visiting?
User agent strings are easy to fake. Common Crawl itself warns that other crawlers falsely claim to be CCBot, and recommends reverse DNS verification. For a precise view:
- Filter logs by the user agent names above.
- Check source IPs against ranges the vendor publishes where available (Perplexity, for example, publishes lists for both of its agents).
- Compare allowed versus blocked requests over a week.
If you run GeoGen, the LLM traffic feature uses a tracking script to show AI crawler visits and AI referral visits alongside your visibility data.
Where does llms.txt fit?
It is a separate, optional file, not a crawler control. Read the evidence before investing time in what llms.txt is and whether it works, and use it as a supplement to, never a replacement for, crawl access.
What is a quick optimization checklist?
- robots.txt reviewed and policy written down per bot.
- Search and user-fetch agents you want are allowed.
- Training agents handled deliberately.
- Key text present in raw HTML.
- CDN and WAF rules checked for AI bots.
- XML sitemap current.
- Logs reviewed monthly.
- Vendor documentation re-checked quarterly, since agent names change.
Then confirm the result for your own domain with the AI crawlability checker and score your key pages with the GEO content audit.
Frequently asked questions
What are AI crawlers?
AI crawlers are automated agents run by AI companies that fetch web pages. Some collect content for model training, some build search indexes used in AI answers, and some fetch a page on demand when a user asks a question. Each purpose usually has its own user agent name.
Should I block GPTBot?
It depends on your goal. OpenAI says GPTBot is used for content that may train its models, and disallowing it signals you do not want that. Blocking GPTBot does not remove you from ChatGPT search, which is governed by OAI-SearchBot. Decide per bot, based on whether you want training use, search visibility or both.
Do all AI crawlers obey robots.txt?
Not all. Training and search crawlers from major vendors say they honor it, but user-triggered fetchers may not. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Some other crawlers have disputed compliance, so enforce at the server level if you need a hard block.
Does Google-Extended affect my Google rankings?
No. Google states that Google-Extended does not impact a site's inclusion in Google Search and is not a ranking signal. It controls whether content Google crawls may be used for training future Gemini models and for grounding in Gemini Apps and Vertex AI.
How do I check which AI bots visit my site?
Filter your server or CDN logs by the user agent names listed in this guide, and verify against the vendors' published IP ranges where they exist, because user agent strings can be spoofed. A crawlability checker shows what your robots.txt allows, but logs show who actually visits.
Should I add llms.txt for AI crawlers?
It is optional and unproven. Google has said no AI system currently uses it, and a study of about 300,000 domains found no correlation with AI citations. It costs little to add, but do not expect it to replace crawl access, good content or robots.txt policy.
