Skip to content

AI crawlability checker

AI Crawlability Checker: Can GPTBot and Other AI Bots Crawl Your Site?

Enter a domain and see, bot by bot, which AI crawlers your robots.txt allows and which it blocks.

10 free runs left today

How it works

How the crawlability check works

  1. Enter your domain

    We fetch the robots.txt at the root of the domain.

  2. We read the rules

    For every AI crawler we find the group of rules that applies to it and test whether it can fetch your pages.

  3. See who is allowed

    You get a score and a bot-by-bot list, so you can fix a block or keep it on purpose.

What is GPTBot and should I block it?

GPTBot is OpenAI's web crawler. OpenAI uses what it collects to train its models. Blocking it in robots.txt stops that collection, and it does not remove you from ChatGPT's live answers, which rely on a different bot.

That bot is OAI-SearchBot, which indexes pages for ChatGPT search. ChatGPT-User is a third one that fetches a page when a person asks ChatGPT to open it. If you want to appear in ChatGPT answers, allow OAI-SearchBot and ChatGPT-User, whatever you decide about GPTBot.

Which AI crawlers matter?

Search and retrieval bots affect whether you are cited today: OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot. Training bots affect what models know: GPTBot, ClaudeBot, Google-Extended (Gemini), Applebot-Extended, CCBot (Common Crawl) and meta-externalagent.

Google AI Overviews and AI Mode use Googlebot, so blocking Googlebot removes you from both Google search and those answers. Google-Extended is a separate control for Gemini's use of your content.

How do I allow or block a specific AI bot?

Add a group to your robots.txt. To allow a bot, write User-agent: OAI-SearchBot followed by Allow: /. To block one, write User-agent: GPTBot followed by Disallow: /. Rules for a named bot override the generic User-agent: * group.

Robots.txt is a request, not a lock. Well-behaved crawlers respect it, and some do not. If you need to block a bot for certain, enforce it at your firewall or CDN as well.

AI crawler questions

How do I check if GPTBot can crawl my site?

Enter your domain in the checker above. It reads your robots.txt and tells you whether GPTBot, and nine other AI crawlers, are allowed or blocked. A missing robots.txt means every bot is allowed.

Will blocking GPTBot stop ChatGPT from mentioning my brand?

Not by itself. GPTBot gathers training data. ChatGPT's live answers use OAI-SearchBot and ChatGPT-User. Blocking those two removes you from ChatGPT search results, while blocking only GPTBot leaves them intact.

Does robots.txt actually stop AI crawlers?

It stops crawlers that honour it, which includes the major ones from OpenAI, Anthropic, Google, Perplexity and Apple. It cannot stop a crawler that ignores the rules, so sensitive content needs a login or firewall rule instead.

What does the score mean?

The score is the share of the AI crawlers we test that are allowed to fetch your homepage. 80 and above is excellent, 50 to 79 is partial and below 50 is limited. A low score is only a problem if you want to be visible in AI answers.

Should I add an llms.txt as well?

It is a low-cost addition. llms.txt is a plain file that points AI tools to your most useful pages, and it does not replace robots.txt. Our llms.txt generators can build one in a minute.

Keep going

All free tools

Be the brand AI recommends.

Start tracking how ChatGPT, Google AI Overviews, Gemini, Perplexity and 9 more AI engines talk about you. Results in minutes.

Get started Book a demo