AI crawler checker
Can ChatGPT, Claude, Perplexity and Google's AI read your website? Enter an address to see, for 26 AI and search crawlers, what your robots.txt allows — and whether your server actually hands the page over when it's requested under 9 AI crawlers' own user agents.
Why it matters
More and more buyers ask an AI assistant for a recommendation instead of searching. An assistant can only quote or recommend a page it was allowed to read. Two things decide that, and they fail in different places:
- robots.txt — the file at the root of your site that tells crawlers which pages they may fetch. Copied boilerplate and "block AI" snippets often shut out more crawlers than intended.
- Your firewall or CDN — bot protection that refuses visitors by name or behaviour before robots.txt is ever read. It doesn't show up in any file, which is why this tool also makes a live request.
AI answers vs AI training
Each AI company runs several crawlers, and they do different jobs. The ones marked AI answers fetch pages to answer a question and cite the source: block them and that assistant can't read you. The ones marked AI training collect content to train models: blocking them is a legitimate choice and doesn't remove you from answers. This tool reports a training opt-out as a note, never as a problem.
The crawlers it checks
| Crawler | Used for | Live request |
|---|---|---|
| OAI-SearchBotOpenAI. Finds pages to show and link to in ChatGPT's search answers. | AI answers | Yes |
| ChatGPT-UserOpenAI. Fetches a page when a ChatGPT user's question needs it. | AI answers | Yes |
| Claude-SearchBotAnthropic. Indexes pages so Claude can find and cite them when it searches. | AI answers | Yes |
| Claude-UserAnthropic. Fetches a page when a Claude user's question needs it. | AI answers | Yes |
| PerplexityBotPerplexity. Indexes pages to show and link to in Perplexity's answers. | AI answers | Yes |
| Perplexity-UserPerplexity. Fetches a page when a Perplexity user's question needs it. | AI answers | Yes |
| GPTBotOpenAI. Collects pages that may be used to train OpenAI's models. | AI training | Yes |
| ClaudeBotAnthropic. Collects pages that may be used to train Anthropic's models. | AI training | Yes |
| anthropic-aiAnthropic. A retired Anthropic name, still found in many robots.txt files. | AI training | robots.txt only |
| Claude-WebAnthropic. A retired Anthropic name, still found in many robots.txt files. | AI training | robots.txt only |
| Google-ExtendedGoogle. Controls whether Google may use your pages to train and ground its Gemini models. | AI training | robots.txt only |
| Applebot-ExtendedApple. Controls whether Apple may use pages Applebot collects to train its AI models. | AI training | robots.txt only |
| CCBotCommon Crawl. Builds Common Crawl, a free public copy of the web that many AI models are trained on. | AI training | robots.txt only |
| BytespiderByteDance. Collects pages for ByteDance's AI models. | AI training | Yes |
| Google-AgentGoogle. Lets agents running on Google's servers browse and act on the web when a user asks. | AI answers | robots.txt only |
| Google-GeminiNotebookGoogle. Fetches a page a Gemini Notebook user has added as a source. | AI answers | robots.txt only |
| meta-webindexerMeta. Indexes pages to improve Meta AI's search results; Meta says allowing it helps Meta AI cite and link to you. | AI answers | robots.txt only |
| meta-externalfetcherMeta. Fetches a page when a Meta AI user asks for it. | AI answers | robots.txt only |
| Amzn-SearchBotAmazon. Indexes pages for search in Amazon's products, Alexa among them. | AI answers | robots.txt only |
| Amzn-UserAmazon. Fetches a page when an Alexa question needs up-to-date information. | AI answers | robots.txt only |
| MistralAI-IndexMistral. Builds the index behind Mistral's search answers; Mistral says it is not used for training. | AI answers | robots.txt only |
| MistralAI-TrainingMistral. Collects pages to train Mistral's AI models. | AI training | robots.txt only |
| Diffbot-UserDiffbot. Fetches a page when a person using Diffbot's software asks for it. | AI answers | robots.txt only |
| KagibotKagi. Crawls the web for the Kagi search engine. | AI answers | robots.txt only |
| GooglebotGoogle. Google Search's crawler. Google's AI Overviews are built from the same index. | Search engine | robots.txt only |
| BingbotMicrosoft. Bing's crawler. Microsoft Copilot's answers draw on Bing's index. | Search engine | robots.txt only |
Googlebot and Bingbot are never requested under their own names: bot protection rightly refuses anyone claiming to be them from someone else's address, so such a test would report blocks the real crawlers never meet. Google-Extended and Applebot-Extended are robots.txt settings rather than visitors, so there is nothing to request.
How to fix a block
In robots.txt
Find the line the result names and remove it, or give the crawler its own group. A crawler that has a group of its own follows only that group and ignores the rules under User-agent: *. For example, to let the AI answer crawlers in while opting out of training:
User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot Allow: / User-agent: GPTBot User-agent: ClaudeBot Disallow: /
In your firewall or CDN
If robots.txt allows a crawler but the live request was refused or shown a bot check, the block is in front of your site. Check your CDN's bot settings (Cloudflare, for example, has a setting that blocks AI crawlers), any firewall rules that match on user agent, and security plugins such as those common on WordPress. Ask your provider whether it lets verified AI crawlers through: the real crawlers are checked by the addresses they come from.
What this check can't see
- Blocks by address. Our requests come from our server, so a rule that refuses (or admits) a crawler's real address range doesn't show here.
- Pages that need JavaScript to show their text. Most AI crawlers don't run it; the full scan checks for that.
- Whether an assistant actually mentions you. Being readable is the first step, not a guarantee.
Questions
Does blocking GPTBot remove my site from ChatGPT?
Not on its own. OpenAI runs separate crawlers: GPTBot collects training data, OAI-SearchBot finds pages for ChatGPT's search answers, and ChatGPT-User fetches a page when someone asks about it. Blocking GPTBot opts you out of training; it is OAI-SearchBot and ChatGPT-User that decide whether ChatGPT can read and cite you.
My robots.txt allows the crawler, so why does the live request say it was refused?
Because something in front of your site turned it away before robots.txt mattered: a firewall rule, your CDN's bot protection, or a security plugin. robots.txt is a request crawlers choose to follow; a firewall is a wall. Look for a rule that blocks AI crawlers or that user agent by name.
If the live request gets the page, does that prove the real crawler gets in?
No. Our request carries the crawler's published user agent but comes from our server, not from the vendor's addresses. Bot protection that verifies crawlers by address can let the real one through while refusing us, or refuse the real one while letting us through. The live column tells you whether your site blocks a crawler by name — the commonest kind of block — and nothing more.
Does Google-Extended affect Google Search or AI Overviews?
Google says Google-Extended does not affect your inclusion or ranking in Google Search. It controls whether pages Google has crawled may be used for its Gemini models. Google's search features, AI Overviews included, work from Googlebot's crawl, so Googlebot is the row that matters for them.
Should I block AI training crawlers?
That's a business choice, and this tool doesn't score it either way. Blocking a training crawler such as GPTBot or ClaudeBot doesn't remove you from AI answers. Blocking the crawlers that answer questions — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot — does.
Will these requests show up in my server logs?
Yes: one request for robots.txt and one for the page as FocalScoreBot, then one per AI crawler listed as tested below, each carrying that crawler's user agent. They come from FocalScore's server, not from OpenAI, Anthropic or Perplexity.
The background, in plain English: Why AI assistants can't see your website, and how to fix it