Scan your site

Why AI assistants can't see your website, and how to fix it

Last updated

By the FocalScore team at Astrolabix LLC. First published . Sources last checked .

The short answer

When ChatGPT, Claude or Perplexity never mention or quote your site, it is often because they can't read it. There are five common causes: robots.txt rules that turn their crawlers away, a firewall or CDN that blocks them, pages whose text only appears after JavaScript runs, a noindex tag, and snippet limits that leave nothing to quote. Each can be checked in minutes, and the free AI crawler checker covers the first two.

1. robots.txt turns the crawlers away

robots.txt is the file at the root of your site (yoursite.com/robots.txt) that tells crawlers what they may fetch. A line such as User-agent: * followed by Disallow: / shuts out every crawler that obeys it. More often the block is targeted: a "block AI" snippet copied from a forum that names the crawlers assistants use to answer questions, not only the ones that collect training data.

Each AI company runs different crawlers for different jobs, and they are named separately:

Fix: remove the lines that block the answer crawlers, or give them their own group that allows them. Changes are not instant: OpenAI says its search crawler picks up a robots.txt change in about 24 hours, and Perplexity says its systems may take up to 24 hours. The robots.txt guide has a safe example file.

2. Your firewall or CDN blocks them

Bot protection decides before robots.txt is ever read. If your CDN, hosting firewall or a security plugin refuses AI crawlers, they get an error page or a challenge instead of your content, and nothing in your files shows it, which makes it the easiest block to miss.

Cloudflare, for example, sorts AI bots into three kinds: Search (crawlers that index your content to answer questions later), Agent (automated activity acting in real time for a person, such as chat fetch bots) and Training. For each you can block on all pages, block only on pages that show ads, or allow. Cloudflare replaced its older single "Block AI bots" setting with these on 15 September 2026, so if your site is on Cloudflare, look at what yours are set to now. Blocking Agent stops the assistant fetching your page when a customer asks about you.

Some protection judges crawlers by the network address they come from, not just their name. OpenAI publishes the address ranges its crawlers use and recommends allowing them; Anthropic warns that blocking by IP address may not work reliably as an opt-out, because it stops its crawler reading your robots.txt.

Fix: in your CDN or firewall, allow the AI search and agent categories (or the named crawlers), and check any security plugin's bot settings. If a host manages this for you, ask them which AI crawlers they block.

3. The text only appears after JavaScript runs

Some sites send an almost empty page and build the text in the browser with JavaScript. People see the finished page; a crawler that doesn't run JavaScript sees the empty one. Google renders JavaScript, but its own JavaScript SEO guide still recommends server-side rendering or pre-rendering, noting that not all bots can run JavaScript.

How to check: open your page, view its source (Ctrl+U in most desktop browsers) and search for a sentence you can see on screen. If it isn't in the source, a crawler that doesn't run JavaScript can't see it either. The same goes for answers hidden in tabs or accordions that only load when clicked.

Fix: ask your developer or platform for server-side rendering or pre-rendering, so the text is in the page when it arrives.

4. A noindex tag hides the page

A noindex rule, either a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex header, tells search engines to drop the page. Google says it removes the page from Search entirely, and Google's AI Overviews only link to indexed pages. One way the tag gets there: a site builder's "hide from search engines" setting, switched on while the site was being built and never switched off.

Fix: remove the tag or header from each page you want found, and switch off any "hide from search engines" option in your site builder.

5. Snippet limits leave nothing to quote

nosnippet, data-nosnippet and max-snippet limit how much of a page search features may show. Google lists them as the controls for what appears in its AI features, and a page must be eligible to show with a snippet to be a supporting link. Used on purpose they are useful; added by accident site-wide, they leave an assistant nothing to quote.

How to check your site

  1. Run the AI crawler checker. It reads what your robots.txt allows each AI crawler and requests your page under the published user agents of the AI crawlers it tests, so a firewall block by name shows up too.
  2. Run the robots.txt checker for mistakes that block more than you meant.
  3. View the source of your homepage and one service page, and search for a sentence of your own text.
  4. Search the source for noindex and nosnippet.
  5. Look at your CDN's and security plugin's bot settings.

The free scan covers these across your pages, including whether your firewall and robots.txt agree about AI training crawlers, and whether AI crawlers reach your inner pages and not just the homepage.

Checklist

What the free scan checks for this

These are the FocalScore checks that cover this guide's subject. Each one in your report says what it found, shows the evidence and gives the fix.

Sources

Last checked . Documentation, settings and rules change; if one of these pages now says something different, it is right and this guide is out of date. Tell us at info@focalscore.com.

  1. OpenAI: Overview of OpenAI crawlers
  2. Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
  3. Perplexity: Perplexity crawlers
  4. Cloudflare: Block AI bots and AI bot policies
  5. Google Search Central: Understand the JavaScript SEO basics
  6. Google Search Central: Block search indexing with noindex
  7. Google Search Central: AI features and your website