robots.txt for small businesses: what to allow, what to block

Last updated 9 October 2026

By the FocalScore team at Astrolabix LLC. First published 9 October 2026. Sources last checked October 2026.

The short answer

Most small-business sites need a very short robots.txt: allow everything, keep crawlers out of pages with no value to a searcher (a basket, internal search results), and point to your sitemap. Never use it to hide a page from Google (use noindex for that), never block the CSS and JavaScript your pages need, and check that any AI rules block only what you meant to.

What robots.txt does, and what it doesn't

It is a plain text file at the root of your site, at https://yoursite.com/robots.txt, that tells crawlers which addresses they may fetch. Google reads it only from the top level of the site, and its rules apply only to the same host, protocol and port: shop.yoursite.com needs its own file.

What it does not do:

It doesn't keep a page out of Google. Google says a page blocked in robots.txt can still be indexed if other sites link to it, and its address can still appear in results. To keep a page out, use a noindex tag, and leave the page crawlable so the tag can be seen.
It doesn't protect anything. Anyone can open it, and well-behaved crawlers obey it by choice. Listing a private folder in it tells everyone where the folder is. Put private pages behind a login.
It doesn't bind every visitor. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.

A safe starting file

For a typical brochure site or small shop, something like this is enough:

User-agent: *
Disallow: /cart/
Disallow: /search

Sitemap: https://www.yoursite.com/sitemap.xml

Swap the two Disallow lines for your own basket and site-search addresses, or remove them; an empty file, or one with Disallow: and nothing after it, allows everything. The Sitemap line must be a full address, starting with https://.

How crawlers read the rules

These are the rules as Google documents them; the main AI crawlers say they honour the same standard.

A crawler follows one group only: the one with the most specific User-agent line that matches it. If you add a group for GPTBot, GPTBot ignores everything under User-agent: *, so repeat any general Disallow lines it should still obey.
The most specific path wins. When an Allow and a Disallow rule both match, Google uses the longer one, and the less restrictive one if they are equally long.
Paths are case-sensitive. Disallow: /Admin does not block /admin.
Size limit. Google reads the first 500 KiB and ignores the rest.
Errors matter. If the file returns a 4xx error, Google treats the site as having no robots.txt. If it returns a 5xx server error, Google stops crawling the site for a while.
Crawl-delay is not supported by Google. Anthropic says its crawlers do support it. A long delay slows every crawler that honours it.

Choosing what AI crawlers may do

AI companies name their crawlers by job, so you can allow the ones that let assistants find and cite you while opting out of training. This example lets the answer crawlers in and keeps two training crawlers out:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: *
Disallow: /cart/

Sitemap: https://www.yoursite.com/sitemap.xml

Two things to know. First, the answer crawlers' group above doesn't include Disallow: /cart/, so they may fetch the basket; add it to their group if that matters to you. Second, Google's training opt-out is a separate token, Google-Extended, and Google says it does not affect your inclusion or ranking in Google Search. Never block Googlebot to opt out of AI: that stops Google crawling your pages for Search.

Changes take time to reach the crawlers: OpenAI says about 24 hours for its search crawler.

Common mistakes

Disallow: / left over from building the site. It blocks everything. Check the file whenever a site goes live or moves host.
Blocking CSS, JavaScript or image folders. Google asks you not to block files your pages need to display, or it can't understand the page properly.
Putting noindex in robots.txt. Google stopped supporting it on 1 September 2019. Use the meta tag or the X-Robots-Tag header instead.
Blocking a page you also marked noindex. The crawler can't read the page, so it never sees the tag, and the address can stay in results.
A named group that drops your general rules, as described above.
A copied "block AI" list that includes OAI-SearchBot, Claude-SearchBot or PerplexityBot, which removes you from those assistants' answers.
A broken sitemap line: a relative address, an old domain, or a sitemap that no longer loads.
A robots.txt that is really an error page, or a server error that tells Google to stop crawling.

How to check yours

Run the free robots.txt checker: it reads your file, flags syntax mistakes, tells you whether it blocks your homepage or the files your pages need, shows what each major crawler is allowed, and checks that its sitemap lines load. For AI crawlers specifically, the AI crawler checker also tests whether your firewall lets them through. If assistants still can't see you, read why AI assistants can't see your website.

Checklist

The file loads at /robots.txt as plain text, without an error.
No Disallow: / under User-agent: * unless you really mean it.
CSS, JavaScript and image folders are not blocked.
Nothing private is listed in it; private pages are behind a login.
Pages you want out of Google use noindex and are not blocked here.
Every named group repeats the general rules it should keep.
AI search crawlers are allowed; any training opt-out names only training crawlers.
The Sitemap line is a full https address that loads.

What the free scan checks for this

These are the FocalScore checks that cover this guide's subject. Each one in your report says what it found, shows the evidence and gives the fix.

robots.txt
Rendering files are crawlable
The sitemap in robots.txt loads
AI crawlers allowed
Search crawlers allowed
Named crawler groups keep your Disallow rules
Crawl delay isn't starving AI crawlers

Sources

Last checked October 2026. Documentation, settings and rules change; if one of these pages now says something different, it is right and this guide is out of date. Tell us at info@focalscore.com.