robots.txt checker
Enter your website to fetch and read its robots.txt the way search engines do. You'll see whether it blocks your homepage or the files your pages need, what Google, Bing and the AI crawlers may fetch, which lines no crawler can read, and whether its sitemap lines lead anywhere. Enter a page's full address to test that page instead of the homepage.
What it checks
- Delivery. Whether /robots.txt answers at all, and how: a file, a 404 (no rules — everything allowed), a server error (Google pauses crawling), or a web page where the file should be.
- Whole-site and homepage blocks. The classic leftover from a staging site:
Disallow: /underUser-agent: *. - Each major crawler. Googlebot, Bingbot and the AI crawlers, with the exact line that decides each one — read with the same matcher our full scan uses, which combines groups and picks the longest matching rule as the robots.txt standard (RFC 9309) says.
- Stylesheets and scripts. We read the page you entered, list the CSS and JavaScript files it loads from your site, and test each against Googlebot's rules.
- Lines no crawler reads. Misspelt fields (
Dissallow,User agent), paths without a leading slash, full URLs where a path belongs, rules before any User-agent line, andNoindex:, which Google stopped reading in 2019. - Format. The Content-Type it's served with, an invisible byte order mark at the start, and a file larger than the 500 KiB Google reads.
- Sitemap lines. Whether there is one, whether it's a full address, and whether the first three actually load as sitemaps.
What a good robots.txt looks like
For most small-business sites, this is all it needs:
User-agent: * Disallow: /search Allow: / Sitemap: https://www.example.com/sitemap.xml
Keep it short, block only what genuinely shouldn't be crawled (internal search results, cart and account pages), and never block the folders that hold your CSS, JavaScript or images. Your site builder's SEO settings usually have a robots.txt box; on other hosts it's a plain text file in your site's top folder.
What this check can't see
- Rules served differently to different crawlers, or set by a firewall rather than in the file.
- Pages blocked by a noindex tag or header — those are page settings, not robots.txt. The full scan reads them.
- Every sitemap entry. We confirm the sitemap loads; the full scan samples the pages it lists.
Questions
Do I need a robots.txt at all?
No. A site with no robots.txt (one that answers 404) is crawled freely — that's the safest default there is. A robots.txt is worth having to name your sitemap and to keep crawlers out of things like internal search results, not because crawlers need permission.
Does Disallow keep a page out of Google?
Not reliably. Disallow stops crawling, not indexing: a blocked page that other sites link to can still appear in results, just without a description. To keep a page out of search, let it be crawled and add a noindex meta tag — a crawler that is blocked never sees the tag.
What happens if my robots.txt returns an error?
A 404 means "no rules", and everything may be crawled. A server error (5xx) is different: Google treats it as "crawl nothing for now", and if it keeps failing your whole site can drop out of the crawl. Fix a failing robots.txt before anything else.
Why does blocking CSS or JavaScript matter?
Google renders pages the way a browser does. If robots.txt blocks the stylesheets or scripts a page needs, Google sees a broken or unstyled version and judges the page — including whether it works on a phone — on that.
Is Crawl-delay worth setting?
Rarely. Google ignores it. Bing and a few others honour it, and a long delay slows how quickly they pick up new pages. If a crawler is overloading your server, your host or CDN has better tools.
Does a group for Googlebot add to the * rules?
No — it replaces them. A crawler follows the most specific group that names it and ignores the * group entirely, so rules you want everywhere must be repeated in each named group.
The background, in plain English: robots.txt for small businesses: what to allow, what to block