Robots.txt TesterAuditReady
We read a site's robots.txt and sitemaps, then test whether an address may be crawled by Googlebot, Bingbot or AI crawlers such as GPTBot and ClaudeBot, and point out rules that block more than intended.
Add more paths to test under the address bar. No account needed. 6 of 6 checks left today.
How to use it
Four steps from an address to results you can act on.
The file is fetched and its rules grouped by crawler.
Sitemaps listed in robots.txt or at the usual address.
Your address is matched against each crawler's rules.
Which rule decided, and what to change.
What it brings you
Every result comes with the proof behind it, so you can check it yourself.
Unknown lines and rules that never apply.
CSS and scripts that pages need to render.
Which AI crawlers are allowed or blocked.
Size, errors and addresses that fail or are blocked.
What the result looks like
Results are sorted so the most important rows come first. This example uses sample data to show the layout.
| Crawler | Address | Rule | Status |
|---|---|---|---|
| Googlebot | /blog/post | Allow: / | Allowed |
| Googlebot | /search?q=x | Disallow: /search | Blocked |
| GPTBot | /blog/post | Disallow: / | Blocked |
Questions
No. It stops crawling. A blocked page can still be indexed from links. Use noindex to keep it out.
It depends on whether you want AI assistants to read and quote your pages. The tester shows what your file says today.