Robots.txt tester
Enter a page address and pick a crawler. The tester reads the site’s live robots.txt and shows whether that crawler may fetch the page, and which rule decides it.
FAQ
Robots.txt tester questions
Does robots.txt stop a page from appearing in Google?
Not reliably. robots.txt controls crawling, not indexing. Google can still index a blocked URL without reading it if other pages link to it, usually with no description. To keep a page out of search results, let Google crawl it and add a noindex tag instead.
How does Google decide between Allow and Disallow rules?
The most specific rule wins, meaning the longest matching path. If an Allow and a Disallow rule are equally long, Allow wins. Google also supports * as a wildcard and $ to mark the end of a URL, and this tester follows the same logic.
Which group of rules applies to a crawler?
A crawler follows the group that names it (for example User-agent: Googlebot) and ignores the rest. Only if no group names it does it follow the User-agent: * group. That’s why adding a Googlebot group can unexpectedly unblock or block pages.
Do AI crawlers like GPTBot and ClaudeBot follow robots.txt?
OpenAI and Anthropic say their crawlers respect robots.txt, using the tokens GPTBot and ClaudeBot. Blocking them stops future crawling for training and search features; it doesn’t remove content they collected earlier.
Does Google follow Crawl-delay?
No. Googlebot ignores the Crawl-delay line (Bing and some other crawlers use it). Google adjusts its crawl rate automatically based on how quickly your server responds.
Check your whole site
The free site audit crawls every page and lists what stops them ranking: broken links, redirect chains, missing tags, blocked pages, with fixes.