Tool #26All 80 Tools
Rule Validator100% Free & Client-SideReal-Time Computation
Robots.txt Tester
Test specific URL paths against robots.txt directives to verify whether search bots are allowed or blocked.
Crawler Access Verdict
BLOCKEDUser-Agent: Googlebot
Triggered Rule: Disallow: /admin/
How to Use Robots.txt Tester
1Step 1
Paste your active `robots.txt` file contents.
2Step 2
Enter a test URL subpath (e.g. `/checkout/success` or `/blog/seo-guide`).
3Step 3
Choose a user-agent to inspect the immediate ALLOWED or BLOCKED verdict.
Frequently Asked Questions
Q.Does a Disallow rule prevent a page from being indexed?
No! If external websites link to a disallowed URL, Google may still index the URL without crawling its contents. Use `noindex` if you want it de-indexed.