Test crawler access before launch
Paste robots.txt rules and check whether a specific path is allowed or blocked for Googlebot, a custom crawler or the default wildcard user agent.
Free Tool
Robots.txt Tester for Smarter Website Decisions
Paste robots.txt rules and test crawler access for a path before publishing crawl directives.
Loading tool...
What It Checks
Focus Areas for This Tool
These are the signals I would review manually when deciding what needs fixing first.
01
Allow/block result
Review this area to understand whether it is helping or hurting growth.
02
User-agent testing
Review this area to understand whether it is helping or hurting growth.
03
Sitemap detection
Review this area to understand whether it is helping or hurting growth.
04
Crawl delay
Review this area to understand whether it is helping or hurting growth.
Optimization Guide
How This Tool Helps Before Publishing
A little context makes the numbers more useful. Use these notes to understand where this tool fits inside content, SEO and website reviews.
See the rule that wins
The tester reports the matched allow or disallow directive so migration QA and SEO reviews can focus on the rule creating the crawl decision.
Review sitemap and crawl-delay hints
Sitemap directives and crawl-delay values are surfaced alongside the access result, making copied robots files easier to inspect quickly.
Browser-only robots testing
Rules are parsed locally in your browser, so staging paths, private URL examples and unpublished crawl directives do not need to be uploaded.
FAQs
Frequently Asked Questions
Quick answers about how this tool works and when to use it.
It checks a pasted robots.txt file against a site URL, path and user agent, then reports whether the path is allowed or blocked.
Yes. Enter Googlebot, another crawler name or the wildcard user agent to test the matching robots.txt group.
No. Paste the rules you want to test. The checker runs locally and does not fetch or upload robots.txt content.
Yes. Sitemap lines and crawl-delay values are included in the report when they are present in the pasted rules.