Test crawler access before launch
Paste robots.txt rules and check whether a specific path is allowed or blocked for Googlebot, a custom crawler or the default wildcard user agent.
Robots.txt Tester
Test whether robots.txt allows or blocks a URL path.
Loading tool...
Paste robots.txt rules and test crawler access for a path before publishing crawl directives.
- Allow/block result
- User-agent testing
- Sitemap detection
- Crawl delay
See the rule that wins
The tester reports the matched allow or disallow directive so migration QA and SEO reviews can focus on the rule creating the crawl decision.
Review sitemap and crawl-delay hints
Sitemap directives and crawl-delay values are surfaced alongside the access result, making copied robots files easier to inspect quickly.
Browser-only robots testing
Rules are parsed locally in your browser, so staging paths, private URL examples and unpublished crawl directives do not need to be uploaded.
Frequently asked questions
It checks a pasted robots.txt file against a site URL, path and user agent, then reports whether the path is allowed or blocked.
Yes. Enter Googlebot, another crawler name or the wildcard user agent to test the matching robots.txt group.
No. Paste the rules you want to test. The checker runs locally and does not fetch or upload robots.txt content.
Yes. Sitemap lines and crawl-delay values are included in the report when they are present in the pasted rules.