# robots.txt & Sitemap Validator

Validate a site's robots.txt and XML sitemaps, see which search and AI crawlers it allows, and test any URL against its rules.

Canonical: https://tamperbox.dev/robots-txt-checker

Reviewed: 2026-10-16

## Capabilities and processing

Enter a website, or paste a robots.txt file or an XML sitemap. For a website, our server fetches its robots.txt and up to six sitemaps it lists (the usual `/sitemap.xml` when it lists none). Pasted files are read in your browser.

robots.txt is read the way RFC 9309 and Google describe: groups by user agent, the most specific group for each crawler, and the longest matching rule, with Allow winning a tie. Test any path for any crawler and the deciding line is marked in the file. Search engines and AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and others) are shown as allowed, blocked or partly blocked at the home page. Sitemaps are checked for the 50,000 URL and 50 MB limits, full URLs, valid lastmod dates and duplicates, and sitemap URLs that robots.txt blocks are flagged.

## Example workflow

The sample shop blocks GPTBot and CCBot, keeps Google-Extended out of its trade pages, and lists a product URL in its sitemap that its own rules close to crawlers. Break it blocks everything, misspells Disallow, puts a rule above every User-agent and adds a Noindex line, each built from your file when you entered one, and shows a server error, a file served as a web page and a missing file.

## Limits and interpretation

robots.txt is a request, not access control; crawlers choose whether to obey, and Crawl-delay and Noindex lines are explained, not followed. At most six sitemaps are read, and an index lists at most five that are checked. Nothing is stored.

## Related tools

[llms.txt checker and generator](https://tamperbox.dev/llms-txt) · [Open Graph preview](https://tamperbox.dev/open-graph-preview) · [Broken link checker](https://tamperbox.dev/broken-link-checker) · [All tools](https://tamperbox.dev/) · [Privacy](https://tamperbox.dev/privacy)