Check robots.txt and sitemaps, and test any URL against the rules.
Example
Test a URL
robots.txt
Crawlers at the home page
Sitemaps
Findings
Capabilities and processing
Enter a website, or paste a robots.txt file or an XML sitemap. For a website, our server fetches its robots.txt and up to six sitemaps it lists (the usual /sitemap.xml when it lists none). Pasted files are read in your browser.
robots.txt is read the way RFC 9309 and Google describe: groups by user agent, the most specific group for each crawler, and the longest matching rule, with Allow winning a tie. Test any path for any crawler and the deciding line is marked in the file. Search engines and AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and others) are shown as allowed, blocked or partly blocked at the home page. Sitemaps are checked for the 50,000 URL and 50 MB limits, full URLs, valid lastmod dates and duplicates, and sitemap URLs that robots.txt blocks are flagged.
Example workflow
The sample shop blocks GPTBot and CCBot, keeps Google-Extended out of its trade pages, and lists a product URL in its sitemap that its own rules close to crawlers. Break it blocks everything, misspells Disallow, puts a rule above every User-agent and adds a Noindex line, each built from your file when you entered one, and shows a server error, a file served as a web page and a missing file.
Limits and interpretation
robots.txt is a request, not access control; crawlers choose whether to obey, and Crawl-delay and Noindex lines are explained, not followed. At most six sitemaps are read, and an index lists at most five that are checked. Nothing is stored.
Related tools
llms.txt checker and generator · Open Graph preview · Broken link checker · All tools · Privacy