Robots.txt Reader
Paste a robots.txt file to inspect its crawler rules.
Paste your source on the left to see the analysis here.
Read & Understand robots.txt
Paste the contents of any robots.txt file and this reader parses it into clear groups â showing which crawlers each rule targets, what is allowed or disallowed, crawl-delay settings, and declared sitemaps. It is the quickest way to make sense of a file that controls how search engines crawl a site.
What it parses
- User-agent groups (including stacked agents)
- Allow and Disallow path rules
- Crawl-delay directives
- Sitemap declarations
Crawling, not indexing
Remember that robots.txt only asks crawlers not to fetch a path â it does not remove a page from search results. Use a noindex meta tag to keep a page out of the index.
Private & Processed Locally
You paste the file's contents, so parsing happens entirely in your browser. To grab a site's robots.txt, visit https://thatsite.com/robots.txt and copy the text.
Frequently Asked Questions
Common questions about the Robots.txt Reader.