Robots.txt Generator
Build a robots.txt file visually to control crawler access.
Build a robots.txt File
robots.txt is a plain-text file at the root of your domain that tells search-engine crawlers which parts of your site they may or may not request. It is the first file most crawlers fetch before exploring a site.
Key directives
- User-agent â which crawler the rules apply to
- Disallow â paths crawlers should not request
- Allow â exceptions within a disallowed path
- Sitemap â where to find your XML sitemap
Important caveat
robots.txt controls crawling, not indexing. A page blocked here can still appear in search results if other sites link to it. To keep a page out of the index, use a noindex meta tag instead.
Where it goes
The file must be served at https://yourdomain.com/robots.txt â crawlers only check that exact location. Test changes carefully: a stray Disallow: / will tell every crawler to ignore your entire site.
Private & Processed Locally
This generator runs entirely in your browser. Nothing you type is uploaded to a server, so it is safe for staging URLs, internal paths, and unreleased content, and it keeps working even when you are offline.
It is free to use with no sign-up and no limits â copy or download the output and drop it straight into your site.
Frequently Asked Questions
Common questions about the Robots.txt Generator.