Robots.txt Generator Online
Create search engine crawler control rules.
Runs entirely in your browser
Rate Robots.txt Generator4.9(172 votes)Rate Robots.txt Generator
How to use Robots.txt Generator
- 1Add rules visually: select the user-agent (Googlebot, all crawlers, etc.) and whether to allow or disallow each path.
- 2Add your sitemap URL in the designated field.
- 3Copy the generated robots.txt content and upload it to your website's root directory.
Privacy & security
robots.txt generation runs entirely in your browser. Your configuration is never submitted to any server.
robots.txt generation runs locally. No URL or configuration data is transmitted.
Frequently asked questions
Where does robots.txt go?
robots.txt must be at the root of your domain — exactly at https://yourdomain.com/robots.txt. It cannot be in a subdirectory.
Can I block specific crawlers?
Yes — specify the user-agent by name (e.g. User-agent: Googlebot) to apply rules only to that crawler. Use User-agent: * to apply rules to all crawlers.
Does robots.txt prevent pages from being indexed?
Disallowing a page prevents it from being crawled, but not necessarily indexed if other sites link to it. To prevent indexing, use a noindex meta tag.