Robots.txt Generator
Create valid, crawler-compliant robots.txt files with crawl delay settings, sitemap directives, disallowed paths, and AI scraper bot blocking.
General Crawler Settings
Disallowed Directories / Paths
How to Use Robots.txt Generator
- Set Access Rules: Choose default access (Allow All or Disallow All) and optional crawl-delay interval.
- Add Disallowed Directories: List private URL paths (e.g. /admin/, /api/, /tmp/) that search bots should avoid indexing.
- Toggle AI Bot Protection: Enable AI training crawler blocking (GPTBot, CCBot, Google-Extended, Anthropic-AI) if desired.
- Copy or Download: Copy the code or click 'Download robots.txt' to place in your web server root.
Frequently Asked Questions
Where should the robots.txt file be located on my website?
The robots.txt file must always be placed at the root level of your domain (e.g. https://example.com/robots.txt).
Does robots.txt guarantee that private pages won't be seen by users?
No. Robots.txt is a voluntary protocol for search crawlers. For true security, sensitive administrative pages must be protected with authentication.
How do I block AI web crawlers from scraping my content?
Check the 'Block AI Training Crawlers' box to append explicit Disallow rules for GPTBot, CCBot, Google-Extended, Bytespider, and Anthropic-AI.
Should I include the Sitemap link in robots.txt?
Yes! Adding 'Sitemap: https://example.com/sitemap.xml' helps search engines discover your sitemap index automatically.
What does the Crawl-Delay directive do?
Crawl-delay specifies the number of seconds search engine spiders should wait between successive page requests to prevent server overload.
Is this generator free to use?
Yes, completely free with no usage limits.