Robots.txt Generator

Write a correct robots.txt in seconds: allow everything, block everything, block AI crawlers, or fine-tune paths yourself.

Sitemap URL

Crawl delay in seconds (blank = omit)

Rules

Allow all crawlers

Block all crawlers

Block AI crawlers

Extra Disallow paths (one per line)

Extra Allow paths (one per line)

robots.txt

How it works

Presets cover the common cases; anything you type in the extra fields is appended verbatim under the right group.
The AI-crawler preset blocks GPTBot, CCBot, ClaudeBot, Google-Extended, PerplexityBot, Bytespider and Amazonbot with a named User-agent group each.
Robots.txt is only a request — crawlers that ignore it still need a server-side block.