Robots.txt Generator
Generate a robots.txt file for your website to control search engine crawlers. Add User-agent groups, Allow/Disallow rules, Crawl-delay, and Sitemap URLs. 100% browser-based, no signup.
Crawler-Grade Output
Outputs standard robots.txt syntax — User-agent groups, Allow/Disallow path rules, Crawl-delay, and Sitemap directives that Googlebot, Bingbot, and other crawlers understand natively.
100% Private
Everything is generated in your browser. No server calls, no signup, no upload. Your crawler rules and sitemap URLs never leave your device.
Live Preview
The output textarea updates instantly as you add rules, groups, or sitemaps. Copy to clipboard or download as robots.txt with a single click.
11+ Common Bots
Built-in suggestions for Googlebot, Googlebot-Image, Bingbot, DuckDuckBot, Slurp (Yahoo), Baiduspider, YandexBot, AhrefsBot, SemrushBot, GPTBot — plus free-text for any custom UA.
What is a robots.txt file?
robots.txt is a plain-text file placed at the root of your website (e.g. https://example.com/robots.txt) that tells web crawlers which pages they may or may not crawl. It is a public file — anyone can read it — and it is advisory, not a security boundary.
Is robots.txt a security control?
No. robots.txt is a polite request, not a security enforcement. Malicious crawlers can ignore it. Never use it to hide sensitive pages — protect those with authentication instead. If a URL is linked from anywhere public, search engines may still index the URL (without crawling the content).
What is the difference between Allow and Disallow?
`Allow:` explicitly permits a path, `Disallow:` blocks it. Path matching uses a simple pattern language: `*` matches any sequence, `$` anchors the end. For example, `Disallow: /*?q=` blocks any URL containing `?q=`.
What does Crawl-delay do?
Crawl-delay asks the crawler to wait N seconds between requests. It is honored by some crawlers (Bingbot, Yandex) but ignored by Googlebot. Set it sparingly — most sites do not need it.
Do I need a Sitemap line?
A `Sitemap:` line tells crawlers where your XML sitemap lives. It is optional but recommended — it helps crawlers discover all your pages faster. Each sitemap URL goes on its own `Sitemap:` line.
Is there a daily limit?
No. Generate as many robots.txt files as you want. The tool is 100% free, no signup, no API calls — everything happens in your browser.
Ready to use the Robots.txt Generator?
Free, no signup, no watermark. 100% browser-based. Your data never leaves your device.
Open the tool