Free Robots.txt Generator

Build a search-engine-ready robots.txt with the right disallow rules in 30 seconds — no AI, no signup.

Your site URL (for sitemap reference)

Adds Sitemap: yoursite.com/sitemap.xml at the bottom.

Add explicit Allow: /

Most sites want this. Default-allow + explicit disallow is the standard pattern.

Common paths to block

Custom paths to block

Block AI training crawlers (optional)

Each is a separate User-agent block — adds Disallow: / for that bot only.

OpenAI GPTBot GPTBot

Block if you don't want your content training ChatGPT.

CommonCrawl CCBot

Block if you don't want your content in CommonCrawl (used to train many LLMs).

Google AI training Google-Extended

Block to opt-out of Google Bard/Gemini training while keeping regular Google Search crawl.

Anthropic ClaudeBot anthropic-ai

Block if you don't want your content training Claude.

Your robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /login
Disallow: /dashboard/
Disallow: /checkout

How to install

1.Download or copy the content above.
2.Save as robots.txt at your site’s root — must be served at https://yoursite.com/robots.txt (not a subdirectory).
3.Verify by visiting https://yoursite.com/robots.txt in your browser — it should display as plain text.
4.Test with Google’s robots.txt Tester in Search Console.

Heads up: robots.txt is a request, not enforcement. Well-behaved crawlers (Google, Bing) respect it; bad actors ignore it. Don’t use robots.txt to hide secrets — use auth or noindex meta tags for that.

The full SEO surface

Run a free SEO audit

robots.txt is one of 14 technical SEO checks the audit runs. See whether yours is helping or hurting in 60 seconds.

Run audit