Robots.txt Generator

Build compliant robots.txt files with custom user-agents, disallow rules, and sitemap directives.

User-Agent(s)

Sitemap URL

Disallow Paths (One per line)

Allow Paths (One per line)

output: robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/
Allow: /public/
Allow: /static/

Sitemap: https://example.com/sitemap.xml

How to Use Robots.txt Generator

1Step 1

Select target search bots (All Bots, Googlebot, Bingbot, AI Scrapers).

2Step 2

Specify paths to disallow (e.g. `/admin/`, `/private/`) or allow.

3Step 3

Add your XML sitemap URL and download or copy the valid `robots.txt` output.

It must always be placed at the absolute root of your domain (e.g. `https://yourdomain.com/robots.txt`).