SEO•Client-Side Processing
Robots.txt Generator
Create robots.txt files with crawl rules for Googlebot, Bingbot, disallows, and sitemap URL.
Developer workbench
⌘ / Ctrl + Enter to runConfigure. Run. Inspect.
Edit the JSON configuration. Generated templates are starting points and do not claim to describe an existing system.
98 input bytes
Output
How to Use Robots.txt Generator
01
Set permissions for Googlebot, Bingbot, or all crawlers (*).
02
Add paths to disallow (e.g. /admin/, /api/).
03
Include your XML sitemap URL and download robots.txt.
Frequently Asked Questions
Where does robots.txt belong?
Robots.txt must be accessible at the exact root of your domain: https://example.com/robots.txt.
Related Developer Tools
Explore more free client-side developer utilities in this category