1
Step 1
Select whether to allow all crawlers, block all, or add custom disallow paths.
Robots.txt Generator Online
Use NewTools to create robots.txt rules directly in your browser. Generates allow, block, custom disallow, and sitemap rules for a basic robots.txt file.
Fast SEO Robots.txt Setup. Instantly create a standard robots.txt file to guide search engine crawlers.
No installation
Fast processing
Browser friendly
Privacy-aware
Features
Crawler Directives: Configure rules to allow all crawlers, block all, or disallow specific custom paths.
Sitemap Integration: Append an absolute URL to your XML sitemap for search engine discovery.
Instant Formatting: Automatically structures the User-agent and Disallow strings into the correct format.
Local Generation: The text file is generated instantly in your browser.
How It Works
2
Step 2
Optionally paste your absolute Sitemap URL (e.g., https://example.com/sitemap.xml).
3
Step 3
Generate and download your robots.txt file.
Common Use Cases
Basic Website SEO
Create a standard 'Allow: /' robots file for a new website.
Blocking Admin Pages
Use custom paths to prevent search engines from indexing /admin or /private directories.
Hiding Staging Environments
Generate a 'Disallow: /' file to keep staging sites out of search results.
Tips & Best Practices
Tip 1
Always upload the generated file to the exact root of your domain (e.g., example.com/robots.txt).
Tip 2
Ensure your custom disallow paths start with a forward slash (/).
Tip 3
Double-check that you haven't accidentally selected 'Block all' on a live production site.
Robots.txt Generator
Safe and Local Execution
This tool assembles standard SEO directives locally in your browser. No API calls are made to remote servers.
Frequently Asked Questions
Does this tool upload the file to my server?
No. This tool only generates the text configuration. You must download it and upload it to your website's root directory manually.
Does it check if my sitemap is valid?
No. It simply appends the provided sitemap URL to the configuration text.
Can I target specific bots?
This generator applies rules globally using 'User-agent: *'.