How to Use
- 1.
Choose a preset such as allow all, block all, or block private paths.
- 2.
Enter the user-agent you want the rules to apply to, usually * for all crawlers.
- 3.
Add one allowed path per line if you need explicit Allow rules.
- 4.
Add one blocked path per line for admin, dashboard, private, or low-value sections.
- 5.
Add your sitemap URL and optional crawl-delay if your platform requires it.
- 6.
Copy or download the robots.txt file and upload it to the root of your domain.
Features
Allow all, block all, and block private path presets
Custom user-agent field
Multiple Allow and Disallow directives
Optional Crawl-delay directive
Sitemap URL directive
Live robots.txt preview
Copy robots.txt to clipboard
Download ready robots.txt file
SEO guidance and safe usage notes
Frequently Asked Questions
What is a robots.txt file?+
Robots.txt is a plain text file that gives crawler instructions about which parts of a site should or should not be crawled.
Where should I upload robots.txt?+
Upload it to the root of your domain, for example https://example.com/robots.txt. It should not be inside a subfolder.
Can robots.txt hide private pages?+
No. Robots.txt is not security. Do not use it to protect private data, admin areas, or confidential files.
Should I include my sitemap in robots.txt?+
Yes, adding a Sitemap line can help crawlers discover your XML sitemap location more easily.
What does Disallow: / mean?+
It asks crawlers not to crawl the entire site. Use it carefully because it can prevent search engines from crawling important pages.
Can robots.txt remove indexed pages from Google?+
Not reliably. To remove or prevent indexing, use noindex where appropriate, improve canonical handling, or use Search Console removal tools for urgent cases.
About this tool
Free Robots.txt Generator
Use the free WebToolsEdge Robots.txt Generator to create a clean robots.txt file for your website. Choose a preset, enter user-agent rules, add allowed and blocked paths, include your sitemap URL, and preview the final file before copying or downloading it. A robots.txt file sits at the root of a domain, such as example.com/robots.txt, and gives search engine crawlers instructions about which areas of a site they may crawl. It is commonly used to keep admin paths, account pages, internal search results, staging folders, and other low-value URLs out of regular crawling while keeping public pages accessible. This generator supports common SEO workflows: allow all crawlers, block all crawling, block private paths while allowing public pages, or write custom rules for a specific user-agent. You can add multiple Allow and Disallow lines, include a crawl-delay directive when appropriate, and add a sitemap directive so crawlers can discover your XML sitemap faster. Robots.txt can improve crawl management, but it is not a security feature. A disallowed URL can still be visible to anyone who knows the address, and search engines may still index a URL if other pages link to it. Do not use robots.txt to protect private files, customer data, admin panels, or sensitive documents. Use authentication, noindex controls where appropriate, and server-level access protection for sensitive content. For SEO, be careful not to block important pages, CSS files, JavaScript files, images, or sections needed for rendering. After uploading the file to your site root, test it in Google Search Console and confirm that important pages remain crawlable and indexable.
After creating crawl rules, review your page-level SEO with the SEO Analyzer. If you are preparing new pages for search, draft titles and descriptions with the SEO Generator, or browse all SEO Tools.