Robots.txt Generator
Create a robots.txt file.
Create a robots.txt file.
How to use Robots.txt Generator
- Choose an allow-all, block-all, or custom mode
- Set user-agent, Disallow and Allow paths, crawl-delay, and sitemap
- Copy or download the generated robots.txt file
About Robots.txt Generator
The Robots.txt Generator builds a valid robots.txt file for any website in seconds. Start from a preset to allow every crawler or block them all, then switch to custom mode when you need finer control: set a specific user-agent such as Googlebot, add Disallow rules for the paths you want kept out of search, add Allow rules to carve out exceptions, set an optional crawl-delay, and drop in your sitemap URL. The file updates live as you type, so you can see exactly what each rule produces before you copy or download it.
Because the output is a standard robots.txt file, it works wherever your site lives. Upload it to the root of a WordPress install, paste it into Blogger's custom robots.txt setting, or drop it on any host — the syntax is identical everywhere. It is a practical starting point for basic SEO housekeeping: pointing crawlers at your sitemap, keeping admin and duplicate URLs out of the index, and telling Google and Bing which sections to skip.
The tool is completely free with no sign-up and no limits, and it runs entirely in your browser. Nothing you type is uploaded or stored on a server, so your paths, user-agents, and site structure stay private on your own device.
Frequently asked questions
- Is the robots.txt generator free?
- Yes, it is 100% free with no account, sign-up, or usage limits. Generate as many robots.txt files as you like.
- Can I create a custom robots.txt with my own rules?
- Yes. Custom mode lets you set the user-agent, list as many Disallow and Allow paths as you need, add a crawl-delay, and include your sitemap. The output reflects your exact rules, so you are not locked into a fixed template.
- How do I make a robots.txt file for WordPress?
- Generate the rules here, then either upload the file to the root folder of your WordPress site by FTP so it loads at yoursite.com/robots.txt, or paste the content into an SEO plugin such as Yoast or Rank Math that lets you edit robots.txt from the dashboard. A common WordPress setup disallows /wp-admin/ while allowing /wp-admin/admin-ajax.php.
- How do I set up a custom robots.txt for Blogger?
- Blogger uses a custom robots.txt setting rather than a file you upload. Build your rules here, then in Blogger open Settings, enable 'Custom robots.txt', and paste the generated content into the box. Include your blog's sitemap URL so Google can find every post.
- How do I block or allow Googlebot specifically?
- Switch to custom mode and set the user-agent to Googlebot, then add the Allow or Disallow rules you want to apply only to Google's crawler. Leaving the user-agent as * applies your rules to every crawler at once instead.
- How do I add my sitemap to robots.txt?
- Enter the full URL of your sitemap in the sitemap field and the generator writes a Sitemap: line into the file. Crawlers read that line to discover your sitemap.xml, whether it lives at the standard https://yoursite.com/sitemap.xml or somewhere else.
- Does a robots.txt file help SEO?
- Indirectly. robots.txt does not boost rankings by itself, but it helps crawlers spend their budget on your important pages by keeping them out of admin areas, filtered URLs, and duplicate content, and it points them to your sitemap. Used carelessly it can also hide pages you wanted indexed, so double-check your Disallow rules.
- Where do I put the robots.txt file?
- Place it in the root directory of your domain so it is reachable at https://yoursite.com/robots.txt. Crawlers only look for it at that exact root location, not in subfolders.
- What is the difference between Disallow and Allow?
- Disallow tells crawlers not to crawl a path, while Allow explicitly permits one. Allow is mainly useful for carving an exception out of a broader Disallow, for example blocking /private/ but allowing /private/public.html.
- Does robots.txt keep my pages private or secure?
- No. robots.txt is a public, advisory file. Well-behaved search engines follow it, but it does not block access and malicious bots can ignore it. Use authentication or noindex for real privacy, not robots.txt.
- Does Google respect the crawl-delay directive?
- Googlebot ignores crawl-delay; you control Google's crawl rate in Search Console instead. Other crawlers such as Bing and Yandex may honor it, so it is still worth including if you need to throttle them.
- Is my data uploaded anywhere?
- No. The tool runs entirely in your browser using JavaScript. The paths, user-agents, and sitemap URL you enter are never sent to or stored on any server, so everything stays private on your device.