Robots.txt Generator
Create a robots.txt file.
Loading tool…
About Robots.txt Generator
The robots.txt Generator builds a valid robots.txt file for your website in seconds. Pick a preset to allow all crawlers or block them entirely, or switch to custom mode to set a user-agent, add Disallow and Allow paths, an optional crawl-delay, and your sitemap URL. The output updates live and is ready to copy or download.
It is completely free with no sign-up, and it runs entirely in your browser. Nothing you type is uploaded or stored on a server, so your paths and site structure stay private on your own device.
Frequently asked questions
- Is the robots.txt generator free?
- Yes, it is 100% free with no account, sign-up, or usage limits. Generate as many robots.txt files as you like.
- Is my data uploaded anywhere?
- No. The tool runs entirely in your browser using JavaScript. The paths, user-agents, and sitemap URL you enter are never sent to or stored on any server, so everything stays private on your device.
- Where do I put the robots.txt file?
- Place it in the root directory of your domain so it is reachable at https://yoursite.com/robots.txt. Crawlers only look for it at that exact root location, not in subfolders.
- What is the difference between Disallow and Allow?
- Disallow tells crawlers not to crawl a path, while Allow explicitly permits one. Allow is mainly useful for carving an exception out of a broader Disallow, for example blocking /private/ but allowing /private/public.html.
- Does robots.txt keep my pages private or secure?
- No. robots.txt is a public, advisory file. Well-behaved search engines follow it, but it does not block access and malicious bots can ignore it. Use authentication or noindex for real privacy, not robots.txt.
- Does Google respect the crawl-delay directive?
- Googlebot ignores crawl-delay; you control Google's crawl rate in Search Console instead. Other crawlers such as Bing and Yandex may honor it, so it is still worth including if you need to throttle them.