Robots.txt Generator
Robots.txt Generator
Build a robots.txt file: choose a user-agent, add Allow and Disallow rules, add your sitemap, then download.
A Robots.txt Generator creates the small instruction file that tells search engine crawlers which parts of a website they may visit and which they should leave alone. The file sits at the root of a domain, at yoursite.com/robots.txt, and every well-behaved bot reads it before crawling. Site owners use it to keep admin areas, search result pages, duplicate print versions, and private folders out of the crawl, which focuses a search engine's attention and crawl budget on the pages that actually matter for visitors and rankings in search results.
This generator keeps the format simple and correct. You choose whether the rules apply to all bots or to a specific crawler such as Googlebot, then add as many Allow and Disallow rows as you need, one path each. An optional sitemap line points crawlers straight to your XML sitemap so new pages are discovered faster. The finished file can be copied or downloaded ready to upload.
How to use
- Choose the user-agent your rules apply to, usually All bots for a general file.
- Edit the starter rows or remove them, then add rows for each path you want to block or allow.
- Use paths starting with a slash, such as /admin/ or /private/, exactly as they appear in your addresses.
- Paste your full sitemap address if you have one, so crawlers find your pages quickly.
- Generate the file, download it as robots.txt, and place it at the root of your domain.
Frequently asked questions
Does robots.txt hide a page from Google completely?
Not by itself. A blocked page can still appear in results as a bare link if other sites link to it, though its content will not be crawled. To keep a page out of results entirely, use a noindex instruction on the page itself. Robots.txt controls crawling, while noindex controls listing, and the two are often confused.
What does Disallow with an empty path or just a slash mean?
Disallow: / blocks the entire site for that bot, which is occasionally used on a development copy but is dangerous on a live site because search engines will stop crawling everything. An empty Disallow, written as Disallow: with nothing after it, means everything is allowed. Check your slashes carefully before uploading any generated file.
Can a bad bot ignore my robots.txt?
Yes. The file works on trust, and reputable search engines honour it, but scrapers and malicious bots may ignore it completely. That is why it must never be your only protection for private material. Keep real secrets behind logins, and treat robots.txt purely as traffic direction for cooperative crawlers.
Where do I test the file after uploading?
Open yourdomain.com/robots.txt in a browser to confirm it loads and reads correctly, then use the testing tools inside Google Search Console, which report syntax warnings and let you check whether a specific address is blocked. Recheck after every edit, because one misplaced line can block more than intended.