Build a clean robots.txt in seconds — everything runs in your browser, nothing is uploaded.
Click "Generate robots.txt" to build your file.
Robots.txt is a small text file at the root of your site that tells search engine crawlers which parts they may and may not crawl. It does not hide pages from search results on its own, and it does not block determined scrapers — it is a request that well-behaved crawlers honour, not a security wall. Use it to keep crawlers out of admin areas, internal search results and staging folders, and to point them at your sitemap.
Some publishers block AI training crawlers such as GPTBot and ClaudeBot because they do not want their content used for model training; others allow them for the visibility that AI answers can bring. There is no universally right answer — it depends on your content and business model. This generator lets you add one explicit rule for the main AI crawlers either way, instead of editing the file by hand. If you are unsure, leaving AI crawlers out (the default) means your existing search engine rules apply and no separate AI rule is declared. Our guide explains the full picture: free robots.txt generator — what robots.txt does and whether to block AI crawlers.
Copy the output or download it, name it exactly robots.txt, and upload it to your site's root folder so it is reachable at yourdomain.com/robots.txt. Then test it in Google Search Console's robots.txt tester to confirm the rules behave as you intended before relying on them.