Robots.txt Generator
Build a robots.txt file. Save the output at /robots.txt in your site root.
User-agent: *
Disallow: /admin
Disallow: /private
Sitemap: https://example.com/sitemap.xmlAbout the Robots.txt Generator
The Robots.txt Generator builds a syntactically correct robots.txt file for your site based on the choices you make — which directories to allow or disallow for all crawlers, per-bot overrides (Googlebot, Bingbot, GPTBot, ChatGPT-User), crawl-delay, and the Sitemap directive. Robots.txt lives at the root of your domain (`https://yoursite.com/robots.txt`) and is the first thing every search engine and crawler fetches.
A well-configured robots.txt keeps crawlers away from admin areas, staging paths, faceted search URLs and other pages you don't want indexed — while making sure your public content stays fully accessible. A misconfigured robots.txt can accidentally block your entire site from search engines (a `Disallow: /` under `User-agent: *` is a very common self-inflicted SEO wound).
In 2026, robots.txt has grown beyond just search engines — AI crawlers (GPTBot, Google-Extended, ClaudeBot, Bytespider) also respect it. Blocking these prevents your site's content from being used to train large language models, which many publishers now choose to do.
How to use this tool
- 1Enter the required value in the input field above (domain, IP, URL, or text depending on the tool).
- 2Click the action button to run the check — results are computed instantly from our edge network.
- 3Review the parsed output, key fields and any warnings shown in the result card.
- 4Copy the result, share the page URL, or jump to a related tool from the sidebar to continue debugging.
Key features
- Per-bot rules for search engines and AI crawlers
- Sitemap directive included
- Common templates for WordPress, Shopify and custom sites
- Copy-paste ready output