Decide who gets your content. Toggle allow or block for each AI crawler — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more — then copy a ready-to-publish robots.txt. The fastest way to block AI crawlers while keeping Google & Bing indexing intact. 100% in your browser.
User-agent: * allowed so Googlebot, Bingbot etc. index your site (recommended).https://yourdomain.com/robots.txt. robots.txt is a voluntary standard — reputable AI crawlers honor it, but it is not a hard firewall.Until recently, robots.txt was a quiet file that told search engines which paths to skip. The rise of generative AI gave it a second, far more contentious job: deciding whether your content may be used to train and ground large language models. Publishers, documentation teams, e-commerce stores and individual bloggers are all now asking the same question — should ChatGPT, Claude, Gemini and Perplexity be allowed to ingest my work? This generator lets you answer that precisely, crawler by crawler, and emits a correct robots.txt you can publish in under a minute.
The mechanism is the same time-tested syntax, just with new user-agent strings. To block OpenAI's training crawler you write User-agent: GPTBot followed by Disallow: /. To allow it, you use Allow: / instead. The fiddly part is that there are now many AI agents, each with its own exact token — and a misspelled user-agent silently does nothing. The toggles above hold the canonical list so you do not have to memorize them.
No — and this is the most important thing to understand. The AI training crawlers above are distinct from the crawlers that build search indexes. Googlebot indexes pages for Google Search; Google-Extended only governs Gemini training. Blocking Google-Extended, GPTBot, ClaudeBot and CCBot therefore leaves Googlebot and Bingbot completely free to crawl and rank you. That is exactly why the tool keeps a default User-agent: * Allow: / rule for everything else — you opt out of AI training while preserving full organic search visibility. If you would rather be more visible to AI answer engines, do the opposite: allow the crawlers and publish a strong llms.txt to guide them.
The reputable, clearly identified crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and the other named agents above — publicly state that they honor robots.txt. It is, however, a voluntary standard: it is a posted sign, not a locked gate. Bad actors that ignore it exist, and stopping those requires server-level or WAF-level blocking by user-agent or IP. For the major model-builders, though, a correct robots.txt is the right, expected, and effective control — and it is the signal every reputable crawler looks for first.
Copy or download the generated file and upload it to the root of your domain so it is served at https://yourdomain.com/robots.txt. It must be at the root — a robots.txt in a subfolder is ignored. If you run multiple subdomains, each needs its own robots.txt. Add your sitemap URL in the field above to include the standard Sitemap: directive, which helps the search crawlers you do allow. Once it is live, complete the set with an llms.txt file for AI guidance, and use our token counter if you build LLM features yourself.
User-agent: GPTBot then Disallow: / to your robots.txt. This generator does it for you — toggle GPTBot to Block (or use "Block all AI"), copy the file, and upload it to your domain root.