Controlling AI crawlers: a practical Technical SEO guide

Controlling AI crawlers

Your content trains the next generation of AI models. Right now, OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's bot, and a dozen others are crawling your site to build their datasets. The question isn't whether they're visiting—they are. The question is whether you want them there, and on what terms. Controlling AI crawlers through robots.txt and related directives is one of the few levers you have to decide who uses your content and for what purpose.

I've watched this play out across several platforms I've built. At first, it felt like a non-issue. Then we started seeing our content appear verbatim in AI-generated summaries, and our analytics showed zero traffic attribution. That's when I realized: we needed a deliberate policy, not just acceptance by default.

Know the bots—who's actually crawling you

Before you block anything, know who you're dealing with. The major AI training crawlers have distinct user agents, and each serves a different purpose. Understanding the landscape helps you make decisions that align with your business, not just react to headlines.

GPTBot is OpenAI's crawler, feeding into ChatGPT and GPT-4. It's aggressive and frequent. If you've noticed a bot hammering your servers with user agent "GPTBot," that's the one.

ClaudeBot belongs to Anthropic and powers Claude's training pipeline. It's similarly persistent and comes from Anthropic's IP ranges.

PerplexityBot crawls for Perplexity AI's answer engine. Perplexity has been more careful about attribution—they often link back to sources in their responses—but they still consume your content at scale.

CCBot is Common Crawl's bot, a non-profit that builds the massive dataset powering many open-source models. It's often benign, but it's worth knowing it exists.

Google-Extended is distinct. It's not about Google Search indexing (that's Googlebot, which you've dealt with for years). Google-Extended specifically controls whether your content can be used for Gemini model training. This is a newer distinction, and it matters.

You can see which bots visit your site by checking CloudFlare logs, your web server logs, or tools like Semrush and Ahrefs. Filter for user agents containing "bot" and look for the names above. Most sites with decent traffic will see all of them within a week.

robots.txt rules—allow or disallow AI crawlers individually

The primary mechanism for controlling AI crawlers remains the humble robots.txt file, placed at the root of your domain (e.g., yoursite.com/robots.txt). Respectful crawlers check this file before requesting pages. Not all AI crawlers respect it—some do, some don't—but the major ones (OpenAI, Anthropic, Perplexity, Google) generally do.

Here's how to block specific AI crawlers while keeping others (or search engines) active:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

This approach blocks all five major AI crawlers but allows every other bot (including Googlebot for traditional search indexing). The last rule (User-agent: *) acts as a catch-all default.

If you want to be more granular—blocking only specific sections—you can do that too. Maybe you block GPTBot from your blog but allow it to crawl your public API docs, because those docs benefit from wider distribution:

User-agent: GPTBot
Disallow: /blog/
Allow: /docs/

User-agent: *
Allow: /

Or you might block all AI crawlers except Perplexity, because Perplexity's model includes good source attribution:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

The key principle: robots.txt directives are evaluated top-to-bottom, and the first matching rule wins. So order matters. If you list a specific bot before the catch-all, its rules take precedence.

One caveat: robots.txt is advisory. It's not a legal barrier. Rogue crawlers and bad actors ignore it. But the major, legitimate crawlers respect it because they want to maintain goodwill with publishers. Blocking in robots.txt sends a clear, standardized signal: "Don't use this for training."

Google-Extended—understanding the training vs. indexing distinction

Google-Extended deserves its own section because it represents a shift in how search engines think about AI and content ownership.

For decades, robots.txt controlled whether Google could index your site for search results. Blocking Googlebot meant no Google Search ranking. Google-Extended is different. It controls whether your content can be used to train Google's generative AI models (like Gemini) separately from search indexing.

You can block Google-Extended while keeping Googlebot active. This means your pages still appear in Google Search results—you get the SEO benefit—but your content doesn't train Gemini. Here's how:

User-agent: Google-Extended
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: *
Allow: /

This is powerful because it decouples two decisions that were previously bundled together. Before Google-Extended, you had to choose: either let Google index you for search (and implicitly let them use you for AI training), or block both.

From my perspective building platforms, I see the appeal of this trade-off for some companies. Your site gets search traffic—Googlebot still crawls you. But your proprietary content, your unique research, your customer stories—those don't become training data for Gemini. Google respects this directive in their documentation, though like all robots.txt rules, it's based on trust.

One gotcha: Google-Extended might still crawl you if you haven't explicitly blocked it, but Googlebot will. So if you want to be clear, add the rule. If you're unsure about the impact, test it: monitor your Google Search Console before and after adding the Google-Extended rule to see if there are any drops in search visibility (there usually aren't, since Googlebot continues to index).

Trade-offs—blocking may reduce citations in AI answers

Blocking AI crawlers isn't free. There are real trade-offs you should consider before implementing a blanket ban.

When you block PerplexityBot, your content stops appearing in Perplexity's answers. That means no citations linking back to your site from Perplexity's users. Same with GPTBot and ChatGPT—blocked, and you lose the chance for ChatGPT users to see your content cited (though admittedly, ChatGPT doesn't always link anyway).

For content-driven businesses like blogs, journalism sites, and technical publications, that loss can hurt. I worked on a SaaS product where our technical blog was a major SEO asset. We considered blocking AI crawlers but realized that citations in ChatGPT's responses—"According to [company] blog"—drove curious users to our domain. We got better ROI from a few ChatGPT citations than from some organic keywords. So we let the crawlers in.

The decision depends on your business model:

• High-volume content sites (news, blogs, tutorials): Citations in AI answers can drive traffic. You might allow crawlers, especially Perplexity (which cites sources consistently).
• Proprietary research or competitive analysis: You might block all AI crawlers to prevent your analysis from being summarized for free and offered to competitors.
• Paywall or subscription content: Blocking AI crawlers protects your monetization. If an AI summarizes your paywalled content, users don't need to subscribe.
• API documentation or technical resources: You might allow crawlers here; wider distribution of your docs improves adoption.

I've seen teams take a hybrid approach: allow Googlebot and Googlebot-Extended (to keep search visibility and Gemini availability), block GPTBot and ClaudeBot (to prevent broader AI training), but allow PerplexityBot (because Perplexity's citations drive traffic). That's a deliberate choice, not a default.

Checking and testing your robots.txt

After you update your robots.txt, verify it works. Google Search Console has a robots.txt tester (under Index → Inspect). You can test specific URLs and user agents to confirm your rules are parsed correctly.

For AI crawlers, testing is trickier because you can't easily see real-time results. But you can monitor your logs. Use your web server logs or a tool like CloudFlare analytics to see whether requests from GPTBot, ClaudeBot, etc. decrease after you deploy the rules. They should drop significantly within a week.

One more thing: make sure your robots.txt is actually being served. I've seen teams update the file on disk, but the web server wasn't configured to serve it at the right path, or caching was hiding the new version. Test it directly by visiting yoursite.com/robots.txt in a browser and confirming your rules are there.

Common mistakes

Mistake 1: Thinking robots.txt is a security boundary. It's not. A determined attacker or scraper can ignore it. Use robots.txt for honest signals to cooperative crawlers. For real protection, use authentication, rate limiting, and CAPTCHAs.

Mistake 2: Blocking AI crawlers but not realizing some have already cached your site. If ClaudeBot crawled you yesterday and you block it today, Claude's training data still includes your content. robots.txt prevents future crawls, not past ones. If you need content removed from an AI model's training set, you may need to contact the company directly (OpenAI has a form for this).

Mistake 3: Assuming all AI crawlers respect robots.txt equally. Major companies respect it. Smaller or rogue crawlers might not. Also, some AI companies build their training data from Common Crawl snapshots, not live crawls. Blocking CCBot after Common Crawl has already processed you won't undo it.

Mistake 4: Blocking Google-Extended without understanding the implications. If you block Google-Extended but not Googlebot, your pages still rank in Google Search. Google-Extended only controls Gemini training. If you accidentally block Googlebot too, you lose search visibility—a much bigger loss than missing Gemini citations.

Mistake 5: Not revisiting your policy. AI landscape changes fast. New crawlers emerge, companies adjust their behavior, and your business needs evolve. Review your robots.txt rules every quarter. What made sense six months ago might not now.

Key takeaways

• Controlling AI crawlers starts with knowing who they are: GPTBot, ClaudeBot, PerplexityBot, CCBot, and Google-Extended. Each serves a different model and has different implications for your content.
• Use robots.txt to signal your intent. Block specific crawlers with User-agent: [BotName] and Disallow: /. Respect it, trust is maintained; ignore it, you risk reputation damage.
• Google-Extended is separate from Googlebot. You can block Gemini training while keeping search indexing active, decoupling two decisions that were previously bundled.
• Blocking AI crawlers has trade-offs. You reduce the chance of citations in AI answers, which can hurt traffic for content-driven businesses. Decide based on your business model, not just principle.
• Test your robots.txt after updating, and monitor logs to confirm AI crawlers are actually being blocked. Revisit your policy quarterly as the landscape changes.
• robots.txt is advisory, not a legal barrier. For real protection of sensitive content, use authentication and access controls. For removal from existing AI training sets, contact the company directly.