Most "block AI crawlers" advice treats every AI bot as the same thing. They are not, and the difference decides whether your site gets visitors from ChatGPT, Claude and Perplexity or quietly disappears from their answers. Some of these crawlers collect pages to train future models. Others fetch pages to answer a question someone is asking right now, and link to you when they do.

On our own site, roughly three quarters of the people using our tools in a recent week arrived from ChatGPT. Blocking the wrong user agent would have cut that off overnight. So before copying a list of bots into robots.txt, it is worth knowing what each one does.

The short answer: if you want to opt out of AI training but stay visible in AI search, block the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and leave the search and user-facing agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) allowed.

The crawlers, by what they actually do

Training crawlers collect content that may be used to train AI models. Blocking them does not affect whether AI products can find and cite you today.

  • GPTBot: OpenAI's training crawler.
  • ClaudeBot: Anthropic's training crawler.
  • Google-Extended: not a separate crawler but a robots.txt token. It controls whether content Googlebot already crawls can be used for Gemini models. It does not affect Google Search, and it does not remove you from AI Overviews, which are part of Search.
  • Applebot-Extended: the equivalent token for Apple's AI models. Regular Applebot powers Siri and Spotlight suggestions.
  • CCBot: Common Crawl, a public web archive that many AI datasets are built from.

Search crawlers build the indexes AI assistants search when they answer with links.

  • OAI-SearchBot: used for ChatGPT's search results. Block it and your pages stop appearing as ChatGPT search sources.
  • Claude-SearchBot: Anthropic's crawler for search in Claude.
  • PerplexityBot: Perplexity's index crawler.

User-initiated agents fetch a specific page because a person asked the assistant to look at it or follow a link.

  • ChatGPT-User, Claude-User and Perplexity-User. Several providers say these fetches act on behalf of a user and may not be governed by robots.txt in the same way as automated crawling, so check each provider's current documentation if you need certainty.

Company documentation changes, and new agents appear. Treat this list as current at the time of writing, and check the providers' crawler pages before you rely on a rule.

A robots.txt that blocks training and keeps AI search

# Opt out of AI model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else, including search and AI search crawlers
User-agent: *
Disallow: /admin/
Disallow: /cart

Sitemap: https://www.example.com/sitemap.xml

Two things to notice. First, the search agents are not listed at all, so they follow the * group, which is what you want. Second, a crawler only follows its own group if one exists. If you add User-agent: OAI-SearchBot with a single rule, it ignores everything in the * group. Repeat any rules it still needs, or simply leave it out.

Blocking everything AI, deliberately

Some publishers decide they want no AI use at all. That is a legitimate choice, but be clear about the trade-off: blocking the search and user agents removes you from AI answers that would otherwise link to you, and AI assistants are becoming a real source of visits for practical, question-shaped content. If your business depends on being recommended, measure that traffic before you cut it off. A first-touch source cookie or properly tagged analytics will tell you how much of it there is.

robots.txt is a request, not a wall

robots.txt only works for crawlers that choose to respect it. The major AI companies say their declared crawlers do, but some less reputable scrapers ignore it or disguise their user agent. If you need to actually stop a bot, that has to happen at your server, firewall or CDN, which can block by user agent and, more reliably, by verified IP ranges.

The reverse problem is just as common and much less discussed: your hosting provider or CDN may already be blocking AI crawlers you wanted to allow. Many CDNs now offer one-click "block AI bots" settings, and some hosts rate-limit unfamiliar crawlers by default. On our own site, a check showed OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot all being served normally while GPTBot received "429 Too Many Requests" from the host, despite nothing in robots.txt asking for that. If AI visibility matters to you, check the response your server actually gives each agent, not just what your robots.txt says.

How to check what you have

  1. Read your live robots.txt in the Robots.txt Checker, which lists each user-agent group and its rules.
  2. Confirm no group accidentally covers an agent you wanted to allow, and that no agent has its own group missing rules it needs from *.
  3. Check your CDN or host's bot settings for AI-specific blocking.
  4. If you are writing the file from scratch, the Robots.txt Generator produces a valid starting point you can add these groups to.

Find out which AI bots actually visit you

Rules are guesses until you look at real traffic. Your server's access logs record the user agent of every request, and most hosting control panels let you download them or view them in a log viewer. Search them for the names above, such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, and look at three things:

  • Which ones appear at all. Many small sites see only one or two AI crawlers regularly.
  • The response codes they receive. A column of 403 or 429 responses means something other than robots.txt, usually a firewall, CDN or host rule, is turning them away.
  • Which pages they request. User-initiated agents fetching specific articles is a good sign that people are asking AI assistants about your content.

Pair that with your analytics. Visits referred by chatgpt.com, perplexity.ai or similar show up as referral traffic, although some AI apps send visitors without a referrer, so they can also appear as direct. A link tagged with utm_source by the assistant, which ChatGPT commonly adds, is the most reliable signal. Knowing how much of your traffic arrives this way is what turns "should we block AI bots?" from a debate into a decision.

What about llms.txt?

llms.txt is a proposed convention: a plain-text file at your site root that gives language models a curated map of your most useful pages. It is not a standard, and no major provider has said it uses it for ranking. It is cheap to add and harmless, which is why we publish one, but do not expect it to change your visibility on its own. Allowing the right crawlers matters far more.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT search. OpenAI documents OAI-SearchBot as the crawler for search results and GPTBot as the training crawler. Blocking GPTBot opts you out of training; blocking OAI-SearchBot is what removes you from search answers.

Does blocking Google-Extended affect my Google rankings?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. It also does not stop your content appearing in AI Overviews, which are governed by regular Googlebot.

How quickly do changes take effect?

Crawlers re-read robots.txt periodically, usually within a day. Content already collected for training is not removed by a later block.

Short version

Separate training from search. Block the training crawlers if you want to opt out, keep the search and user agents allowed if you want AI assistants to recommend and link to you, and check that your host or CDN is not blocking crawlers behind your back. For how robots.txt groups and rules work in general, see what robots.txt is and how to check it.