AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot and Your robots.txt

Key takeaways

AI crawlers from OpenAI, Anthropic and Perplexity index your site to power AI-generated answers.
A single misconfigured robots.txt line can silently block all of them and kill your GEO visibility.
Each crawler has a distinct purpose: training data, search results, or real-time user queries.
Allowing the right bots requires explicit Allow directives, server-side HTML and verified crawlability.
Google's Google-Extended token controls AI training use; it does not block AI Overviews.

AI crawlers from OpenAI, Anthropic and Perplexity decide whether your site can appear in AI answers. GPTBot, ClaudeBot and PerplexityBot each read your pages before ChatGPT, Claude or Perplexity can ever cite you. One misconfigured line in robots.txt can block all of them at once, and most site owners never notice. Here is what each crawler does, and how to configure robots.txt so AI can see you.

What are AI crawlers, and why should you care?

AI is the new search engine. The brands it cites get bought. The rest get ignored.

Behind every ChatGPT answer, every Perplexity citation, every Claude response that names a vendor, there is a crawler that visited a website first. These are AI crawlers: automated bots that read your pages, extract your content, and feed it into the systems that generate AI answers. If those crawlers cannot access your site, you do not exist in AI search.

This is not theoretical. Researchers Aggarwal et al. (Princeton, 2023) introduced the concept of Generative Engine Optimization (GEO), showing that the way content is structured and indexed by AI systems directly affects how often it surfaces in generative engine responses (arxiv.org/abs/2311.09735). Visibility in AI-generated answers is a measurable, optimizable outcome, and it starts with crawl access.

The good news: controlling which AI crawlers can access your site is straightforward. It lives in one file, robots.txt. The bad news: most sites have it wrong.

Meet the major AI crawlers: GPTBot, ClaudeBot, PerplexityBot and Google

Not all AI crawlers do the same thing. Here is the breakdown.

GPTBot: OpenAI's training and search crawler

GPTBot is OpenAI's primary web crawler. Its official purpose is to crawl content that may be used to train OpenAI's generative AI foundation models. OpenAI actually operates two distinct crawlers relevant to visibility:

GPTBot crawls for training data. Disallowing it signals your content should not be used to train OpenAI models.
OAI-SearchBot crawls specifically to surface websites in ChatGPT's search features. If you block OAI-SearchBot, your site will not appear in ChatGPT search answers.

The user-agent string for GPTBot looks like this:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Both bots are independent. You can allow one and block the other. OpenAI publishes its IP ranges and crawler details in its official docs. Source: OpenAI bots documentation.

ClaudeBot: Anthropic's training crawler

ClaudeBot is Anthropic's crawler for collecting web content that could contribute to training Claude models. Anthropic also runs two additional bots:

Claude-User is triggered when a Claude user asks a question that requires fetching a live web page. Blocking it reduces your visibility in user-directed Claude queries.
Claude-SearchBot indexes content to improve Claude's search result quality. Blocking it reduces your visibility in Claude search responses.

ClaudeBot respects robots.txt and supports the non-standard Crawl-delay directive. Source: Anthropic crawler documentation.

PerplexityBot: Perplexity's search crawler

PerplexityBot is designed to surface and link websites in Perplexity search results. Perplexity explicitly states it is not used to crawl content for AI foundation models: it is purely for search indexing. Perplexity also runs Perplexity-User, which fetches pages in real time when a user asks a question. Perplexity-User generally ignores robots.txt because the fetch is user-initiated.

The PerplexityBot user-agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Source: Perplexity documentation.

Google-Extended: Google's AI training token

This one confuses a lot of people. Google-Extended is not a separate crawler. It is a robots.txt policy token that tells Google whether content already crawled by Googlebot can be used for AI-related purposes, like training Gemini or grounding AI responses.

Crucially: blocking Google-Extended does not remove you from AI Overviews. AI Overviews pull from Google's existing search index, which is controlled by Googlebot. If you want to appear in AI Overviews, you need Googlebot to be able to crawl and index your pages normally. Source: Google crawler documentation.

How to check if AI crawlers can access your site

Before you configure anything, verify the current state of your robots.txt.

Step 1: Find your robots.txt file

Go to yourdomain.com/robots.txt. Every site has one, or should. If you get a 404, that is already a problem: most crawlers will default to assuming everything is allowed, but some behave differently.

Step 2: Look for these patterns

Scan for any of the following:

User-agent: * followed by Disallow: / blocks all bots, including every AI crawler.
User-agent: GPTBot with Disallow: / blocks OpenAI's training crawler.
User-agent: OAI-SearchBot with Disallow: / removes you from ChatGPT search results.
User-agent: ClaudeBot with Disallow: / blocks Anthropic's training crawler.
User-agent: PerplexityBot with Disallow: / removes you from Perplexity search results.

Step 3: Check your server logs

If you have access to server logs, search for the bot user-agent strings listed above. If you are not seeing any hits from GPTBot or PerplexityBot, either you are blocking them or your content is not being prioritized for crawling.

Step 4: Test with Google Search Console

Use the URL Inspection tool in Google Search Console to verify Googlebot can access your pages. For AI Overviews specifically, you need standard Googlebot access, not Google-Extended.

Step 5: Check your WAF

Web Application Firewalls (Cloudflare, AWS WAF and similar) can block bots at the network level, before robots.txt is even read. If your WAF is set to block unknown bots or rate-limit aggressively, AI crawlers may be silently rejected. Perplexity's official docs include explicit WAF configuration guides for Cloudflare and AWS for exactly this reason.

How to allow AI crawlers in robots.txt (with a real example)

Here is a production-ready robots.txt that explicitly allows all major AI crawlers while keeping control over training-data use. The configuration, block by block:

Allow all standard crawlers: User-agent: * with Allow: / and a Disallow for your private paths such as /admin/ and /private/.
OpenAI: allow both OAI-SearchBot and GPTBot with Allow: / to keep ChatGPT search and training access open.
Anthropic: allow ClaudeBot, Claude-User and Claude-SearchBot, each with Allow: /.
Perplexity: allow PerplexityBot with Allow: / for search indexing.
Google: allow Googlebot with Allow: /, and optionally set Google-Extended to Disallow: / if you want to keep your content out of Gemini training without affecting AI Overviews.
Finish with your Sitemap: line pointing at your sitemap.xml.

A few notes on this configuration:

The Google-Extended disallow is optional. If you are fine with Google using your content for Gemini training, remove that block.
Each bot directive is independent. Allowing OAI-SearchBot and blocking GPTBot is a valid strategy if you want ChatGPT search visibility without contributing to training data.
Order matters within a user-agent block, but not between blocks. Each User-agent section is evaluated independently.

Why blocking AI crawlers kills your GEO visibility

Here is the uncomfortable truth: a lot of sites accidentally block AI crawlers. The most common culprit is a legacy robots.txt that was set up to block scrapers and now catches every AI bot in the process. A blanket Disallow: / under User-agent: * is the single most damaging line you can have in your file right now.

My take, from building Howseen: the sites bleeding the most AI visibility almost never blocked crawlers on purpose. It is a stale robots.txt line, copied from some old scraper-defense setup, that nobody has reopened in two years. Before you touch anything else, go read that one file line by line.

Why does it matter so much?

AI answers are citation-based. When ChatGPT, Claude or Perplexity answer a question, they cite sources. Those sources are pages that were crawled, indexed and deemed relevant. If your page was never crawled, it cannot be cited. If it cannot be cited, your brand does not exist in that answer.

GEO is the new SEO. The research from Aggarwal et al. (arxiv.org/abs/2311.09735) showed that optimizing content for generative engines, including ensuring crawl access, can measurably improve how often a source appears in AI-generated responses. Crawl access is the floor. Without it, nothing else matters. Once the crawlers are in, structure, schema and off-page citations are what move you up the answer, the full method is in our GEO playbook.

The stakes are asymmetric. Allowing AI crawlers costs you nothing. Blocking them costs you every AI-driven referral, every brand mention in a ChatGPT answer, every Perplexity citation that could have driven a click. Get this wrong and AI recommends your competitors instead of you, silently.

The one legitimate reason to block: if you have proprietary content you do not want used for model training. In that case, block GPTBot and ClaudeBot specifically, but keep OAI-SearchBot, Claude-SearchBot and PerplexityBot open so you still appear in AI search results.

Server-side HTML: why it matters for AI crawlers

Fixing robots.txt is step one. Step two is making sure AI crawlers can actually read your content once they get in. Most AI crawlers do not execute JavaScript. They fetch the raw HTML response from your server and parse what is there. If your site renders content client-side, meaning the page arrives as an empty shell and JavaScript fills it in, AI crawlers see a blank page.

This is what server-side rendering (SSR) solves. With server-side HTML, the full page content, your headings, your body copy, your structured data, is present in the initial HTTP response. The crawler does not need to run JavaScript. It reads the HTML directly.

Does this affect you?

If your site is built with a JavaScript framework (React, Vue, Angular, Next.js in client-only mode), check your rendered output. The quickest test: open your page in a browser, right-click and select View Page Source (not Inspect). If the source shows mostly empty div tags and script references, you have a client-side rendering problem. The fix depends on your stack:

Next.js: use server components in the App Router, or getServerSideProps in the Pages Router.
Nuxt: enable SSR mode.
React SPA: consider a pre-rendering service or static site generation for key pages.

Googlebot has invested years in JavaScript rendering. Most AI crawlers have not. GPTBot, ClaudeBot and PerplexityBot are primarily HTML parsers. If your content is not in the initial HTML response, it is invisible to them, regardless of what your robots.txt says. Server-side HTML is non-negotiable for AI visibility.

How to verify AI crawler access is actually working

Configuring robots.txt and enabling server-side rendering is only half the job. Verification closes the loop.

Check your server access logs

Filter for the bot user-agent strings: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. If you are seeing regular hits from these bots on your key pages, you are in good shape. If you see nothing, something is blocking them: robots.txt, a WAF rule, or a CDN configuration.

Fetch your page as a bot

Use curl with a spoofed user-agent to simulate what a crawler sees:

curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://yourdomain.com/your-page

If the response contains your full page content, you are good. If it returns a 403, a CAPTCHA page or an empty body, something is blocking the bot at the server or CDN level.

Monitor AI citation tracking tools

The most direct signal is whether your brand actually appears in AI answers. Tools that track AI visibility, monitoring how often your site is cited across ChatGPT, Perplexity, Claude and Google AI Overviews, will show you whether your crawl configuration is translating into actual GEO visibility. That is exactly what Howseen AI tracks. Prefer to start by hand first? Here is how to check if ChatGPT mentions your brand. Get your AI visibility score.

Does blocking GPTBot remove me from ChatGPT answers?

Not entirely, and it depends on which bot you block. GPTBot controls training data. OAI-SearchBot controls ChatGPT search results. Blocking GPTBot signals your content should not be used to train future models, but it does not necessarily remove you from ChatGPT's search feature. Blocking OAI-SearchBot does. To stay visible in ChatGPT answers, keep OAI-SearchBot allowed.

Will AI crawlers respect my robots.txt?

The major ones, GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot, all officially respect robots.txt. Anthropic, OpenAI and Perplexity each state this in their crawler documentation. That said, Perplexity-User, the real-time fetch triggered by a user query, generally ignores robots.txt because it is responding to a direct user request rather than crawling autonomously.

Does blocking Google-Extended stop me from appearing in AI Overviews?

No. Google-Extended is a policy token that controls whether your crawled content can be used for AI training purposes like Gemini. It does not affect Google's standard search index, and AI Overviews pull from that index. To appear in AI Overviews, you need Googlebot to crawl and index your pages normally. Blocking Google-Extended has no effect on that.

Do I need server-side rendering for AI crawlers if my site uses React?

Yes, if your content is rendered client-side. Most AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, are HTML parsers that do not execute JavaScript. If your page content only appears after JavaScript runs, those crawlers see an empty page. Server-side rendering or static site generation ensures your content is present in the initial HTML response, which is what AI crawlers actually read.

How often do AI crawlers visit my site?

There is no fixed schedule published by any of the major AI companies. Crawl frequency depends on your site's authority, how often your content changes, and how the crawler prioritizes its queue. What you can control is making sure crawlers are not blocked when they do visit. Anthropic supports the non-standard Crawl-delay directive if you want to throttle ClaudeBot's visit rate without blocking it entirely.

Start tracking what AI crawlers actually see

Fixing robots.txt and enabling server-side HTML gets you into the game. But you need to know whether it is working. Are you being cited in ChatGPT answers? Does Perplexity link to your pages? Does Claude mention your brand when users ask about your category?

Howseen AI tracks your visibility across every major AI engine, ChatGPT, Perplexity, Claude and Google AI Overviews, and shows you exactly where you appear, where competitors are being cited instead, and what to fix. Track your AI visibility with Howseen.