Tag: GPTBot

When a buyer asks ChatGPT or Perplexity about your market, those tools rely on automated readers that have already visited your website, or tried to. AI crawlers do not behave like human visitors, and they do not behave like Googlebot either. Here is what they actually see, and the five gaps that hide many B2B sites from them.

What is an AI crawler? (And why there is more than one)

If you have spotted a name like ClaudeBot or GPTBot in your server logs and searched to find out what it was, you are in good company. Thousands of site owners do the same every month, and the volume of those searches has grown tenfold in a year. AI crawler traffic has become impossible to miss.

An AI crawler is an automated programme that requests and reads web pages on behalf of an AI platform. It arrives with no mouse, no patience and a strict time budget, takes the raw code your server returns, and moves on. What it collects ends up in one of two places, and that difference matters more than any other fact in this article.

Training crawlers read the web to build what an AI model knows, whereas retrieval crawlers, sometimes called search or user-fetch bots, collect pages to answer a live question from a real person as they ask a question. One company usually runs both. OpenAI’s own documentation describes four separate bots with three separate jobs: GPTBot gathers content for model training, OAI-SearchBot builds the index behind ChatGPT’s search results, ChatGPT-User fetches pages when a user asks about them directly, and OAI-AdsBot validates ads placed on ChatGPT (OpenAI, 2026). Blocking one typically has no effect on the others.

Here is who is likely visiting you:

Crawler	Run by	What it does
GPTBot	OpenAI	Model training
OAI-SearchBot	OpenAI	ChatGPT search index
ChatGPT-User	OpenAI	Live fetches for user requests
ClaudeBot	Anthropic	Model training
Claude-SearchBot	Anthropic	Claude search results
PerplexityBot	Perplexity	Search and answers
Google-Extended	Google	Robots.txt control for AI training use
CCBot	Common Crawl	Open dataset used to train many models
Bytespider	ByteDance	Model training

Most of this crawling is not about answering questions at all. Cloudflare’s analysis of crawler activity across its network found that around 80% of AI crawling in the year to mid-2025 was for model training, against 18% for search (Cloudflare, 2025). A bot visit is not the same thing as visibility.

Gap one: a robots.txt file written for a post Googlebot world

robots.txt is the small text file that tells crawlers what they may and may not read on your website. Anyone can check robots.txt for any site: type yoursite.com/robots.txt into a browser. Most B2B versions were written years ago, for a world where Googlebot was the only reader worth thinking about, and they fail in two quiet ways.

The first failure is blocking by accident. Broad disallow rules, old security plugins, or a firewall or CDN configured to swat away unfamiliar bots can shut out AI retrieval crawlers. The consequence is severe and invisible: OpenAI states plainly that sites opted out of OAI-SearchBot will not appear in ChatGPT’s search answers (OpenAI, 2026): no error message or warning, just absence.

The second failure is allowing by default. Training crawlers raise a genuine data-rights question: do you want your content absorbed into future AI models? There are sound reasons to say yes (models that have read your site describe your brand better) and sound reasons to say no (control over your intellectual property). The practical position for most B2B sites is to allow the retrieval bots if you want AI visibility, and decide whether to allow the training bots.

Gap two: JavaScript your buyers can see and AI cannot

Vercel, working with the search consultancy MERJ, analysed crawler behaviour across its hosting network and found that none of the major AI crawlers execute JavaScript; GPTBot fetched JavaScript files in around 11.5% of its requests, yet never ran them (Vercel, 2024). Google’s Gemini is the main exception, because it borrows Googlebot’s rendering machinery.

The consequence is strange to say out loud. A page whose content is assembled in the browser by JavaScript can rank well on Google, look flawless to every human visitor, and be an empty shell to ChatGPT, Claude and Perplexity. Websites built as single-page applications are the usual suspects, along with content hidden inside tabs and accordions that only loads when clicked.

The test takes thirty seconds. Open your page, view the page source (the raw source, not the browser inspector), and search for a sentence of your main content. If it is there, AI crawlers can read it. If it only appears after the page loads, they cannot. The fix is to serve your content in the HTML your server first returns, through server-side rendering, static generation or pre-rendering. Your developer will know which fits your stack.

The quieter gaps: speed, structure and schema

These three gaps attract less attention, but are just as important.

Page speed and the crawler’s time budget

Retrieval bots fetch pages against tight timeouts measured in seconds. A slow page does not get a second chance; the answer gets built from a faster source instead. Any website page speed work you are already doing for Core Web Vitals serves AI readers too.

Heading structure and buried answers

A crawler uses your heading hierarchy, H1 to H3, to work out which section answers which question. Pages with decorative headings, or none, give it nothing to hold on to. The same goes for answers buried beneath four paragraphs of scene-setting: extraction favours pages that answer first and elaborate second. This is a happy alignment, because human readers have always preferred the same thing. 

Our guide to simplifying technical content explains this in detail.

Missing schema markup

Schema markup is structured code that states the facts of a page outright: this is an organisation, this is its name, this is an article, these are its FAQs. Without it, a crawler can read your words but has to infer your facts, and inference means uncertainty. Across the AI readiness checks we have run, our consistent finding is that sites with good design and genuinely interesting content are often let down by absent or generic schema and small machine-readability faults their teams don’t know about. The encouraging flip side is that these are building blocks rather than rebuilds. They fold into an existing publishing workflow with modest effort.

Three checks you can run today (and what they cannot tell you)

Check your robots.txt. Visit yoursite.com/robots.txt and look for the crawler names in the table above. No mention of a bot means it is allowed by default; explicit disallow lines mean a decision was made by someone, or a tool default setting, at some point. Check whether this was deliberate.

Check your page source. View the page source on your homepage and one key service page, and search for your own opening sentence, as described in gap two.

Check your schema. Paste a page URL into Google’s free Rich Results Test. It shows what structured data the page carries, if any.

We audited our website using our AI readiness methodology to see how AI crawlers read our content

When we audited our own website, it passed all three checks: server-rendered, crawlers allowed, schema in place. AI tools still did not surface us for category-level questions, because those answers are also built from third-party sources, directories and round-ups that on-site fixes won’t remedy. You can read our full report here:

What We Found When We Ran Our AI-Readiness Check On Our Own Website

The key takeaway is that readability is the entry ticket, but not the whole game. Because different AI engines draw on different sources, the wider work belongs to answer engine optimisation as a whole, which our plain-English guide to AEO covers.

Frequently asked questions about AI crawlers

The questions site owners ask most, taken straight from the search data.

What is ClaudeBot and why is it in my website logs?

ClaudeBot is Anthropic’s web crawler. It gathers content used to train the Claude AI models, which is why it can appear often in server logs. Anthropic also runs Claude-SearchBot, a separate bot that fetches pages for Claude’s live search answers.

What is GPTBot, and should I block it?

GPTBot is OpenAI’s training crawler. Blocking it keeps your content out of future model training but has no effect on ChatGPT’s search results, which use a different bot. Whether to block it is a data-rights decision each business should make deliberately.

What is the difference between GPTBot and OAI-SearchBot?

Both belong to OpenAI, with different jobs. GPTBot collects content for model training. OAI-SearchBot builds the index behind ChatGPT’s search answers; block it and your site will not appear in those answers.

How do I check whether my website is blocking AI crawlers?

Open yoursite.com/robots.txt in a browser and look for lines naming GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot under “User-agent”, each followed by an Allow or Disallow rule. Also review your CDN or firewall settings, which can block bots regardless of what robots.txt says.

Do AI crawlers read JavaScript?

They fetch JavaScript files but do not execute them, so content that only appears after scripts run is invisible to them. Google’s Gemini is the main exception. Content present in the server-rendered HTML is readable by all of them.

Does page speed affect AI search visibility?

Yes. Retrieval crawlers work to tight time budgets and abandon slow pages, so the answer gets built from a faster source. Speed improvements made for Core Web Vitals help AI readability too.

How do I block AI bots in robots.txt?

Add a User-agent line naming the bot, followed by a Disallow rule. Reputable AI crawlers respect this. Before blocking anything, check which kind of bot it is: blocking retrieval bots removes you from AI answers, which is usually the opposite of what a business wants.