What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?

GPTBot is OpenAI's training crawler: it collects pages to improve future GPT models. OAI-SearchBot is the indexing crawler behind ChatGPT's search feature and the citations that come from it. ChatGPT-User is not a crawler at all in the traditional sense: it fetches a single page in real time when a person or a Custom GPT asks ChatGPT to open a specific link. All three carry the OpenAI name, but each one has its own user-agent string, its own robots.txt token, and its own effect on your site.

If I block GPTBot, will ChatGPT stop citing my pages?

No. Blocking GPTBot only opts your content out of OpenAI's model training. Citations inside ChatGPT's search feature depend on OAI-SearchBot, a separate crawler with its own robots.txt token. Sites that disallow GPTBot but allow OAI-SearchBot keep their citation eligibility while withholding training consent. Confusing the two is the single most common robots.txt mistake in GEO.

What happens to AI crawlers on Cloudflare after September 15, 2026?

Starting September 15, 2026, Cloudflare applies a new default on ad-monetized pages for any new domain, new customer, or existing Free-tier account: Training-purpose crawlers and Agent-purpose crawlers are blocked, while Search-purpose crawlers stay allowed. Under OpenAI's own declared purposes, GPTBot falls under Training, ChatGPT-User falls under Agent, and OAI-SearchBot falls under Search. Site owners can change these defaults any time in Security settings before the deadline.

Should I block GPTBot on my site?

That depends on whether you want your content used to train OpenAI's models, a separate question from whether you want to appear in ChatGPT's answers. Blocking GPTBot has no effect on citation eligibility, since OAI-SearchBot handles that job on its own token. Many publishers block GPTBot while leaving OAI-SearchBot and ChatGPT-User untouched, which keeps citation access open without granting training consent.

Does a robots.txt block actually stop ChatGPT-User from fetching my page?

Not reliably. OpenAI's own documentation notes that ChatGPT-User acts on a person's direct instruction, and robots.txt rules may not apply to that kind of request the way they apply to automatic crawling. Treat a ChatGPT-User disallow rule as a strong signal, not a guarantee, and pair it with server-level or CDN blocking if you need a harder stop.

What tools show crawler-level bot traffic, not just citation counts?

Profound built Agent Analytics specifically to read AI bot traffic at the server-log level, separating GPTBot, OAI-SearchBot, and ChatGPT-User requests instead of lumping them into one "AI traffic" line. Temso, the all-in-one GEO platform from $89/mo, includes bot analytics alongside citation tracking in the same subscription. The full comparison of GEO platforms sits at /rankings/geo-tools.

GPTBot vs OAI-SearchBot vs ChatGPT-User: Which Crawler Actually Feeds Your AI Citations (and What Cloudflare's September 15 Block Means for Yours)

Last updated August 2026.

Most robots.txt files still treat “OpenAI” as one line item. That single decision either blocks a crawler you needed for citations or leaves the door open to one you meant to shut out.

The three OpenAI crawlers do three different jobs, and they answer to three different robots.txt tokens. Get the mapping wrong and you either lose citation eligibility by accident or grant training consent you never meant to give.

This is not a hypothetical problem. Cloudflare is about to change the default treatment of all three crawlers on a large share of the web, and the deadline lands in a matter of weeks. Site owners who have never opened their robots.txt file are about to have a default chosen for them.

Three bots, three completely different jobs

Every AI lab now runs more than one crawler, and OpenAI runs three under its own name. Treating them as one traffic source is the fastest way to misconfigure your site.

The stakes are real. According to Cloudflare’s own network data, GPTBot’s share of AI crawler traffic surged from 5% in May 2024 to 30% a year later, making it the third-largest crawler of any kind on Cloudflare’s network by May 2025 (Cloudflare, 2025). The same report found GPTBot was also the single most blocked AI crawler on the network, with more robots.txt disallow rules naming it than any other bot as of June 2025.

That combination, high traffic and high block rate, tells you something. A lot of site owners are blocking GPTBot on purpose. Far fewer are doing it with a clear understanding of what each of the three tokens actually controls.

The confusion is understandable. All three crawlers share the OpenAI name, and most CDN dashboards group them under a single “OpenAI” toggle. But a rule that blocks GPTBot for training reasons and a rule that blocks OAI-SearchBot for citation reasons solve two completely different problems. Flip the wrong switch and you can lose your citation eligibility while thinking you only opted out of training.

The GPTBot vs OAI-SearchBot vs ChatGPT-User taxonomy

This is the table worth bookmarking. Each row is pulled from OpenAI’s own crawler documentation: the exact user-agent string, the declared purpose, what the crawl actually feeds, and the robots.txt token that controls it.

OpenAI did not launch all three at once. GPTBot came first, in August 2023, built purely for training. OAI-SearchBot and ChatGPT-User arrived later, once ChatGPT gained live web search and browsing. The three tokens exist because the product grew new capabilities, not because OpenAI set out to design a taxonomy. That history is exactly why so many robots.txt files still only mention the oldest of the three.

Crawler	User-agent string	Purpose	What it feeds	robots.txt token
GPTBot	`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot`	Model training	Training data for future GPT model generations	`User-agent: GPTBot`
OAI-SearchBot	`Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot`	Search indexing	The search index ChatGPT draws its citations from	`User-agent: OAI-SearchBot`
ChatGPT-User	`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot`	Live user fetch	One page, fetched the moment a person or Custom GPT asks ChatGPT to open it	`User-agent: ChatGPT-User`

Each token works independently. You can disallow GPTBot while allowing OAI-SearchBot, and OpenAI will honor both rules on their own terms. That is the entire point of splitting one company’s traffic into three named agents: it lets you consent to one use of your content while declining another, instead of accepting or blocking OpenAI as a single, undifferentiated actor.

What blocking or allowing each one actually changes

GPTBot: the training crawler

GPTBot exists to collect content for OpenAI’s model training pipeline. A disallow rule here means your pages will not be used to train future GPT generations. It has no bearing on whether ChatGPT can find, fetch, or cite your content today. Publishers who object to AI training on principle, without objecting to AI visibility, block this one and leave the other two alone.

OAI-SearchBot: the citation crawler

OAI-SearchBot builds the index that ChatGPT’s search feature draws answers and citations from. Disallow this one and your pages stop surfacing inside ChatGPT’s search-grounded answers, though they may still appear as plain navigational links in some contexts. If citation share is the goal, this is the token to leave open. Blocking it is the single fastest way to disappear from ChatGPT’s cited sources without realizing why.

ChatGPT-User: the live-fetch crawler

ChatGPT-User is not really a crawler at all. It fires once, on demand, when a real person (or a Custom GPT acting on their behalf) asks ChatGPT to open a specific URL. OpenAI’s own documentation is direct about the limits here: this agent is “not used for crawling the web in an automatic fashion,” and robots.txt rules may not fully apply, since the request is user-initiated rather than automatic. Treat a disallow rule on this token as a strong signal, not a guarantee. If you need a hard stop, pair it with a server-side or CDN-level block instead of relying on robots.txt alone.

Cloudflare’s September 15, 2026 deadline, and what it means for your rules

Cloudflare sorts AI bot behavior into three declared purposes, and the definitions map cleanly onto the taxonomy above. Search is “any behavior that collects or indexes your content, so it can answer questions about it later.” Agent is “automated behavior that is acting, usually in real time, on a person’s behalf, to get something done right now.” Training is a crawler “taking your content to train or fine-tune a model,” where the data is permanently absorbed into the model itself (Cloudflare, 2026).

Match those definitions to OpenAI’s own declared purposes and the mapping writes itself. OAI-SearchBot is a Search-purpose crawler. ChatGPT-User is an Agent-purpose crawler, and Cloudflare names ChatGPT-User directly as an example of the category. GPTBot is a Training-purpose crawler.

Under Cloudflare’s new default, on any page that carries ads, Search stays allowed, while Agent and Training are both blocked. That means GPTBot and ChatGPT-User get blocked by default on your monetized pages come September 15, 2026, while OAI-SearchBot keeps working. If you want a different split, you set it yourself in Cloudflare’s dashboard under Security, Bots, AI Scrapers and Crawlers, before the deadline arrives.

This is not a new problem so much as a formal one. According to Otterly.AI’s 2026 analysis of more than one million data points, 73% of sites already carry a technical barrier, an unintentional robots.txt rule or a CDN security setting, that keeps some AI crawler out entirely (Otterly.AI, 2026). Cloudflare’s new default does not create that gap. It just makes the default choice for anyone who has not already picked one.

How to check where you stand right now

Open yoursite.com/robots.txt and search for GPTBot, OAI-SearchBot, and ChatGPT-User by name. A blanket User-agent: * rule with Disallow: / catches all three, whether you meant it to or not, and so does a single User-agent: OpenAI line copied from an outdated guide.
If you run Cloudflare, check Security > Bots > AI Scrapers and Crawlers. Cloudflare’s dashboard setting can override or duplicate your robots.txt rules, and the two can quietly conflict: a robots.txt that allows a crawler means nothing if the CDN blocks it first at the network edge.
Pull raw server or CDN logs and filter for the three user-agent strings in the table above. Analytics tools that rely on a JavaScript tag will not show you this traffic; crawlers do not execute JavaScript or fire tracking pixels, so a standard analytics dashboard reports zero visits for requests that actually happened.
Decide each token on purpose, not by default. Allow OAI-SearchBot if citation share matters to you. Decide GPTBot based on your position on AI training, separately. Treat ChatGPT-User as a soft signal rather than a hard block, and back it with a server-level rule if you need certainty.
Set a reminder before September 15, 2026. If you have not touched these settings, Cloudflare’s new default makes the Training and Agent decision for you, on every ad-monetized page.

Tools that show you the crawler, not just the citation

Citation-tracking tools tell you when you got cited. They rarely tell you which crawler reached your server, when, or how often it was turned away. That is a separate measurement problem, and it needs a separate capability.

Profound built its Agent Analytics feature specifically for this gap. It reads bot traffic at the server-log level and separates GPTBot, OAI-SearchBot, and ChatGPT-User requests into distinct lines, alongside prompt-volume data, instead of lumping OpenAI’s traffic into one bucket. For teams that need to prove exactly which crawler is or is not reaching a page, that server-log view is the specialist tool for the job. It is built for exactly this kind of technical, engine-by-engine diagnosis, which is also why it shows up in exec-level GEO reporting.

Otterly.AI approaches the same access question from the data side. Its analysis of crawler-access barriers across more than a million data points is the source behind the 73% figure above, and it is a useful reference point if you want to know how common these blocks are across the open web, not just on your own domain.

For teams that want crawler visibility and citation tracking in the same subscription rather than two separate logins, see the full side-by-side comparison of GEO platforms at /rankings/geo-tools, scored against the published methodology.

Audit your robots.txt against the table above before September 15. Temso, the all-in-one AI SEO platform from $89/mo, includes bot analytics alongside citation tracking in the same dashboard, so you can see which OpenAI crawler reached your site and whether it turned into a citation, without stitching together server logs and a separate monitoring tool yourself.