Last updated September 2026.
For 30 years, a crawler could fetch any public page for free. It sent a request. The server sent back a 200 and the content. That was the whole transaction.
Pay per crawl breaks that assumption. HTTP status code 402, Payment Required, has existed since 1997 and was almost never used. Cloudflare turned it into live infrastructure that decides whether an AI crawler gets your content at all.
Here is what changed, how the exchange works, and what it means for tracking AI visibility now.
The numbers that define the moment
- 1 million+: Cloudflare customers who have enabled AI Crawl Control, the toggle that includes pay-per-crawl pricing (Cloudflare, 2026).
- 1 billion+: HTTP 402 responses Cloudflare’s network sends to AI crawlers every day (Cloudflare Blog, “Introducing AI Crawl Control”).
- 35.7%: the share of the world’s top 1,000 websites that block GPTBot, the most-blocked AI crawler tracked, up from about 5% at its August 2023 launch (Originality.AI, August 2024).
Three years ago, none of these numbers existed. Publishers now have a lever between letting every bot in for free and blocking every bot outright.
How the HTTP 402 exchange actually works
Pay per crawl runs on a simple exchange between a crawler and a server. Cloudflare’s implementation supports two paths.
The reactive flow. The crawler requests a page with no price agreement. If the page carries a price and the crawler has not paid yet, the server responds with HTTP 402. The response carries a crawler-price header stating the cost.
The crawler can retry the same URL with a crawler-exact-price header matching that price. If it matches, the server serves the page with HTTP 200 and a crawler-charged header confirming the charge.
The proactive flow. The crawler skips the round trip entirely. It sends a crawler-max-price header on the first request, stating the most it will pay. If the page’s price sits at or below that ceiling, the server serves the content right away with HTTP 200.
AI crawler Website
| |
|--- GET /article --------------------> |
| |
|<-- 402 Payment Required ------------- |
| crawler-price: $0.01 |
| |
|--- GET /article --------------------> |
| crawler-exact-price: $0.01 |
| |
|<-- 200 OK --------------------------- |
| crawler-charged: $0.01 |
| [page content] |The crawler does not identify itself with a user-agent string alone. It authenticates with a signed cryptographic key pair. That way, the server knows exactly which bot is asking, and can hold it to the price it agreed to. That signed identity is what makes the rest of the system, pricing by bot purpose, possible.
Purpose-declared bots: why “AI crawler” is no longer one bucket
Early crawler control was binary: allow GPTBot or block it. Cloudflare’s current model sorts every AI crawler into a purpose-declared category, and site owners set rules per purpose instead of per bot.
- Search: crawlers that index content to answer questions later. Site owners generally want this traffic, since it can drive referrals. Allowed by default.
- Agent: crawlers acting in real time on a person’s behalf, such as a browser-driving assistant completing a task. Blocked by default on ad-monetized pages as of Sept. 15, 2026.
- Training: crawlers harvesting content to train or fine-tune a model, where the content gets absorbed permanently rather than answered from live. Blocked by default on ad-monetized pages as of the same date.
A single bot can carry more than one purpose tag. Cloudflare applies the strictest rule that matches. A crawler flagged as both Search and Training gets blocked if Training is blocked, even though its Search behavior is welcome.
What changed, and when
| Event | Date | Source |
|---|---|---|
| Cloudflare launches Pay Per Crawl private beta, introducing the 402-based pricing exchange | July 1, 2025 | Cloudflare Blog |
| AI Crawl Control reaches general availability | August 2025 | Cloudflare Blog |
| AI Crawl Control consolidated into one console, available on every plan tier, including free | April 13-17, 2026 | Cloudflare Blog, Agents Week 2026 |
| Cloudflare shifts its commercial model from per-fetch pricing toward paying publishers when content gets cited, launching with Ceramic.ai and You.com | July 1, 2026 | Cloudflare, reported by TechCrunch |
| Agent and Training category crawlers become blocked by default on ad-monetized pages | Sept. 15, 2026 | Cloudflare Blog |
Pay per crawl has not stayed still since launch. The per-fetch price was the first version.
Cloudflare’s own July 2026 announcement named the weak spot. More than half of crawl traffic from bots it classifies as legitimate goes toward re-fetching pages that have not changed since the last visit. A flat per-fetch price rewards that waste as much as it rewards real use.
The newer pay-per-citation model, still rolling out through partners, prices the outcome instead of the request.
Cloudflare is not the only company building this layer. TollBit runs a separate licensing and metering platform that sits between publishers and AI companies, tracking bot traffic and brokering paid access deals outside Cloudflare’s own network. Both are infrastructure, not GEO tools: they decide whether a crawler reaches your content, not whether an engine chooses to cite it once it does.
Why this matters for tracking AI visibility
Pay per crawl adds a layer that citation tracking did not need to account for before. A page can be perfectly optimized and still never get read, if the crawler that would have fetched it hit a 402 it would not pay.
That makes crawler-level visibility, not just citation counts, part of the measurement problem now.
Profound fits this most directly. Its Agent Analytics product tracks crawler activity at the request level, showing which bots reach your pages and how often. That is the closest match to reading pay-per-crawl and 402 data as a GEO signal.
Temso approaches the same problem from an all-in-one angle. It bundles AI bot analytics into its broader citation-tracking subscription, starting at $89/mo, so a team that wants crawler visibility alongside citation share and content execution does not need to run a separate tool for each.
Scrunch takes a readiness angle. Instead of only reporting what already happened, it evaluates whether your site’s structure and access rules are set up for AI agents to navigate it in the first place, which matters directly once Agent-category crawlers face default blocks on ad-monetized pages.
None of the three replaces the others. Read the scoring methodology behind how GEO platforms get evaluated on this site.
The bottom line
Pay per crawl turns crawler access into a priced transaction, enforced with HTTP 402 responses and cryptographically signed bot identity. It is not a ranking or citation signal by itself. It decides whether a crawler reaches your page at all, and that now sits upstream of every other GEO metric you track.
Check whether AI crawlers can actually reach your content. Temso tracks AI bot activity alongside citation share across engines from $89/mo, so a blocked crawler shows up before it costs you a citation. See the full comparison in the GEO tool ranking, or look up related terms in the glossary.