Last updated September 2026.
An AI engine cannot cite a page it never saw. Every downstream GEO tactic, a front-loaded answer, tight schema, a fresh update date, depends on one precondition most GEO advice skips: the crawler has to get in the door first.
Pay-per-crawl is the mechanism, built by Cloudflare and now spreading across the infrastructure layer of the web, that decides what happens at that door. It replaces the old binary of allow or block with a third option: charge. And starting September 15, 2026, the default answer for a large share of AI crawlers flips from allow to block.
Why crawler access is the zeroth step of GEO
Call it the zeroth step. Not step one, because step one assumes a crawler already reached your content. Zeroth, because if that assumption fails, nothing else in your GEO programme matters.
GPTBot has been the most commonly blocked AI crawler for two years running. Originality.AI’s August 2024 analysis of the world’s top 1,000 websites found 35.7% disallowed GPTBot in robots.txt, ahead of CCBot (22.1%), Google-Extended (13.6%), and ChatGPT-User (12.7%).
Access barriers show up in referral traffic too. Similarweb data reported by Modern Retail in September 2025 found Amazon receiving less than 3% of its referral traffic from ChatGPT, a gap the report tied directly to Amazon’s deliberate AI-crawler blocking. Walmart and Etsy, by contrast, each pulled more than 20% of their referral clicks from ChatGPT the same month.
Block the crawler, and your citation share for that engine stays at zero, no matter how good the page is.
How pay-per-crawl actually works
Cloudflare’s model rests on two mechanics: how a crawler declares itself, and what happens once it requests a page.
Bot purpose declaration and mixed-use crawlers
Cloudflare sorts automated traffic into three purposes.
- Search. Indexes content to answer questions about it later, the traditional search-engine crawl.
- Agent. Fetches one specific page in real time, on behalf of a person who asked a question right now.
- Training. Harvests content in bulk to train or fine-tune a model.
A crawler that identifies itself cleanly under one purpose gets treated by that purpose’s rules. A mixed-use crawler, one user agent that blends search, agent, and training fetches without separating them, does not get that benefit. Publishers cannot set different rules for a bot they cannot tell apart, so undeclared, mixed-use traffic gets treated as the most restrictive category by default.
Being a Cloudflare “Verified Bot” no longer implies access either. Verification now confirms identity, not permission. Only the declared, allowed purpose category grants entry.
The HTTP 402 payment flow
Once a crawler’s purpose is known, Cloudflare checks the publisher’s policy for that bot and that purpose: allow it free, block it, or charge it. The charge option runs on HTTP 402 Payment Required, a status code that sat almost unused in the HTTP specification for three decades before Cloudflare repurposed it.
STEP 1: Crawler requests a page
GET /article User-Agent: GPTBot
STEP 2: Cloudflare checks the publisher's per-bot policy
Blocked -> 403 Forbidden (no content, no citation, ever)
Free -> 200 OK (content served, no charge)
Priced -> 402 Payment Required (header: crawler-price: USD 0.02)
STEP 3: Crawler responds to the 402
Declines -> stops here, no content
Accepts -> retries with header: crawler-exact-price: USD 0.02
STEP 4: Cloudflare settles as merchant of record
Charges the crawler, credits the publisher, returns:
200 OK (header: crawler-charged: USD 0.02, content served)Cloudflare sits in the middle at every step: billing the crawler, crediting the publisher, and logging the transaction. Neither side has to build custom payment infrastructure.
What changes on September 15, 2026
The date matters because it marks when Cloudflare’s defaults flip, not just an option a publisher can turn on manually.
| Crawler purpose | Before September 15, 2026 | After September 15, 2026 (ad-monetized pages) |
|---|---|---|
| Search | Allowed by default | Still allowed by default |
| Agent | Allowed by default on new domains | Blocked by default unless the publisher opts in |
| Training | Allowed by default on new domains | Blocked by default unless the publisher opts in |
| Mixed-use, purpose undeclared | Treated as allowed | Blocked by default |
| ”Verified Bot” badge | Often read as implied access | No longer implies access on its own |
The default applies to new domains, new sites on existing accounts, and free-plan customers. Paid customers with an existing configuration keep it. The ad-page trigger is deliberate: Cloudflare treats an ad on a page as a signal that page was built for a human visitor, which makes the case for gating Agent and Training crawlers by default while leaving Search crawlers, the ones that still send return traffic, untouched.
The scale of the problem, in numbers
This is not a niche setting buried in an advanced tab. Cloudflare says its network handles more than 1 billion HTTP 402 responses to AI crawlers on an average day (Cloudflare, 2026), and more than 1 million Cloudflare customers have already enabled AI crawler blocking controls (Cloudflare, 2025).
And the gap is not always intentional. Otterly.AI’s 2026 report, “The AI Citation Economy: What 1+ Million Data Points Reveal,” found that 73% of sites carry a technical barrier, an inherited robots.txt rule, a CDN security default, a JavaScript rendering wall, that blocks AI crawler access without the site owner choosing it on purpose.
Put those together and the picture is clear: crawler access has quietly become infrastructure-level policy, not a checkbox you set once and forget.
Pre-deadline self-audit checklist
Work through this before September 15, 2026, whether or not you run on Cloudflare.
- Confirm your Cloudflare plan and domain status: new or existing, paid or free. The default block applies differently to each.
- Open AI Crawl Control in your Cloudflare dashboard and review the current Search, Agent, and Training settings for your domain.
- Check robots.txt for disallow rules against GPTBot, PerplexityBot, ClaudeBot, Google-Extended, and Amazonbot.
- Test your actual product and article pages, not just your homepage, against each major AI user agent.
- Decide, bot by bot, whether you want to allow it free, block it, or set a price.
- Flag which of your pages carry ads. Those are the pages the September 15 default touches first.
- Re-test after the deadline to confirm the settings you intended actually took effect.
Tools that check whether AI crawlers reach you
Reading raw server logs for user-agent strings works, but it does not scale past a handful of pages. A few tools automate the check.
Profound’s Agent Analytics maps which bots hit which URLs, and how often, against your own traffic data, useful once you need to prove access, or the lack of it, to a team that reports up to leadership. Knowatoa runs a dedicated crawler-access check against the major AI user agents and flags the robots.txt and CDN rules quietly blocking them, a fast way to find the gap before you fix it.
If you would rather fold that access check into a broader GEO subscription instead of running a separate tool, Temso includes AI bot analytics as part of its all-in-one platform, from $89/mo, alongside citation tracking and content execution.
The full comparison of GEO platforms, including how each one handles crawler-access checks, lives at /rankings/geo-tools, with the scoring criteria behind it at /methodology. Definitions for terms like citation share are at /glossary.
FAQ
What is pay-per-crawl?
Pay-per-crawl is a mechanism, built by Cloudflare, that lets a website charge an AI crawler for access instead of only blocking it or letting it through free. The publisher sets a price per bot, and each crawler request gets served free, blocked, or answered with a 402 Payment Required response carrying that price.
What does the HTTP 402 status code do here?
HTTP 402 Payment Required sat almost unused in the HTTP specification for decades. Cloudflare repurposed it: a priced page returns 402 with a price attached instead of content, and the crawler can retry with a payment confirmation to receive a 200 OK and the actual page.
What changes on September 15, 2026?
Cloudflare sets new default access rules for AI crawlers that day. On pages that carry ads, Training and Agent crawlers get blocked by default while Search crawlers stay allowed, and mixed-use crawlers that do not declare a purpose get treated as blocked too. The default applies to new domains, new sites on existing accounts, and free-plan customers; paid customers keep their existing configuration.
Does the September 15 default block affect my existing site?
Only if you are a new Cloudflare customer, add a new domain, or run a free-plan site that carries ads. Existing paid Cloudflare customers keep whatever crawler settings they already configured. Check your AI Crawl Control settings before the deadline if you are unsure which bucket you fall into.
How do I check if AI crawlers can actually reach my site?
Test robots.txt against the major AI user agents, review CDN and WAF rules for AI-bot presets, and check server logs for blocked-versus-served responses on real product and article pages, not just the homepage. Profound’s Agent Analytics and Knowatoa’s crawler-access checks both automate that comparison.
Does blocking AI crawlers help or hurt my chances of getting cited?
Blocking a crawler removes any chance that engine can ever cite you. Otterly.AI’s 2026 report found that 73% of sites carry a technical barrier, often unintentional, that blocks AI crawler access. Crawl access is a precondition, not a strategy: allowing the crawler does not guarantee a citation, but blocking it guarantees you never get one.
Run the self-audit above before September 15, 2026. Crawl access is binary: either the bot reaches your page, or it does not. No amount of schema, no front-loaded answer, and no content polish fixes a page an engine never saw. See how GEO platforms handle crawler-access monitoring at /rankings/geo-tools.