Last updated September 2026. This checklist reflects Cloudflare’s published policy as of September 15, 2026, the date its new AI Crawl Control defaults take effect.
Most site owners think one line in robots.txt settles the question of AI crawler access. It does not. Your CDN enforces its own bot rules above that file, and Cloudflare’s rules just changed in a way that can silently cut off crawlers you were counting on.
On September 15, 2026, Cloudflare split its old, single AI-bot toggle into three named categories: Search, Agent, and Training. New customers, new sites on existing accounts, and every free-tier account now block Agent and Training crawlers by default on any page that carries an ad unit. Bots that blend more than one category, including Googlebot, Applebot, and Bingbot, inherit whichever rule you set is strictest, so blocking Training can knock out a crawler you thought you had allowed.
None of this is hypothetical. Otterly.AI’s analysis of more than one million AI citations found that 73% of sites already carry a technical barrier, a robots.txt block, a CDN rule, or a JavaScript rendering gap, that keeps AI crawlers out before a page can even compete for citation share. The 14 checks below turn that risk into something you can verify in minutes, not guess at.
What changes on September 15, 2026
Cloudflare now classifies every AI crawler by what it does on your site, not just by name:
- Search: crawls and indexes content to answer questions about it later.
- Agent: fetches a page in real time because a person asked their AI assistant to.
- Training: collects content to train or fine-tune a model.
A crawler that does more than one of these, Cloudflare calls it a mixed-use crawler, inherits your strictest rule across every category it belongs to. Block Training anywhere on the site, and a mixed-use bot like Googlebot gets blocked everywhere, including the Search behavior you meant to keep.
| Setting | Before September 15, 2026 | After September 15, 2026 |
|---|---|---|
| Search crawlers on ad-monetized pages | Allowed | Still allowed |
| Agent crawlers on ad-monetized pages | Allowed by default | Blocked by default |
| Training crawlers on ad-monetized pages | Allowed unless manually blocked | Blocked by default |
| Mixed-use crawlers (Googlebot, Applebot, Bingbot) | Treated as Search only | Blocked entirely if you block Training |
| Accounts affected automatically | None | New customers, new sites, existing free-tier accounts |
| Existing paid accounts | N/A | Keep current settings unless you opt in |
If every site on your Cloudflare account predates September 15 and you are on a paid plan, nothing changes without your action. But a new site added to that same account, or any client running on the free tier, inherits the new blocks the moment it goes live. Check every property you manage, not just the flagship domain.
The 14-point access checklist
Each check below pairs a claim about how AI crawler access actually breaks with a command you can run against your own domain right now. Swap in your real domain for yoursite.com and, where a check names specific bots, swap in whichever ones matter to your engine mix.
1. Confirm your AI Crawl Control tier settings
Claim: Your account’s Search, Agent, and Training toggles decide the outcome before any bot reaches your server. A setting you never touched can still be blocking traffic.
Verify: In the Cloudflare dashboard, open Security > Bots > AI Crawl Control on every zone and read the status next to Search, Agent, and Training. The setting is per site, not per account, so repeat this for each property.
2. Fetch your live robots.txt for AI bot rules
Claim: robots.txt only stops crawlers that choose to obey it, but a Disallow line for GPTBot, ClaudeBot, or PerplexityBot is still the first thing most engines check.
Verify:
curl -s https://yoursite.com/robots.txt | grep -iE "gptbot|claudebot|perplexitybot|google-extended|ccbot|bytespider"Any match tells you exactly which Training bots are disallowed somewhere on the site.
3. Curl-test each Training bot directly
Claim: Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider) feed model training, not live answers. Know whether Cloudflare’s new default silently blocked them before you assume it did.
Verify:
for ua in GPTBot ClaudeBot Google-Extended CCBot Bytespider; do
echo "$ua:"; curl -s -A "$ua" -o /dev/null -w "%{http_code}\n" https://yoursite.com/
doneA 403 or 429 means that bot is blocked at the edge, whatever your robots.txt says.
4. Curl-test each Agent bot directly
Claim: Agent bots (ChatGPT-User, Perplexity-User, Claude-User) fetch a page because a person asked their assistant a live question right now. Losing this tier costs you the answer, not just future training data.
Verify:
for ua in "ChatGPT-User" "Perplexity-User" "Claude-User"; do
echo "$ua:"; curl -s -A "$ua" -o /dev/null -w "%{http_code}\n" https://yoursite.com/
done5. Curl-test the mixed-use Search bots
Claim: Googlebot, Applebot, and Bingbot carry both Search and Training behavior under Cloudflare’s new rules. Block Training anywhere, and these three inherit that block everywhere.
Verify:
for ua in Googlebot Applebot Bingbot; do
echo "$ua:"; curl -s -A "$ua" -o /dev/null -w "%{http_code}\n" https://yoursite.com/
doneA 403 here is the costliest failure on this checklist: it costs you organic search, not only AI citations.
6. Check whether a page carries the ad signal that triggers the new default
Claim: The Agent and Training block only fires by default on pages Cloudflare reads as ad-monetized. A page with no ad script keeps the older, more permissive behavior.
Verify:
curl -s https://yoursite.com/ | grep -iE "adsbygoogle|doubleclick|googlesyndication|ad-slot"A match means treat this page as covered by the new default, and confirm your check 1 settings apply to it.
7. Rule out legacy Bot Fight Mode overriding the new settings
Claim: Bot Fight Mode and Super Bot Fight Mode predate AI Crawl Control and can still challenge or block a crawler even when your AI-specific toggle reads “allow.”
Verify: In Security > Bots, confirm Bot Fight Mode is off, or that AI crawler user agents sit on its allow list. There is no reliable curl test here: a challenged bot gets a 200 status with a JavaScript challenge page instead of a clean 403, so the block hides in plain sight.
8. Audit custom WAF rules for accidental crawler blocks
Claim: A firewall rule written to stop scrapers, “block requests with no Referer header” or “block traffic from ASN X,” often catches AI crawlers too, since they rarely send a Referer and route through a handful of data-center ASNs.
Verify: In Security > WAF > Custom rules, read every rule’s match conditions against the user agents from checks 3 through 5. Any rule matching on a missing header, an ASN, or a country deserves a second look.
9. Check whether rate limiting throttles crawlers to near zero
Claim: A rate limit tuned for human traffic can cap a crawler at one request every few minutes. Your dashboard still shows “allowed,” but the crawler experiences the site as unreachable.
Verify:
for i in 1 2 3 4 5; do
curl -s -A "GPTBot" -o /dev/null -w "%{http_code} " https://yoursite.com/
done; echoIf later requests return 429 while the first returns 200, a rate limit is the real blocker, not your AI Crawl Control toggle.
10. Confirm key pages don’t hide content behind client-side rendering
Claim: Most AI crawlers fetch raw HTML and do not execute JavaScript the way a browser does. A page that renders its answer only after a script runs stays invisible to them, regardless of any bot setting.
Verify:
curl -s https://yoursite.com/your-key-page | grep -c "<p"Compare that count to what you see in a browser’s rendered page. A large gap means the content loads client-side and crawlers see an empty shell.
11. Grep your server logs for what actually happened
Claim: Dashboard toggles describe intent. Your logs describe outcomes, including bots served through a proxy layer that isn’t Cloudflare at all.
Verify:
grep -E "GPTBot|ClaudeBot|PerplexityBot|Google-Extended|ChatGPT-User|Perplexity-User" access.log | awk '{print $9}' | sort | uniq -cThis counts the status codes each bot actually received. A cluster of 403s or 429s confirms a block your settings review missed.
12. Check Managed Challenge and CAPTCHA rules for AI user agents
Claim: A Managed Challenge returns HTTP 200 with a page a human can click through. A crawler cannot. It never shows up as a block, so it is the easiest failure on this list to miss.
Verify: In Security > WAF, filter your firewall event log by the user agents from checks 3 through 5 and read the Action column. “Managed Challenge” or “JS Challenge” against an AI crawler is a silent block, even though the HTTP status looks clean.
13. Compare llms.txt against robots.txt for contradictions
Claim: A published llms.txt file should point to the same pages robots.txt allows. A curated llms.txt that lists pages robots.txt disallows sends AI crawlers toward a URL they are also told to skip.
Verify:
curl -s https://yoursite.com/llms.txt
curl -s https://yoursite.com/robots.txtRead both side by side. Every URL in llms.txt should resolve without a matching Disallow rule in robots.txt.
14. Put this checklist on a recurring schedule
Claim: Cloudflare’s defaults, bot names, and category rules will keep changing after September 15. A check you ran once tells you nothing about next month.
Verify: Save checks 2 through 11 as a script and run it monthly, or after any CDN, WAF, or hosting change. A dedicated crawler-access tool can run this automatically and alert you the moment a status code changes.
Tools that automate these checks
Running all 14 checks by hand once is reasonable. Running them monthly, across every property you manage, is not.
Profound pairs Agent-tier access with citation tracking, which fits checks 4 and 5 directly: it shows you not just whether ChatGPT-User or Perplexity-User can reach a page, but whether that access is turning into an actual mention.
Knowatoa runs crawler-access checks as a packaged workflow: robots.txt parsing, per-bot reachability tests, and alerts when a status code changes, which covers most of checks 2 through 11 without a manual script.
Otterly.AI, the source of the 73% technical-barrier figure above, folds a crawlability audit into its broader GEO monitoring, alongside citation tracking across ChatGPT, Perplexity, and Google AI Overviews.
If you want one subscription that covers crawler-access monitoring alongside citation tracking and content execution, Temso is the easiest all-in-one option, from $89/mo, with unlimited projects, users, and recommendations on every plan.
The full side-by-side comparison of GEO platforms is at /rankings/geo-tools. For how we test crawler-access signals across every platform we review, see /methodology.