Disable Cloudflare's AI bot block (the hidden 403)

Cloudflare · WAF · Bot management

In short: Since July 2025 Cloudflare blocks AI crawlers by default on new sites, and Bot Fight Mode can behave similarly — owners think robots.txt is open while the edge returns 403. Verifying takes one curl command; disabling takes three minutes in the Cloudflare dashboard.

Live data — sites we actually scanned

AI/arama botlarına edge engeli

Method: 209 completed self-serve checks (anonymized — no domains). Rows show the share of sites where this check came back warn/fail. Data: 2026-09-23. Cite via the open research dataset →

Why is the problem 'hidden'?

Because there are two separate layers: your site's declaration (robots.txt — what you wrote) and the network's behaviour (Cloudflare WAF/bot rules — what you don't see). Even with zero blocks in robots.txt, the edge layer can return 403 Forbidden or a challenge page to bots. Open the site in your browser and everything looks normal — the problem only fires on requests carrying a bot user agent.

Is there a block? Verify in one minute

Run two requests in a terminal:

curl -s -o /dev/null -w "%{http_code}" https://yourdomain.com/ — your normal request should return 200.
curl -s -o /dev/null -w "%{http_code}" -A "OAI-SearchBot" https://yourdomain.com/ — if the same page returns 403/503, an edge block exists.

Try other bots too: -A "ChatGPT-User", -A "ClaudeBot", -A "PerplexityBot". Seeing which one is blocked helps identify which rule fires. Our scan does this for five bots automatically.

Steps to disable it in Cloudflare

AI Audit: Cloudflare dashboard → your site → check the AI crawler controls under Security/AI — if "Block AI bots" is on and you want visibility, turn it off.
Bot Fight Mode: under Security → Bots — aggressively challenges known bots; turn it off if unneeded or move to a softer mode than "Definitely automated".
WAF rules: check Security → WAF → Custom rules for user-agent-based block rules; you can add a rule that explicitly skips search bots.
Super Bot Fight Mode: on paid plans there's a "verified bots" distinction — allow verified search crawlers.

This article is also for you if you want to block

Conversely — deliberately blocking AI bots is a legitimate choice too (content licensing, bandwidth). What matters is that the block is your decision: writing it in robots.txt (declaration) or enforcing via a WAF rule (enforcement) knowingly is different from staying blocked without realising. The scan reports both situations separately.

Related check in our scan: AI/search bots blocked at the edge (AI-EDGE-PROBE)
We measure this technically on your site — findings land in the report with evidence.

Frequently asked

No Disallow in robots.txt but bots are still blocked — normal?

Yes, and it's the most common hidden problem — robots.txt is only a declaration; edge layers like Cloudflare block independently of it.

I don't use Cloudflare — can the same problem exist?

Yes — other CDNs, hosting WAFs and CMS security plugins like Wordfence produce similar blocks. The verification command is the same.

Once I remove the block, do I appear in AI immediately?

Not immediately — bots need to rediscover your site and engines need to process the content. Removing the block is the prerequisite, not a results guarantee.

Test your site free Research data

This page is technical guidance; no ranking or AI citation guarantee is claimed.