Blog

AI Crawlers Blocked Without Anyone Deciding: 6 Misses We Found on Our Own Sites

Cloudflare settings, our own robots.txt and a bot check were turning AI crawlers like GPTBot and CCBot away. Six misses from an audit of our own sites, and how to test yours.

A post going around said Cloudflare blocks Common Crawl's CCBot without asking the site owner. We run about 15 domains on Cloudflare, so we checked every one of them: the Cloudflare settings, the live robots.txt files, and a few days of request logs filtered by crawler user agent.

Cloudflare had been refusing verified AI crawlers on two of our domains until recently. Most of what we found, though, we had written ourselves. Below are the six misses, why each one happens, and how to check your own site in a few minutes.

Why robots.txt is only half the answer

Most guides about AI crawlers stop at robots.txt. robots.txt is a request to well-behaved crawlers, and it is only read if the crawler gets through. A CDN or firewall can refuse a crawler before robots.txt comes into it. In that case your robots.txt can say Allow: / while GPTBot gets a 403 on every page, and nothing in your CMS or your repository will show it.

So there are two places to look: what your robots.txt says, and what your edge actually answers when a crawler asks for a page.

Miss 1: Cloudflare's AI bot blocking was switched on

On July 1, 2025 Cloudflare announced it was changing the default to block AI crawlers unless they pay. On July 1, 2026 it split AI traffic into three groups, Search, Agent and Training, each of which can be allowed, blocked, or blocked only on pages that show ads. According to the Cloudflare bots changelog, since September 15, 2026 new domains get Training and Agent bots blocked on pages that display ads, and crawlers that do both search and training are caught by the Training block.

On two of our domains the older "Block AI bots" setting was on. Verified AI crawlers were turned away there until late September on one and early October on the other. Nothing on the sites themselves showed it. We found it only by filtering the logs by crawler user agent.

How to check: in the Cloudflare dashboard, open Security settings and look at the AI bot policies, AI Crawl Control and Bot Fight Mode. Then check that each crawler you want is set to Allow.

Miss 2: our own robots.txt blocked training crawlers

Four of our sites had User-agent: CCBot and User-agent: Bytespider groups with Disallow: /, and a Content-Signal line with ai-train=no. They were added during an earlier round of changes meant to make the sites ready for AI agents, and nobody had looked at them since.

Training crawlers matter even if you only care about being cited. Common Crawl's archive feeds many training datasets, and what a model knows about your brand, product and prices comes partly from crawls like these. Block them and the model answers about you from older or second-hand sources. You can block them on purpose. Just make sure it is a decision someone made.

Cloudflare can also add these lines for you. Its managed robots.txt puts Content-signal: search=yes, ai-train=no and Disallow: / groups for Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent in front of your own file, and the two are served as one response. With that setting on, a site that has no robots.txt of its own gets one created with those rules.

How to check: open https://yourdomain.com/robots.txt in a browser and read every group. Read the live file, not the copy in your repository, because the copy in your repository may not be what crawlers get.

Miss 3: render files disallowed

Two of our sites disallowed /_next/, /static/ and *.json. Those paths hold the JavaScript, styles and data that a crawler needs to render the page. Google's robots.txt introduction is direct about this: if the absence of these resources makes the page harder for Google's crawler to understand, don't block them. AI crawlers that render pages are in the same position.

How to check: search your robots.txt for build folders such as /_next/, /static/, /assets/ or /build/, and for file patterns like *.js, *.css or *.json. Keep disallows for private paths such as /api/ or /admin/, and remove the ones that cover render files.

Miss 4: Crawl-delay lines

The same two sites had Crawl-delay lines. Google's robots.txt documentation says fields such as crawl-delay aren't supported, so Google ignores them. Some other crawlers do honor them, so the only effect was to slow down everyone except Google. Unless a crawler is actually overloading your server, remove them.

Miss 5: Browser Integrity Check and error 1010

Cloudflare's Browser Integrity Check was on for all our zones. It returns error 1010, "The owner of this website has banned your access based on your browser's signature", to clients whose request looks wrong to it. In our tests it refused requests with Python and Java HTTP library user agents and some malformed ones. The major crawler user agents passed.

That still matters. Many fetch tools, feed readers and AI agents that pull a page during a conversation use plain HTTP libraries. We turned the check off. One of our sites is on Cloudflare Pages, and there the pages.dev hostname itself returned 1010 to the same clients, which no zone setting changed.

Miss 6: refusals that are not settings at all

Not every refusal comes from a setting. In the same logs one section of a site returned 522 timeouts to GPTBot, and another site's preview images returned thousands of 500 errors in three days. The crawler sees an error either way, and no checklist of settings will catch it.

How to check: in your CDN or server logs, group responses by user agent and status code for the crawlers you care about, and look at every 4xx and 5xx.

How to test your own site

  1. Read your live robots.txt from top to bottom, including anything your CDN adds in front of it.
  2. Request one of your pages with a browser user agent, then again with each crawler's user agent, and compare the status codes. A quick way is curl -s -o /dev/null -w "%{http_code}" -A "CCBot/2.0 (https://commoncrawl.org/faq/)" https://yourdomain.com/. A 403, a challenge page, or a 402 (Cloudflare's pay per crawl) means that crawler doesn't get the page.
  3. In Cloudflare, check AI bot policies, Bot Fight Mode, Browser Integrity Check, managed robots.txt, and any WAF custom rules that match user agents.
  4. Look at your logs for errors returned to crawlers.

One caveat about step 2: your test comes from your own IP address, not the crawler's published ranges. A verified crawler can sometimes get through where your test is refused, and the reverse can happen with rules based on IP addresses. User-agent rules and AI bot blocking usually catch both, so a refusal in this test is worth checking.

What we changed in the IndexLinks check

We added this test to the free AI Search Readiness check. It now requests your page as 10 AI crawlers with their real user agents:

Each answer is compared with what a browser gets, and the report names any crawler that was refused, with the status code and the Cloudflare error code when there is one. Training-crawler blocks, in robots.txt or at the edge, now fail the check instead of showing as a note. The homepage site check runs the same test. Customers who turn on the weekly re-check get an email that names any crawler that has become blocked since the last run, so a settings change or a new default doesn't go unnoticed for weeks the way ours did.

None of this makes a page appear in an AI answer. It only tells you whether the crawlers can reach the page at all, which is the first thing to rule out.

Use this information as a starting point, then verify the page-level evidence available to you.