Starting September 15 on new and free tier sites, the change forces Google, Apple, and Microsoft to identify their search indexing bots and their AI training crawlers as separate streams on ad supported publisher pages.
Cloudflare is using its position in front of a material share of the web to do what no government has yet required: force Google, Apple, and Microsoft to identify their search and AI-training crawlers as separate streams on ad-supported publisher sites. The content delivery network said on Tuesday it will default-block, on new and free-tier sites that have not changed their settings, crawlers that mix search indexing with AI-training scraping. The September 15 deadline turns a regulator-favored idea ("crawler separation") into a hosting-layer product default.
The "ad-supported" surface in Cloudflare's product map covers the news and content sites that rely primarily on advertising for revenue, exactly the publishers most exposed to the search-versus-AI bind.
The existing opt-outs do not actually solve that bind. Each of the three vendors exposes a separate AI-training signal: Google's Google-Extended, Apple's Applebot-Extended, and Bing's noarchive meta tag. But none of these work if a bot identifies itself only as a search crawler. Blocking Googlebot also means disappearing from Google search, and Google remains, per Cloudflare's traffic data, the largest referrer for most publishers. Cloudflare's January blog post on the topic called the failure mode "crawler separation": the only structural fix is for search crawlers and AI or agent crawlers to be distinct user agents, so publishers can allow one while blocking the other.
The default does not flip every customer at once. It applies to new Cloudflare customers, new sites on existing accounts, and free-tier customers who have not changed their settings. Legacy paying customers are not switched over automatically, which softens the immediate reach and turns the deadline into a market test: paid customers can keep admitting mixed crawlers if they choose, while new sites start with the stricter default.
The split is not theoretical. Apple disclosed in June that Applebot data trains Apple foundation models, confirming that the same bot which indexes search results also feeds Apple's AI. That is exactly the dual-use pattern Google, Microsoft, and any other operator of a mixed crawler is being asked to formalize. Cloudflare is also launching a Business Insights Dashboard that shows publishers how bots consume their content and how much AI-driven traffic flows back to them.
Cloudflare chief executive Matthew Prince, in The Register's write-up, pointed to the claim that the majority of internet traffic is now non-human. That figure is Cloudflare's own measurement, and is the load-bearing fact in the company's case.
The company has reached for this lever before. In July 2025 it launched Pay Per Crawl, a marketplace that lets AI vendors pay publishers directly for crawl access. That program has now been rebranded Pay Per Use, with publisher-payment partnerships announced alongside API-search vendor Ceramic.ai and agent-search vendor You.com. Adoption numbers have not been disclosed, and the partnerships are forward-looking rather than completed integrations, but the direction is clear: blocking creates a price signal, and Cloudflare intends to operate the tollbooth.
The change lands inside a regulatory shift that has only just begun. In October 2025 the UK Competition and Markets Authority designated Google as having Strategic Market Status in general search, citing 90 percent UK search share under the DMCC Act 2024 and opening a consultation on conduct requirements for AI-era search. Cloudflare has publicly supported that process while arguing it does not go far enough.
What makes this structurally interesting is the mechanism: hosting-layer gatekeeping as a substitute for platform-level rule-making. A single infrastructure provider is taking on a job the UK regulator has only opened for consultation, forcing the major crawler operators to split their search and AI traffic into distinct, independently controllable streams. Whether Google, Apple, and Microsoft comply depends on whether losing Cloudflare-defaulted sites from search costs less than rewriting crawler identity in advance. The September 15 default is the first test.