Starting September 15, 2026, Cloudflare will flip a default switch for a large share of the publisher web, blocking AI training and AI agent bots unless their operators pay through Cloudflare's Pay Per Use marketplace.
The infrastructure company that sits between most of the web and its visitors just changed how AI companies are allowed to reach publisher content, and it did so by default rather than by asking. On July 1, Cloudflare announced that starting September 15, 2026 it will block two of the three main classes of automated web traffic, the bots that scrape pages to train AI models and the bots that fetch pages on a user's behalf to power AI "agents," from any site the customer has marked as ad-supported, unless the bot's operator has agreed to pay for access through Cloudflare's Pay Per Use marketplace. Search crawlers, the bots that build the indexes Google and Bing use to answer queries, are still allowed through.
Cloudflare is a content delivery network, a layer of distributed servers that sits between websites and their visitors to speed up page loads and filter unwanted traffic. Independent reporting on the announcement treats the company's infrastructure position as large enough that a default change in its dashboard functions as a default change for a meaningful share of the publisher web.
The three-way split matters because the three classes of bot look similar in a server log but pay publishers very differently. Search crawlers lead users back to the original page, where the publisher can show ads or collect subscription revenue. AI training crawlers ingest the page into a model that may never link back. AI agent crawlers, the bots that answer a user's question by fetching a page in the background, sit somewhere in between: they consume the publisher's content but rarely route a user to it. Cloudflare's policy framing treats search as a known quantity and groups the other two as a single risk profile that publishers should be paid to bear.
The second half of the announcement is a quiet rebrand. In July 2025 Cloudflare declared "Content Independence Day", which blocked AI training crawlers by default for sites that opted in, and introduced Pay Per Crawl, a marketplace where AI companies could pay publishers per page fetched. As of this announcement, that product becomes Pay Per Use, and the billing trigger shifts: the billable event is no longer the fetch but the moment the content actually surfaces inside an AI chatbot's answer. That is a different product and a different fight. A fetch can be measured in a log file. An answer is a model output, and the publisher's share depends on whose attribution system Cloudflare and the AI company agree to trust.
Cloudflare frames the move as ecosystem maintenance. "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," CEO Matthew Prince said in the company's press materials. The phrase is doing a lot of work, because Cloudflare is also the party setting the price.
That is the legitimate critique the policy has to answer. Cloudflare is a single private company, and the new defaults apply to every site running on its network that has not affirmatively opted out. Publishers who want to leave the system have limited substitutes at comparable scale, and AI companies that refuse to participate in Pay Per Use face a binary: lose access to a large share of the publisher web, or scrape only the smaller share that runs on competing infrastructure. The Register and TechCrunch frame the move the same way, as Cloudflare using its traffic-layer leverage to push AI companies toward payment rather than as a neutral arbiter setting traffic rules for the public good.
The program is also launching with a thin partner list. Cloudflare named Ceramic.AI and You.com as launch partners, and neither is a frontier-model lab. The list being small is itself a tell: the policy only matters if major AI companies either accept the new terms or get blocked from a large share of the publisher web. If the largest model providers hold out and route around Cloudflare, the policy becomes a price floor for a niche, not a market reset. Security trade press has corroborated the new controls but has not yet reported major lab sign-ups.
The next test is September 15, 2026, when the default flip takes effect, and in the weeks that follow, when it becomes visible which AI companies have signed Pay Per Use deals, which have stopped crawling Cloudflare-hosted ad pages, and which have simply accepted being labeled. The answer will tell readers whether Cloudflare is pricing the AI content market or just redrawing a small corner of it.