Half the Internet Is Now Bots. The Sites You Read Are Hiding From Them.

A person asks an AI a question, gets an answer, and never lands on the site that produced the knowledge. That site is starting to notice. By Cloudflare's own estimate, more than half of all web traffic right now is AI bots: scrapers fetching pages to train models or to assemble direct answers that bypass the publisher entirely.

Search engines indexed pages in exchange for sending readers back. Googlebot was a free, polite guest, and the click-through was the rent. That bargain built the open web of the last 30 years. AI tools are rewriting it: they crawl the same pages, but they answer inside their own interface and increasingly do not return the favor.

Two weeks from now, on September 15, 2026, Cloudflare plans to flip a default that turns that bargain into a hard wall. Ad-bearing pages on sites behind Cloudflare's network, which sits in front of roughly 30% of the top 10,000 websites, will block AI crawlers unless publishers explicitly opt in. One analysis suggests up to 30% of the world's top sites could vanish from Google AI Overviews overnight once that default lands.

The mechanical question is whether the doors close on the right publishers. The early evidence is not reassuring. A peer-reviewed ACM study found that misinformation-heavy sites are far less likely to block AI crawlers than reliable ones, which means the block-list is structurally biased: the cleanest sources drop out, and the rest of the answer stream slides toward the less-reliable end of the web. A preprint by Allaham and colleagues (v1, 22 May 2026, not yet peer-reviewed) audited ChatGPT, Copilot, Gemini, and Perplexity across 712 real questions in politics, health, and the environment. About 16% of the sources the AIs cited were themselves AI-generated websites. The answer is shorter than the source. It is derivative text stacked on top of derivative text.

The downstream signal is already showing up in traffic data. Wikipedia's English site saw measurable declines after Google AI summaries launched, and the same pattern appeared in other languages as AI summaries rolled out. Wikipedia is the single most-cited source on the open web. If it loses referral traffic, the chain that funds and motivates new editors weakens with it.

The other pressure point is cost. Crawling at AI scale is not free for the sites being crawled. Serving millions of fetches burns bandwidth and origin-server capacity, and there is no equivalent of the ad impression to pay for it. Some AI companies have, at times, bypassed the long-standing robots.txt polite-request standard rather than negotiate licensing. The polite-request layer is not the lede here: it has been politely ignored.

Cloudflare's default-block is the bluntest instrument. Around it, a market is forming: publisher paywalls, formal AI licensing deals that pay per crawl or per answer, and a small set of crawlers that explicitly send traffic back to the source. Search engines without an AI answer layer, Mojeek and Brave's index, are pitching themselves as the crawlers that still link. There is also a maintained "Ban the Bots" directory of search services that refuse AI crawling by design. None of these are substitutes for the open web that Googlebot built, but they are the scaffolding of a replacement deal.

The risk is the model-collapse loop. If AI answers keep leaning on AI-generated text, and the next model trains on those answers, the reference surface drifts further from anything a human fact-checked. The selection bias in who blocks crawlers and the derivative bias in what the AIs cite point the same direction.

The September 15 default is the first measurable test of whether the new deal can be negotiated, or whether publishers are forced to choose between being in the AI answer and being in the budget. The cost of getting it wrong is the difference between asking a question and getting the site that knows the answer.