He Built a 1.5 Million-Page Site From Public Records. Then He Counted 214 Bots for Every Real Visitor.

For every one real human visitor PatronView recorded in a week, its server answered about 214 bot page loads. The roughly 1.5 million-page philanthropy database, built by scraping public IRS 990 forms, donor walls, and nonprofit annual reports, is the doorway into what small-site operators actually fight when they say they fight bots, and into the gap between what a server answers and what a JavaScript-based analytics dashboard can see.

The "99%" in the post title is a window-dependent summary. The underlying number is the gap: PatronView's visitor counter recorded 5,977 pageviews in the publication week, against about 1.28 million full pages the server served and about 2.5 million external requests answered. Daily visible traffic is roughly 500 visitors; the server answers millions of requests weekly. The author is running this measurement on his own logs and Cloudflare dashboard, not generalizing from a third-party dataset.

The operator blocked the polite SEO crawlers years ago (SemrushBot, AhrefsBot, MJ12bot, DataForSEOBot), and his Cloudflare dashboard still showed millions of weekly requests after that. The big offenders are not search engines. They are AI training crawlers and AI search agents.

Single-day observations from his writeup: 3.6 million requests from Chinese bots in one 24-hour window; 117,000 pages a day from Amazon's AI search crawler with zero referred visitors, until he blocked it; a locally measured 35,000 crawls per real visitor from Anthropic's crawler, against a Cloudflare-published typical figure of around 3,000-to-1. The Anthropic ratio is one operator's measurement, not a vendor-published number. In November 2025, a separate pattern appeared: about 4,000 single-page "visitors" with a 99% bounce rate and no referrer over a few days, the signature of a botnet, distinct from the named crawlers he had already blocked.

Robots.txt is polite and gets ignored. CAPTCHA looks like the obvious answer; his measured solve rate was 0.24%, which means most of the traffic is low-effort scraping rather than evasion-capable bots, so a puzzle wall stops the bulk of it. The current working stack is Cloudflare's bot classification layered on top: country filtering, JA3/JA4 fingerprint blocks, and explicit denies for the named AI crawlers. The author did not invent this stack; he just reported which pieces do what on a single site.

PatronView exists because he scraped public IRS 990s and donor walls to build a public-records tool. He is now defending that tool from scrapers, at higher volume than the public records he consumed to build it. The HN discussion around the post raises a separate concern: aggressive bot-blocking centralizes who can see a website in one company. Cloudflare's classifications carry their own false-positive risk, and some of the "bots" are legitimate user-driven scripts or LLM-assisted fetches. The same block list that stops the 117,000-pages-a-day Amazon crawler also decides, for every site behind it, what a reader's automation is allowed to look at.

The numbers on his dashboard are the same numbers any small operator can pull from their own server logs. Plausible, Fathom, and Google Analytics all measure JavaScript-executing humans, which is the roughly 1% the public sees. The roughly 99% that is not in analytics is the part the firewall is actually fighting, and the difference between "my site has 500 daily visitors" and "my site answers millions of requests a week" is the gap an operator has to close to know what is going on.