Crawlability Is Not Citation

Discovery now runs through assistants, and a small business faces two separate fights: letting the bot in, and getting named when the bot answers. The playbook called 'AI visibility' is built around the first fight. The second one is where the rankings are actually won, and it operates on different rules.

website-auditor.io's AI Visibility Study measured 193 SMB sites and 5,978 answers from ChatGPT, Claude, Gemini, and Perplexity. The gate is mostly open: only 8.9% of sites block an AI crawler. The shelf is mostly empty: 94.8% of those same sites are never named in any answer — a figure measured at the site level, while the answer-level citation rate is roughly ten sites out of 193 appearing even once. Letting the crawler in and being chosen are not the same problem.

The instinct shows in how site owners block. Of 39 sites that disallow an AI user-agent, 38 block GPTBot. Only 4 block OAI-SearchBot, the retrieval agent that surfaces answers. The standard robots.txt advice trains for the wrong fight. That gap reflects both site-side factors and assistant-side ranking concentration; the study measures the gap but does not isolate its causes. A readable homepage and a clear sitemap is the cheaper half. The other half is being one of the handful of names an assistant returns.

Upstream is not all on the site. The Hacker News thread on website-auditor.io's data notes assistants concentrate answers on a small set of sources, so absence from a name list is partly model ranking and partly which publishers the model treats as authoritative. SMBs can move the readability lever today. The ranking lever is shared with the assistants and with whoever already sits on the shelf.