AINEWS
A 1,008 SVG study finds no pelican specific training bump. The real capability number lives on a test the lab hasn't seen.
A pelican on a bicycle is, by now, a recognizable image on AI Twitter: a small SVG of a long-beaked bird improbably balanced on two wheels, generated in a single prompt by a frontier model. A kuber.studio essay argues that this and a handful of other launch stunts are not capability demos at all. They are "demo-benchmarks": fixed, public, finite targets that a lab can optimize against on a known runway. The "test" is the launch, the answer is the SVG, and the prompt has been seen before.
A demo-benchmark is the opposite of a held-out eval. The pelican is a known prompt with a known shape; an engineer can iterate on it for weeks before a launch event. The interesting question, in the essay's framing, is what the demo measures: capability, or preparation. "A test you can perfect on a schedule measures preparation instead of capability," the author writes. "To me that's anti the very definition of a benchmark." The distinction matters because the AI feed now runs on these images, and the feed cannot tell a rehearsed pelican from a new one.
The empirical anchor for the lens is Dylan Castillo's pelicanmaxxing study: 1,008 SVGs across 8 animals and 6 vehicles generated by 7 frontier models, LLM-judged, then analyzed with a fixed-effects regression. The result is no statistically significant pelican-specific or bicycle-specific training boost. The improvement that does appear is consistent with general "SVGmaxxing," not with labs having specifically trained on the pelican prompt. An Explainx summary of the study dates the work to July 2026; the code is public on GitHub.
The Castillo result is a falsifier for a narrow claim, not a blanket verdict on the demo-benchmark class. The study tests only the pelican/bicycle prompt; it does not independently test whether labs have trained on the Minecraft-from-a-prompt stunt, the MS Paint self-portrait, the bouncing-ball gravity demo, or the SVG game controller. The Hacker News discussion of the essay flags exactly this: selection bias on which prompts went viral, and misuse of the "one-shot" framing on prompts that are not strictly one-shot. The lens is useful, but the empirical support is partial.
The kuber essay offers a concrete example of how static the eval problem can get: Thinking Machines' Inkling Small reportedly scored within a point of its flagship sibling on the Artificial Analysis Intelligence Index while using less than a third of the parameters, and beat it on Humanity's Last Exam, GPQA Diamond, and SciCode. The author paraphrases an Artificial Analysis post; the cited chain runs through the kuber essay and is not independently re-verified here. That is a different failure mode from the pelican: not a rehearsed test, but a static one where a small model and a large one land within a point on a fixed rubric.
The constructive exit is already running, and it is not the next pelican. LiveBench rotates its questions, so the tested party has not seen them. ARC-AGI keeps a private set, and Humanity's Last Exam holds part of itself back. A held-out eval cannot be clipped, rehearsed, or sandbagged the way a famous SVG can. That is exactly why the AI feed does not run on it. A pelican with pedals is visually obvious and shareable; a number on a rotating benchmark is not.
The next time a launch demo produces an image that looks like a shock, the question is whether the prompt has been on the runway. If yes, the result measures preparation. If no, watch what the held-out evals show, because that is where the capability number lives.