The Pelican Test: What 1,008 AI Drawings Catch, and What They Miss

"Draw a pelican riding a bicycle." That nine-word prompt has become one of AI's most-cited informal benchmarks, run by Simon Willison (the UK-based writer whose "Pelican on a Bicycle" side-quest now shows up in nearly every major model launch) against every major new model release. It is a stress test for a simple reason: a pelican has a long beak, a bicycle has thin spokes, and a frontier model has to get both right in vector graphics, the XML-based image format most modern AI models output by default. A clean rendering signals that a model can plan composition, follow instructions, and produce clean SVG. A bad one ends the launch post.

So when billions of dollars ride on the next "our model is best" claim, the obvious worry is that labs would tune for the test. Dylan Castillo, an applied ML engineer, spent 1,008 generations testing that hypothesis. His answer: not visibly. But his answer is also the wrong question, because the kind of cheating a 1,008-drawing grid can detect is not the kind that would actually pay.

Castillo built a 48-cell grid: 8 animals (pelican, flamingo, heron, otter, raccoon, antelope, whale, cat) crossed with 6 vehicles (bicycle, unicycle, skateboard, scooter, plane, boat), three samples per cell, at temperature 1.0 with matched reasoning effort. He ran the grid through OpenRouter (a model-routing service that lets one API call hit any frontier model) against seven frontier models: GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro. He rendered each SVG to PNG (a flat raster image, the format a browser or social platform would actually display), retrying on failure (only 11 retries across 1,008 generations), then asked GPT-5.6 Luna to grade each output on 1-to-5 scales for animal fidelity, vehicle fidelity, and action coherence.

The result: no model's pelican-on-bicycle cell scored noticeably better than its own other 47 cells. Within each model, the famous prompt is a generic data point, not an outlier. Willison himself called the methodology "significantly more robust" than his own spot-checking and accepted the negative finding.

Wire coverage will stop at that result. The more interesting finding surfaced in the Hacker News thread, where readers raised a more uncomfortable hypothesis: if labs are not tuning for the pelican cell specifically, they may be tuning for the SVG cell generally. A lab that improves its overall SVG output through better pretraining data, a stronger code model, or a fine-tune pass on rendered graphics would still bend Willison's pelican test, but it would not show up as a within-model anomaly on Castillo's grid. Castillo's design, a clever within-model comparison, is structurally blind to the very form of gaming that would actually scale.

Readers on the thread have started calling this "svgmaxxing": uniform improvement across an entire output family, rather than targeting the specific prompt that goes viral. SVG is the test case; the principle extends to every informal benchmark that ends up in a launch post, a comparison article, or a procurement slide. If a lab can inflate its performance on "viral-looking" code-art or reasoning-on-images tasks without anyone noticing, the test still passes and the marketing wins.

Castillo's experiment also has real limits. A single LLM judge (GPT-5.6 Luna) means the result is filtered through one model's preferences, which could mask small pelican-specific bumps. The 1,008-generation grid has the power to see coarse within-model anomalies, not the 0.2-of-a-judge-point improvements a deliberate tuning pass might produce. And the test is a snapshot of seven models at one moment in time; if a lab is currently training with pelican-style prompts in its reinforcement-learning-from-human-feedback (RLHF) or supervised fine-tuning (SFT) mix, Castillo's next rerun could look different.

None of that changes the broader point: the pelican prompt is small, and the principle scales. The next time you read a launch post that crowns a new model on a viral-looking test, the question worth asking is whether the lab got better at the entire family of outputs the prompt belongs to, in a way no within-model grid can detect.

A short reader framework, adapted from the HN discussion: ask four things before trusting an informal benchmark claim. Is the prompt public and frozen, or does the lab get to pick the version? Is judging done by an independent model or pipeline, not the same lab's own model? Is the training-data composition transparent enough to rule out a deliberate tune for the prompt family? Has a third party independently replicated the result? The pelican will not save you. The methodology will.