10% worse, 100x cheaper, 10,000x faster: how AI started building itself

Every year since 2022, one more stage of the AI factory has been handed from a human to another AI. The rubric writer, the data labeler, the exam grader, the textbook author, and now parts of the research loop are models. The tradeoff across those handoffs has stayed consistent enough to name: roughly 100x cheaper and 10,000x faster production, at the cost of about 10% of measured quality (Latent Space, "10% worse, 100x cheaper, 10,000x faster"). The 10% number is the source's stylized frame, not a measured benchmark, and the direction of the gap shows up across every flipped stage.

The first flip was the reward signal. In 2022, OpenAI's InstructGPT showed that a language model trained on human preferences could replace most of the human preference labeling needed to align a chat model (InstructGPT, arXiv:2203.02155). Within a year, Anthropic's Constitutional AI had the model critique itself against a written list of principles (Constitutional AI, arXiv:2212.08073). A 2023 paper from Lee et al. showed AI-generated feedback matched human feedback on a summarization task at a fraction of the cost, and the 2023 RLAIF paper named the move out loud: Reinforcement Learning from AI Feedback (RLAIF, arXiv:2309.00267). The approval apparatus inside model training was, by 2023, largely an LLM.

The second flip was the grading. Zheng et al.'s MT-Bench and Chatbot Arena paper, published in 2023, made a model-as-judge the default evaluation methodology for chatbot research (MT-Bench, arXiv:2306.05685). The pair's AlpacaEval followed. By 2024, the leaderboards most labs cite to compare new models were being scored by other models rather than by humans.

The third flip was the training data. In 2023, Microsoft's research group argued in the title of their paper that "Textbooks Are All You Need": a small model trained on textbook-style synthetic data could match a larger model trained on the open web (Phi, arXiv:2306.11644). The follow-up, phi-1.5, confirmed the result. Apple went further: its WRAP paper rephrased the entire pretraining corpus with an LLM and reported that the resulting model trained roughly 3x more efficiently, a number the paper attributes to its own 1.7T-token web crawl. NVIDIA shipped the same idea as a headline feature of Nemotron-4 340B, a permissively licensed model whose main contribution was a synthetic-data generation pipeline. By 2025, reasoning-trace corpora, chains of thought written by stronger models and used to train weaker ones, had become a standard pretraining and mid-training ingredient.

The fourth and fifth flips are landing in 2024 and 2025. Synthetic rubrics are replacing hand-written evaluation checklists. The "AI researcher" line of work is replacing parts of the experiment-design loop. End-to-end RL environments, practice gyms where models train on other models' feedback, are replacing human-curated task suites. Each of these sits at a "patient zero" paper or product, the moment a synthetic version first becomes load-bearing at a frontier lab rather than a research demo.

The compounding risk is the part that does not fit on a leaderboard. Across every flipped stage, the synthetic version is close-but-not-equal to the human original. The ~10% gap shows up in pairwise preference tests, in benchmark scores, and in the gap between a model's reported and real-world usefulness. The bias of the generator is also encoded in the data it writes, and a policy trained against its own grader can learn to game the very signal meant to bound it. Stack five of those gaps (judge, data, rubric, researcher, environment) and the field is now training and evaluating models in a system where every component is a 10% approximation of the human original.

The mitigation is not to roll any of it back. The 100x-10,000x speedup is real, and synthetic data has unlocked capabilities that human labeling cannot fund. The mitigation is to make the tradeoff legible. When a paper claims a benchmark score, a model card claims a "safe" release, or a vendor claims an "open" weights drop, three questions are now load-bearing: who judged it, who wrote the training data, and who audited the pipeline. If all three answers are "a model," the headline number is a model-graded, model-trained, model-audited result, and the reader should price in a roughly 10% tax on trust.

The factory is not finished handing off stages. The next one on the visible list is the safety reviewer, and when that flip lands, the heuristic will still be the same.