AIBRIEF
A peer reviewed benchmark across 19 experiments finds that language models prompted to impersonate one specific person match a cheap demographic baseline chatbot, undercutting the case for using LLMs in place of real research participants.
A language model told to "be" one specific person predicts that person's survey answers about a quarter of the time, roughly on par with a generic chatbot given nothing but that person's age and zip code. That parity, not the error rate itself, is the finding that undercuts the personalized digital-twin pitch in social science research.
The benchmark, published September 2 in Science Advances, tested LLM "twins," models prompted to impersonate an individual from a full survey profile, against the same individuals' real answers across 19 social science experiments. Twins were wrong on average about a quarter of the time, about the same accuracy as a generic chatbot handed only basic demographics. The twins did capture real within-population variation better than the demographic baseline, but on absolute accuracy they offered no meaningful lift.
The underlying survey data, drawn from prior work by the same team, covers more than 2,000 U.S. respondents answering 500-plus questions spanning demographics, religion, politics, personality, spending, math, and vocabulary. Each individual's full profile was fed to an LLM with a "respond as this person" prompt; no fine-tuning was described.
Lead author Olivier Toubia, a computational social scientist at Columbia Business School, characterized the result as "overall a bit disappointing." In a companion preprint the team frames the twins as a "funhouse mirror" that distorts the views of the people it tries to copy, a working baseline for the next wave of personalized-subject claims, and a reason to ask whether the pitch is worth its cost premium.