A preprint tested four agent configurations on science and economics problems. Diverse specialized models drove the wins; coordination scaffolds and redundant same model sampling barely helped.
A new preprint tested four configurations of multi-agent foundation-model systems on a benchmark spanning calculus, physics, chemistry, biology, economics, optimization, statistics, and mathematics. Its finding inverts the usual pitch for these stacks: it is the variety of models in the ensemble that does the work, not the coordination framework wrapped around them.
The paper, "Collective Intelligence with Foundation Models", isolates the contributions of architecture and diversity using a four-role design. Solver agents produce independent drafts. Critic agents run structured critique and revision. An aggregator synthesizes consensus. A scoring module evaluates the output. That scaffold stays fixed across configurations. What changes is what is inside it.
Four setups go head to head. A single-agent baseline runs alone. A homogeneous framework runs the same model through all four roles. A redundant homogeneous setup adds multiple solver instances of the same model. The heterogeneous framework swaps in diverse specialized models for each role.
The result is consistent across the cross-domain benchmark. Adding more copies of the same model barely helps. Bolting on a richer coordination framework helps modestly. Replacing the homogeneous stack with diverse specialized models produces the large jump, the paper reports in its abstract and ablation setup.
Most multi-agent marketing right now sells the framework, not the model mix. Vendor demos and lab write-ups tend to credit the orchestration layer, the critique-and-revise loop, or the consensus mechanism for whatever gains their systems show. Anthropic's engineering write-up of its research system, for instance, foregrounds how it parallelizes subagents and aggregates findings. The preprint's empirical take suggests that when these stacks are pushed through a careful ablation, the architecture is doing less of the lifting than the marketing implies.
The same question is being attacked from the other end by a separate preprint, "Towards a Science of Scaling Agent Systems". It studies how multi-agent stacks scale with additional compute and agents. Both papers reach a related conclusion: the "more agents = better" intuition is at best incomplete.
The substrate is the lever. Picking models with complementary strengths, different training data, different objectives, different inductive biases, gives a multi-agent system more to work with than adding another copy of the same model. A model tuned for symbolic manipulation can step in where a chat-tuned model stumbles. A model trained on code can salvage a draft that a generalist botched. The framework is plumbing; the diversity is the talent.
There are real limits to the claim. The benchmark covers textbook-style problems in eight domains, not the open-ended research, coding, or customer-facing workflows that production agents are increasingly asked to handle. The paper positions its work as a step toward safer and more reliable AI through cooperative reasoning, but its evaluation does not directly measure that translation. Independent replication on harder benchmarks, and measurement on deployed systems, will matter before the result firms up into a recipe.
What the result does suggest, even with those caveats, is that "buy a fancier coordination framework" is the wrong play for a team trying to get more out of a multi-agent setup. The first question is what each agent in the stack is good at, and whether swapping in a stronger, more specialized model at a given role would do more than rewriting the orchestration layer.
The heterogeneity finding now faces its harder test. Long-horizon research tasks, multi-step coding, real customer queues, and enterprise data-grounded workflows have all become deployment targets. The benchmark in the preprint stops short of measuring them. That gap decides whether the result turns into a recipe.