Anthropic Cofounder Told the Pope AI Labs Cannot Govern Themselves. The Vatican Listened.
When Christopher Olah, the cofounder of Anthropic, stood beside Pope Leo XIV at the Vatican on Monday as the pontiff unveiled his first encyclical on artificial intelligence, he delivered something the AI industry rarely produces publicly: an admission that the people building the technology cannot be trusted to govern it alone.
Reuters reported that Olah said at the event, one of the opening presentations before the Pope spoke: "Every frontier AI lab operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing." He also said, "The development of artificial intelligence cannot be left solely to technology companies," urging greater oversight from religious leaders, governments, and civil society.
The Vatican heard him. Reuters reported Pope Leo's encyclical, Magnifica Humanitas, signed May 15 and released Monday in roughly 43,000 words, calls for slowing AI development, worker protections against displacement at scale, data ownership not left solely in private hands, a cooling of competition between AI companies, and a ban on AI systems making lethal decisions. It does not name Anthropic as a co-author. But it arrives precisely at the moment Anthropic — the company that built its brand on being the industry's conscience — is navigating a federal blacklist, a $1.5 billion copyright settlement, and an escalating geopolitical row over its technology's use.
The accountability paradox at the center of Monday's event is hard to miss. Anthropic spent years positioning itself as the frontier lab with a conscience: refusing autonomous weapons contracts, refusing mass surveillance, cultivating relationships with theologians, ethicists, and now the Pope. Its cofounder just told the world's most powerful religious institution that those same labs — including his own — cannot be the ones to set the rules.
Pope Leo's encyclical provides a moral framework for outside AI governance. Whether it changes anything depends on what governments do next.
The Vatican's involvement is not accidental. Anthropic has been building toward this moment for years. Religion News Service reported Anthropic has participated in Vatican-Minerva Dialogues alongside Eric Schmidt, Sam Altman, and Demis Hassabis since at least 2016, according to Brian Patrick Green, director of technology ethics at Santa Clara University. NCR reported Father Brendan McGuire, a Silicon Valley priest who helped draft Anthropic's Claude Constitution — the document governing the model's behavior — attended meetings at Anthropic's San Francisco headquarters. Bishop Paul Tighe of the Vatican's own Dicastery for Culture and Education co-authored an earlier Vatican document on AI. The company's ethical positioning has been, by its own account, deliberate and expensive: it cost Anthropic a Pentagon contract, a supply chain risk designation by the Defense Department, and a court fight it is still litigating.
Olah's specific framing at Monday's event — that incentive conflicts inside AI labs make outside scrutiny essential — is new in its directness. It is one thing for an AI company to refuse a specific contract on ethical grounds. It is another for its cofounder to stand at the Vatican and tell the world that the structure of how frontier AI companies are organized makes them systematically incapable of being their own check. The encyclical does not quite echo that language verbatim. But its calls for slowing AI development, protecting workers displaced at scale, and banning AI-lethal decisions land in the same territory.
Religion News Service reported David Sacks, the AI czar in the Trump administration, accused Anthropic last month of running what he called a regulatory capture strategy — using the language of safety and external oversight to entrench a competitive position that benefits the company itself. The encyclical, whatever its provenance, gives that accusation a new dimension: if frontier AI labs are genuinely incapable of governing themselves, then the debate over who governs them is not abstract. It is a fight over which external institutions get to define the terms.
The encyclical is a teaching document, not a regulation. What it changes depends on whether anyone with actual enforcement power decides to use it.