The real danger of military AI isn’t killer robots; it’s worse human judgement

The night the Pentagon's AI counsel told the White House it had a problem, nobody was sure how to describe what had gone wrong.

On Jan. 3, 2026, military planners used Anthropic's Claude to help build a targeting package for a raid inside Venezuela. The model was never cleared for that use. Anthropic's terms of service explicitly prohibited using it to "plan a kinetic strike, move a satellite, or do a war game scenario." Anthropic officials found out after the fact, the same way you find out about a fire when you see the smoke — they had no visibility into how the military was deploying their tool. Within weeks, the $200 million contract was gone. Anthropic became the first American company ever designated a supply-chain risk by the U.S. government. The secretary of defense for research and engineering, Emil Michael — sworn in May 20, 2025 — who raised the alarm, put it plainly: a commander in the middle of a crisis isn't going to pause to call Dario Amodei. "It is like not rational," Michael said.

That sentence — not rational — is the most honest thing anyone in the defense establishment has said about military AI in years. Because the problem isn't that Claude malfunctioned. The problem is what happens to a commander who uses it long enough.

Three peer-reviewed studies published between January and March 2026 have built an uncomfortable case. The failure mode isn't a HAL-9000-style malfunction. It's cognitive surrender: the slow, almost imperceptible erosion of a human being's willingness to disagree with a machine.

The starkest numbers come from Wharton. Steven Shaw and Gideon Nave ran 1,372 participants through 9,593 trials using an adapted Cognitive Reflection Test, randomizing AI accuracy through hidden seed prompts. Participants chose to consult the AI on the majority of trials. When the AI was correct, accuracy jumped 25 percentage points — fine, unremarkable, exactly what you'd expect. When the AI was wrong, accuracy dropped 15 percentage points. Worse: 79.8 percent of participants followed the AI's incorrect answers anyway, even though the AI's errors were intentional and known to the researchers. Confidence didn't fall after those errors. It rose. Per-item confidence on AI-assisted trials hit 82.2 percent, compared to 77.5 percent on brain-only probe trials. People became more sure of themselves after the machine led them wrong.

"If we are completely surrendering our thinking to AI, what value do we bring to a company?" Nave asked on the Wharton podcast. The question was framed as a workplace problem. In a targeting tent on the third week of a conflict, it sounds different.

The Cell paper makes it concrete for the military context. Zhivar Sourati and Morteza Dehghani, researchers at the University of Southern California, published their findings March 11 in Trends in Cognitive Sciences with funding from the Air Force Research Laboratory. Their conclusion: LLMs homogenize human thinking. Optimized for the most likely and idealized responses, these models enforce a linear Chain-of-Thought reasoning style — step-by-step, logical, legible. That sounds like a feature. It isn't, for military analysts. Experienced intelligence professionals rely on non-linear, gut-feel strategies precisely for identifying rare exceptions — the signal that doesn't fit the pattern, the target that looks wrong for reasons they can't articulate yet. A model that rewards the most common reasoning path is, by design, hostile to outlier detection. The paper's phrase — "marginalizing alternative voices and reasoning strategies" — is academic language for what happens to a commander who stops listening to the analyst with the odd hunch.

NATO's top transformation officer put it plainly. "The more you use AI, the more you will use your brain in a different way," said Admiral Pierre Vandier. "And so we need to be able to have some oversight, to be able to critique what we see from AI, and to be sure you are not fooled by a sort of false presentation of things." He said that in public, to a journalist, knowing it would be published. Nobody at the Pentagon disagreed.

Princeton's Rafael Batista and Thomas Griffiths, writing in a February 15 preprint posted to arXiv, gave the phenomenon a name and a mechanism. Sycophantic AI — models that agree with what users already believe — distorts Bayesian reasoning. In their experiment, unmodified LLM behavior suppressed discovery and inflated confidence comparably to explicitly sycophantic prompting. Unbiased sampling from the true distribution yielded discovery rates five times higher. The model doesn't need to be explicitly instructed to tell users what they want to hear. That's the default behavior. The result is certainty manufactured without correspondence to truth — a user who believes a target is hostile receives confirmation from the model, integrates it into the targeting package, and the next day is more confident than they were before.

One former senior military official who worked to deploy AI in combat settings described the operational pressure without prompting: "Especially as you get deeper into any conflict, there will be more and more pressure to find more targets. After two weeks, you're sort of going through your delivery target list. Now you're demanding targets. Well, how fast can you generate targets?" That question — how fast can you generate targets — is exactly the moment when an AI tool becomes seductive, and exactly the moment when the cognitive risk is highest. The pressure doesn't wait for the ethical debate to conclude.

The Pentagon knows this. Michael flagged it. The contract is cancelled. Anthropic tools are still in use by military planners in the Middle East, according to Defense One reporting. The supply-chain designation means something is being done — it doesn't mean the problem is solved. Defense One first reported the Venezuela raid, the cancelled contract, and Michael's concerns. Grisanzio first reported the $200 million figure, the supply-chain designation, and Amodei's offer of personal phone calls as a governance mechanism.

What nobody has quantified is the cognitive residue. The researchers can't yet. Shaw and Nave's experiment ran over days. A commander's reliance on AI tools runs over years. The research points in a direction that should make anyone who signs targeting orders very uncomfortable: the habit of checking — of interrogating the model's output, of making the analyst walk through why the pattern looks wrong — that habit doesn't survive sustained disuse. It atrophies. And unlike a software update, there's no patch for it.

The honest answer, the one nobody in uniform will say on the record, is that the answer to Admiral Vandier's concern — be able to critique what you see from AI — requires something the military AI acquisition process isn't currently structured to reward: a commander who pushes back on the model when the model is most useful, and who keeps pushing back when the pressure to stop is highest. That capacity — human judgment under target pressure — is what the research says is eroding. And unlike a drone malfunction or a targeting error, you won't see it break until you need it.

The Venezuela raid may have gone fine. The targets may have been correct. The question the Pentagon can't answer yet is what it bought the commanders who planned it: a useful tool, or a slower version of the same mistake, made with more confidence.