Vague AI Health Explanations Are Most Convincing to the People Who Should Trust Them Least

The pitch for explainable medical AI is simple: show the user why the model picked its answer, and they can decide whether to trust it. A new MIT/Stanford study in dermatology finds the mechanism breaks in a specific way. The explanations helped the people least equipped to use them and did nothing for, or actively got in the way of, the people trained to evaluate them.

The study ran non-experts and primary-care clinicians through a series of skin-disease diagnoses. Half the time they got the AI's bare prediction. Half the time they got the prediction plus a written explanation. When the model was correct, explanations added little. When the model was wrong, non-experts who saw an explanation grew more confident in the wrong answer, not less. They also rated vague or generic explanations as more persuasive than specific ones. A model that hedged with broad language about "lesion characteristics consistent with several conditions" sounded more authoritative than one that pointed to a defined feature, even though the hedge carried less information.

Clinicians, by contrast, were not misled. They performed best when given only the model's prediction with no explanation attached. Adding the explanation did not help them, and in some cases introduced enough noise to drag accuracy down. The pattern, MIT News reports, shows that explanations do not just add information. They transfer authority, and the transfer lands hardest on the users least equipped to evaluate it.

The mechanism, the authors argue, is automation bias. When a model sounds confident, readers anchor on that confidence and stop checking. For a trained clinician, that anchor is one input among many, and the training to question it is built in. For a non-expert, the anchor becomes the answer.

"Those with the least medical knowledge are most likely to be led astray when explainable AI models give an erroneous output," Roxana Daneshjou, an assistant professor of biomedical data science and dermatology at Stanford and a co-author, told MIT News. Her co-author Marzyeh Ghassemi, an associate professor in MIT's Department of Electrical Engineering and Computer Science and a member of the Institute for Medical Engineering and Science, frames the result as a design problem: AI for health needs to be built with the user in mind, including their expertise level, so explanations help calibrate trust rather than manufacture it.

The study, published as a preprint on arXiv and mirrored on medRxiv, is titled "Explainable AI as a Double-Edged Sword in Dermatology: The Impact on Clinicians versus The Public." It has not been peer-reviewed, and the controlled setting means the result is a measurement of how the design interacts with the user, not a verdict on any deployed product.

The design implication matters for the policy debate now live in Washington and Brussels over whether medical AI should be required to explain itself. The argument in favor of mandatory explanations rests on the idea that users can evaluate them. This study says otherwise in a specific case: when the user is the patient, the obligation to show your work can make a wrong answer more dangerous, not less.

The design fix, both researchers argue, is calibration. A vague, authoritative-sounding paragraph is worse for non-experts than a specific, falsifiable one. A clinician might want the model's bare prediction and a way to interrogate it, not a paragraph in between. Consumer-facing apps that bolt on generic explanations to satisfy a transparency requirement may be doing the opposite of what the requirement was meant to do.

The heuristic is portable. The next time a health app explains its reasoning in confident, generic language, the safest response is the one a trained clinician uses in the study: trust the prediction less, not more.