When the algorithm hedges: a new recipe for medical AI that knows what it doesn't know

A diagnostic model trained on Hospital A's patients and deployed on Hospital B's doesn't just lose accuracy. It loses honesty. Under distribution shift, when the patient population, scanners, and labeling conventions change, even a well-validated model can return confident wrong answers. The question worth asking is not whether the model is right, but whether the set of plausible diagnoses it returns is small enough to fit on a real differential.

That distinction, between accuracy and calibrated uncertainty, is the operating unit of clinical AI under domain shift. A new arXiv preprint from Duke University proposes a training recipe designed to deliver both at once (Conformal Risk Minimization for Semi-Supervised Domain Adaptation via Optimal Transport, v1, 24 Aug 2026). The method is proposal-stage. It has not been peer reviewed, replicated, or shipped in a clinical system. The skin-lesion classification case the paper uses to illustrate the idea is a benchmark experiment, not a deployed product. Read it as a sharper question to put to any clinical-AI vendor, not a finished answer.

The hard part is not getting a model to learn from a hospital's labeled scans. It is getting it to keep its confidence honest when the next hospital's data looks different. The subfield that studies this problem is called Semi-Supervised Domain Adaptation, or SSDA: a model pre-trains on a source hospital with plenty of labels, then adapts to a target hospital with only a handful of labeled examples and a much larger pool of unlabeled ones. SSDA methods have historically optimized for one thing: point-prediction accuracy. They tell you which diagnosis the model picks, not how sure it is, and not what other diagnoses are still in play.

Conformal Prediction, or CP, is the standard statistical tool for turning a model's raw score into a calibrated "I don't know" set: a short list of possible diagnoses the model is willing to vouch for at a stated confidence level (say, 90% coverage). The catch: standard CP bolts on after training. When the underlying model has been adapted to a new hospital with very few labels, the resulting sets can balloon. A model that returns 30 plausible diagnoses is technically correct and clinically useless. A 2019 line of work called Conformal Risk Minimization, or CRM, showed a way around this in the fully supervised case, where the target domain has plenty of labels to compute nonconformity thresholds from. It could not be used in the SSDA regime, where the target labels are scarce.

The Duke paper's move is to integrate CRM into the SSDA training loop end-to-end, rather than running it as a post-hoc fix. To do that, the authors need pseudolabels for the many unlabeled target examples, and they generate them with Optimal Transport (OT), a classical mathematical tool that finds the cheapest way to map one distribution of patients onto another. With OT-supplied pseudolabels, CRM has enough signal to compute the right nonconformity thresholds during training, so the model is jointly optimized for domain invariance (it works on the new hospital) and conformal efficiency (its uncertainty sets stay compact).

The claimed result, drawn from the paper's own tables and not from independent replication, is a model that produces prediction sets which are compact, coverage-valid, and able to support domain-specific constraints. The paper's illustrative example is a skin-lesion classifier, where the authors show the model can be configured to exclude mutually contradictory diagnoses from the returned set, a feature, not a default, and an opt-in design choice rather than an automatic safety property.

Conformal coverage and set-size claims are conditional on the paper's own evaluation; the experimental tables, available in the arXiv PDF, were not independently verified. Compute cost, training stability, and method complexity are not quantified in the abstract. No code release is mentioned. There is no third-party validation, no clinical trial, and no comparison to specific deployed systems.

What the preprint does do is name the operational unit of clinical AI under domain shift: the size and validity of the model's hedge. The interesting question for any hospital evaluating an AI vendor is no longer just "what is your accuracy on our population?" It is "what does your model return when it is not sure, and is that list small enough to guide a real differential?" Duke's recipe is one answer. It is not yet a deployed one.