Conformal prediction adds an "I don't know" candidate set to any classifier. An arXiv preprint finds it held 90% on four chemistry datasets while covering rare toxicity compounds just 4.2% of the time.
A statistical wrapper that gives drug-screening AI an honest "how sure is it?" answer hit its 90% global coverage target on four public chemistry benchmarks while covering rare toxicity compounds just 4.2% of the time, according to a new arXiv preprint by Muhammadjon Tursunbadalov.
The method, conformal prediction, runs a classifier so that for any chosen error rate alpha, it returns a small candidate set guaranteed to contain the right answer at least (1 minus alpha) of the time. The guarantee holds on average. The preprint documents a related failure: when one class is rare, the same method can hit its 90% overall target while effectively abandoning that class.
On a benchmark of whether a molecule crosses the blood-brain barrier, marginal conformal prediction covered 64.8% of the minority class. On clinical-trial toxicity, it covered 4.2%, effectively nothing. The collapse reproduces across a random forest, a graph neural network, and a frozen chemical language model, with p<0.001 in each case according to the paper's reported measurements. Architecture is not the lever. The shortfall tracks how well the underlying model is calibrated on rare labels to begin with.
The paper derives a conservation identity that explains the pattern: the minority class's lost coverage equals the majority's surplus amplified by the class imbalance ratio. That equation predicts the measured gap to within roughly one point across the four benchmarks. The global coverage reading stays at the target because the majority class absorbs the slack. Aggregate accuracy and overall coverage stay "reassuringly high," the authors write, which is why the failure mode is easy to miss in standard reporting (HTML version).
The authors held out compounds by their core molecular scaffold, a benzene or pyridine ring shared with both classes, so the model had to score structures unlike anything in training. The minority gap held, and the conservation identity still ordered the datasets from least to most catastrophic.
Conformal prediction under-covers compounds built on generic scaffolds (plain benzene and pyridine cores that appear in both the positive and negative classes), and the authors propose a one-number diagnostic any practitioner can compute: the realized per-class coverage gap at the chosen alpha. On the four benchmarks, that diagnostic returns the same per-class gap the conservation identity predicts and orders the datasets the same way.
Class-conditional conformal prediction, sometimes called Mondrian conformal prediction, runs a separate reliability calculation for each class. On every benchmark in the paper, it restores per-class coverage to the 90% target with a modest increase in average candidate-set size. The variant has prior use in imbalanced bioactivity data and surfaces in cheminformatics surveys on conformal prediction as a known calibration caveat.
The paper's cost model makes the consequence concrete. With realistic penalties for missed toxic compounds and rewards for true positives, a screening campaign that runs marginal conformal prediction at the 90% target lands net-negative on utility. The same campaign run with the class-conditional variant, or with marginal conformal augmented by an abstention rule on the affected scaffolds, lands net-positive. The result depends on the paper's chosen cost assumptions; the actionable item is the gap between the two scenarios.
Two limits on the study. The paper is a single arXiv preprint, not peer-reviewed, dated DataCite July 7, 2026. The four benchmarks are public chemistry datasets widely used for virtual-screening evaluation, and broader claims about deployed pipelines are outside the paper's scope.
Independent replication is the natural next step. The four chemistry benchmarks are public and the per-class diagnostic is computable on a held-out set, so the test is in reach. Reproducing it would make per-class coverage a default reporting line for conformal prediction in drug discovery.