When medical AI hears "no pneumonia," it hears "pneumonia": a new benchmark for chest X-ray search

A clinician types "atelectasis and no pneumonia" into a chest X-ray search bar. The system, trained to match words to images, hands back X-rays showing pneumonia anyway.

That single failure is the news. It is also the failure case the team behind CXR-Retrieve built the test to expose: a new public benchmark of 5,159 chest X-rays and 145 carefully written queries, designed to grade whether a retrieval system actually follows the full clinical constraint in a query, including the "no."

Chest X-rays are one of the highest-volume imaging studies in any hospital, and retrieval systems, the AI search bars clinicians and residents use to find prior cases that look like the one in front of them, are already a quiet piece of clinical infrastructure. Case review, teaching files, and triage support all depend on typing a description and getting back the right images. The team's starting point is CXR-CLIP, a chest X-ray-tuned descendant of the same contrastive image-text approach that powers general image search.

The CXR-Retrieve team argues the dominant approach has a blind spot. Its training objective rewards an image and a sentence for sharing the same words; it does not reward them for honoring every constraint the sentence actually states. "Atelectasis and no pneumonia" is a search for partial lung collapse with pneumonia ruled out, and it shares the words "atelectasis" and "pneumonia" with both an X-ray that has only atelectasis and an X-ray that has both findings. Under the standard objective, those pairs look almost equally good. In a clinical workflow, they are not remotely equivalent.

The benchmark's 145 queries were written to test that exact difference. They include single-finding queries, conjunction queries that demand two findings at once, and, most pointedly, negation queries that demand one finding while explicitly ruling out another. The relevance judgment is whether the retrieved image satisfies every constraint the query asserts, not whether the image's report mentions the same words as the query. The 5,159 test images are drawn from the official test split of MIMIC-CXR-JPG, a public chest X-ray corpus.

When the team put CXR-CLIP through the new test, it failed in predictable ways. The model's Precision@5, the share of the top five returned X-rays that actually satisfy the query, drops sharply on conjunction queries and collapses on negation. On two-pathology conjunctions, the team's label-aware fine-tuning lifts Precision@5 by 8.5 percentage points over the CXR-CLIP baseline. On negation queries, the same method lifts it by 22.0 percentage points. The result is a training-objective fix that takes the public CXR-CLIP checkpoint and teaches it to push image-text pairs apart when their asserted pathology lists conflict, even when one of those lists is "X is confirmed absent."

A standard contrastive training run asks the model to pull matched image-text pairs together in its internal representation and push mismatched pairs apart. The label-aware variant adds a second signal: it inspects the structured pathology labels attached to each image and each query, and explicitly repels pairs whose labels contradict each other. "Atelectasis confirmed, pneumonia confirmed absent" should be far from "atelectasis confirmed, pneumonia confirmed present," even though the natural-language text in both cases looks similar to a standard contrastive training pair.

The fix matters because negation is the part of clinical language that retrieval systems cannot afford to fumble. A teaching file query like "consolidation without pleural effusion" is meaningless if the system happily returns X-rays that do show an effusion. A case-review query like "cardiomegaly, no edema" pulls the wrong cases into the differential if the system ignores the "no." These are routine phrasings, not edge cases for the field.

Whether the broader chest X-ray retrieval community, including vendors, hospital AI teams, and the open CXR-CLIP derivative ecosystem, actually starts reporting against CXR-Retrieve is the open question. Without a public yardstick, every "new medical AI" announcement can keep skipping the queries that break it.