Clinical AI can run the code. It just can't finish the analysis.

On 126 long-horizon clinical tasks drawn from MIMIC patient records, the best of 24 standardized model-scaffold configurations executed every submission but only 56.3% passed the strict scope check. The gap between code that runs and analysis that's correct is the headline finding of ClinLens, a new public benchmark posted to arXiv on July 31, 2026, and the closest thing the field has to a yardstick for clinical data science agents.

Run the code, and the system checks not just the final answer but the cohort selection, the temporal logic across visits, and the artifacts the workflow was supposed to produce. The 44-point spread between execution success and strict pass is what the authors call the runnable-but-wrong failure mode, and it is exactly the kind of error that would slip past a reviewer who only checked whether a notebook ran.

ClinLens draws 200 executable tasks across five linked resources from MIMIC, a widely used credentialed patient-records database: structured electronic health records, clinical notes, electrocardiograms, chest radiographs, and echocardiograms. A 4×5 taxonomy crosses four patient-time scopes (single-visit, longitudinal, comparative, cohort-level) with five analysis capabilities (descriptive statistics, modeling, visualization, cohort building, and operations). The published 126-task subset is what the paper reports on; the 200-task corpus is the broader release.

The result cuts across clinical AI approaches. Five biomedical systems adapted to GPT-4o-mini managed at most 2.9% on the strict scope check. The best of 24 standardized model-scaffold configurations hit 56.3%. A separately configured general-purpose coding agent solved 83 of 126 tasks, beating every purpose-built clinical scaffold in the comparison. That detail is the part of the result clinicians and vendors will want to sit with: the clinical wrapper may be adding friction rather than accuracy, and the underlying code generation is doing most of the real work.

The strict-pass criterion is the load-bearing piece. A submission can produce the right number, on the right cohort, at the right time window, with the required artifacts (cohort definitions, intermediate tables, plots), and the system still fails if any one of those is wrong. A model that picks the right patients but skips a washout period, or that aggregates over the wrong visit, gets marked wrong. The benchmark is, in effect, auditing the clinical reasoning pipeline, not just the final number.

The coding agent's edge matters because it is the natural counterargument to the clinical-AI label. If a vanilla code-generation setup outscores systems that wrap large language models with medical context, retrieval, and tool scaffolds, then either the scaffolds are not yet pulling their weight, or the tasks in ClinLens are closer to data engineering than to clinical judgment. Both readings are uncomfortable, and the benchmark now makes them falsifiable: a public task corpus, a transparent scoring rubric, and a fixed suite that anyone can rerun.

The honest limit is that MIMIC data is credentialed. Reproducing these numbers on local hospital records, or running these agents against a live electronic health record, is a separate exercise with its own governance, mapping, and validation cost. The benchmark scores the analysis; it does not score the deployment.

What ClinLens does give the field is a target. The runnable-versus-correct gap is now named, measured, and reproducible, which is what earlier clinical-AI evaluations often lacked. The next generation of clinical agents has a concrete number to beat, a public task suite to run on, and a failure mode (wrong cohort, wrong window, missing artifact) that is more actionable than a generic accuracy score. Whether the next round of clinical scaffolds closes the 44-point gap is the question worth tracking, and the benchmark is now the place to watch it.