Paige and Microsoft built a pathology model that writes the report instead of scoring the slide

Paige and Microsoft, working with Memorial Sloan Kettering, have built a pathology model that does not score a slide into one of N categories. Trained on 2.3 million whole-slide images paired with 685,507 pathology reports, the model produces the language a pathologist would write about a slide rather than a probability vector over tumor types.

A whole-slide image is a high-resolution scan of a tissue sample, often gigapixels in size, and the report is the narrative the pathologist writes after staring at that scan for hours. PRISM2, released as a joint Paige and Microsoft research model, was built to pull those two together at a scale most academic groups cannot reach.

Two losses push a single slide encoder at once. The encoder, a perceiver-based module that aggregates Virchow2 tile embeddings into one slide-level vector, runs under a contrastive loss that pulls its output toward BioGPT text embeddings and an autoregressive loss that forces the same vector to decode into Phi-3 Mini tokens. The first teaches the encoder to live in the same neighborhood as clinical prose. The second makes that prose recoverable from a slide alone. Stage one ends with one slide vector that supports both a biomarker-style classifier and a language model.

Stage two freezes the encoder and fine-tunes the language model on single-turn Q&A pairs derived from MSK reports by GPT-4o. Each training example is one slide, one question, one answer. The pairs are not a chat log. A pathologist can ask "what is the grade and what is the mitotic count" and get a written response, but the model has no memory of the prior question, no follow-up turn, and no conversational repair when its first answer is wrong. That is the single-turn cap the source paper names.

The release exposes two embeddings. The "base" embedding comes straight from the slide encoder and is meant to transfer to downstream tasks such as biomarker prediction. The "diagnostic" embedding is pulled from the language model's hidden state after the model processes the slide latents alone, without a text prompt. The split lets researchers use PRISM2 as a feature extractor for cancer-genomics work while also using its language head for report-style generation.

The dialogue supervision is single-turn only. The 685,507 reports all come from MSK, one institution with one reporting style. No clinical-outcome data appears in the paper. A model trained on one hospital's prose will absorb that prose; whether its outputs generalize to a hospital that writes reports differently, or to a patient population that is not MSK's, is the open engineering question.

Paige released the model on HuggingFace, the preprint is on arXiv, and the peer-reviewed version is in Nature Medicine. The next test is whether the language head produces outputs pathologists at other institutions would agree with, and whether the base embedding still transfers to held-out biomarker tasks. Neither question is answered by the current release.