Most AI systems built for pathology are single-purpose: one model detects prostate cancer, another flags breast lymph node metastasis, and a third handles something else entirely. Every new clinical question means rebuilding from scratch. PRISM2, a collaboration between Microsoft Research and Paige (now part of Tempus), takes a different bet: train one model on the language of pathology itself, and let it answer any question you ask.
One model, many tasks
Researchers from Microsoft Research and Paige developed PRISM2, a pathology foundation model trained on both tissue images and language based on real pathology reports. The key insight is deceptively simple. Pathology is not only a visual discipline -- it is also a language-driven one. So instead of training on images alone, the team fused both modalities from the start.
Built on over 2.3 million whole-slide images and integrated with Microsoft's Phi-3, PRISM2 sets a new benchmark in performance, interpretability, and large language model compatibility. A whole-slide image (WSI) is a gigapixel-scale digital scan of a tissue sample -- the kind a pathologist would examine under a microscope. Processing them at scale is a serious engineering challenge.
How it was built
PRISM2 was developed by building on Virchow2, the company's state-of-the-art pathology foundation model, trained on more than 2.3 million H&E-stained whole-slide images spanning hundreds of thousands of patient cases. Each slide is paired with its corresponding clinical report, grounding the model in the language of real-world diagnosis. H&E staining (hematoxylin and eosin) is the standard way to prepare tissue for microscopy, coloring cell nuclei blue-purple and cytoplasm pink.
Training happened in two stages:
- Stage 1 -- Vision-language alignment: A perceiver-based slide encoder (620M parameters) is trained with a contrastive objective to align whole-slide embeddings with clinical report text encoded by BioGPT. The Phi-3 Mini LLM is kept frozen but the autoregressive loss is kept active to pre-align the adapter.