Journal Article

A visual-language foundation model for computational pathology

Chen, Bowen, Chen, Richard J., Liang, Ivy, Parwani, Anil V., Lu, Ming Y., Williamson, Drew F. K., Ding, Tong, Zhang, Andrew, Mahmood, Faisal, Odintsov, Igor, Jaume, Guillaume, Le, Long Phi, Gerber, Georg

2024

Overview

The accelerated adoption of digital pathology and advances in deep learning have enabled the development of robust models for various pathology tasks across a diverse array of diseases and patient cohorts. However, model training is often difficult due to label scarcity in the medical domain, and a model’s usage is limited by the specific task and disease for which it is trained. Additionally, most models in histopathology leverage only image data, a stark contrast to how humans teach each other and reason about histopathologic entities. We introduce CONtrastive learning from Captions for Histopathology (CONCH), a visual-language foundation model developed using diverse sources of histopathology images, biomedical text and, notably, over 1.17 million image–caption pairs through task-agnostic pretraining. Evaluated on a suite of 14 diverse benchmarks, CONCH can be transferred to a wide range of downstream tasks involving histopathology images and/or text, achieving state-of-the-art performance on histology image classification, segmentation, captioning, and text-to-image and image-to-text retrieval. CONCH represents a substantial leap over concurrent visual-language pretrained systems for histopathology, with the potential to directly facilitate a wide array of machine learning-based workflows requiring minimal or no further supervised fine-tuning. Developed using diverse sources of histopathology images, biomedical text and over 1.17 million image–caption pairs, evaluated on a suite of 14 diverse benchmarks, a visual-language foundation model achieves state-of-the-art performance on a wide array of clinically relevant pathology tasks.

• Publisher: Nature Publishing Group US,Nature Publishing Group
• Subject: 631/114/1305 / 692/308 / 692/700/139/422 / Arrays / Benchmarks / Biomedical and Life Sciences / Biomedicine / Cancer Research / Deep learning / Histology / Histopathology / Humans / Image classification / Image contrast / Image processing / Image segmentation / Infectious Diseases / Language / Machine Learning / Medical imaging / Metabolic Diseases / Molecular Medicine / Neurosciences / Pathology / Workflow