Journal Article

Vision–language foundation model for echocardiogram interpretation

Vukadinovic, Milos, Yuan, Neal, Christensen, Matthew, Ouyang, David

2024

Overview

The development of robust artificial intelligence models for echocardiography has been limited by the availability of annotated clinical data. Here, to address this challenge and improve the performance of cardiac imaging models, we developed EchoCLIP, a vision–language foundation model for echocardiography, that learns the relationship between cardiac ultrasound images and the interpretations of expert cardiologists across a wide range of patients and indications for imaging. After training on 1,032,975 cardiac ultrasound videos and corresponding expert text, EchoCLIP performs well on a diverse range of benchmarks for cardiac image interpretation, despite not having been explicitly trained for individual interpretation tasks. EchoCLIP can assess cardiac function (mean absolute error of 7.1% when predicting left ventricular ejection fraction in an external validation dataset) and identify implanted intracardiac devices (area under the curve (AUC) of 0.84, 0.92 and 0.97 for pacemakers, percutaneous mitral valve repair and artificial aortic valves, respectively). We also developed a long-context variant (EchoCLIP-R) using a custom tokenizer based on common echocardiography concepts. EchoCLIP-R accurately identified unique patients across multiple videos (AUC of 0.86), identified clinical transitions such as heart transplants (AUC of 0.79) and cardiac surgery (AUC 0.77) and enabled robust image-to-text search (mean cross-modal retrieval rank in the top 1% of candidate text reports). These capabilities represent a substantial step toward understanding and applying foundation models in cardiovascular imaging for preliminary interpretation of echocardiographic findings. A vision–language foundation model, trained on a dataset of more than 1 million echocardiogram video–text pairs, is able to assess various cardiac structural and functional parameters despite not having been directly trained on any specific image interpretation task.

• Publisher: Nature Publishing Group US,Nature Publishing Group
• Subject: 631/114/2398 / 692/699/75 / Aorta / Artificial Intelligence / Benchmarks / Biomedical and Life Sciences / Biomedicine / Cancer Research / Datasets / Echocardiography / Echocardiography - methods / Heart transplantation / Heart valves / Humans / Image Interpretation, Computer-Assisted / Image processing / Imaging / Infectious Diseases / Language / Metabolic Diseases / Mitral valve / Molecular Medicine / Neurosciences / Patients / Robustness / Structure-function relationships / Transplants / Ultrasonic imaging / Ultrasound / Video