Journal Article

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook

Sheikh, Javaid, Renault, Max-Antoine, Damseh, Rafat, Boughorbel, Sabri, Abd-alrazaq, Alaa, AlSaad, Rawan, Ahmed, Arfan

2024

Overview

In the complex and multidimensional field of medicine, multimodal data are prevalent and crucial for informed clinical decisions. Multimodal data span a broad spectrum of data types, including medical images (eg, MRI and CT scans), time-series data (eg, sensor data from wearable devices and electronic health records), audio recordings (eg, heart and respiratory sounds and patient interviews), text (eg, clinical notes and research articles), videos (eg, surgical procedures), and omics data (eg, genomics and proteomics). While advancements in large language models (LLMs) have enabled new applications for knowledge retrieval and processing in the medical field, most LLMs remain limited to processing unimodal data, typically text-based content, and often overlook the importance of integrating the diverse data modalities encountered in clinical practice. This paper aims to present a detailed, practical, and solution-oriented perspective on the use of multimodal LLMs (M-LLMs) in the medical field. Our investigation spanned M-LLM foundational principles, current and potential applications, technical and ethical challenges, and future research directions. By connecting these elements, we aimed to provide a comprehensive framework that links diverse aspects of M-LLMs, offering a unified vision for their future in health care. This approach aims to guide both future research and practical implementations of M-LLMs in health care, positioning them as a paradigm shift toward integrated, multimodal data–driven medical practice. We anticipate that this work will spark further discussion and inspire the development of innovative approaches in the next generation of medical M-LLM systems.

• Publisher: Journal of Medical Internet Research,Gunther Eysenbach MD MPH, Associate Professor,JMIR Publications
• Subject: Application / Artificial intelligence / Chatbots / Clinical medicine / Clinical research / Computational linguistics / Computerized medical records / CT imaging / Data / Data processing / Delivery of Health Care - trends / Electronic Health Records / Ethical dilemmas / Ethics / Forecasts and trends / Future / Genomics / Health care / Health records / Health services / Humans / Innovations / Language / Language modeling / Language processing / Large language models / Magnetic resonance imaging / Medical care / Medical decision making / Medical records / Medical research / Medicine / Multimodality / Natural language interfaces / Natural Language Processing / Neural networks / Patient assessment / Positioning / Retrieval / Technology application / Tomography / United States / Viewpoint