Journal Article

Health system-scale language models are all-purpose prediction engines

Flores, Mona, Kondziolka, Douglas, Yang, Grace, Laufer, Ilya, Cho, Kyunghyun, Weiss, Hannah, Oermann, Eric Karl, Eaton, Kevin, Schnurman, Zane, Costa, Anthony B., Kurland, David, Liu, Xujin Chris, Kim, Nora C., Cao, Ming, Dastagirzada, Yosef, Wang, Duo, Cheung, Alexander T. M., Jiang, Lavender Yao, Livia, Christopher, Nasir-Moin, Mustafa, Riina, Howard Antony, Miceli, Madeline, Punjabi, Paawan, Neifert, Sean, Aphinyanaphongs, Yindalon, Nejatian, Nima Pour, Abidin, Anas, Orillac, Cordelia

2023

Overview

Physicians make critical time-constrained decisions every day. Clinical predictive models can help physicians and administrators make decisions by forecasting clinical and operational events. Existing structured data-based clinical predictive models have limited use in everyday practice owing to complexity in data processing, as well as model development and deployment 1 – 3 . Here we show that unstructured clinical notes from the electronic health record can enable the training of clinical language models, which can be used as all-purpose clinical predictive engines with low-resistance development and deployment. Our approach leverages recent advances in natural language processing 4 , 5 to train a large language model for medical language (NYUTron) and subsequently fine-tune it across a wide range of clinical and operational predictive tasks. We evaluated our approach within our health system for five such tasks: 30-day all-cause readmission prediction, in-hospital mortality prediction, comorbidity index prediction, length of stay prediction, and insurance denial prediction. We show that NYUTron has an area under the curve (AUC) of 78.7–94.9%, with an improvement of 5.36–14.7% in the AUC compared with traditional models. We additionally demonstrate the benefits of pretraining with clinical text, the potential for increasing generalizability to different sites through fine-tuning and the full deployment of our system in a prospective, single-arm trial. These results show the potential for using clinical language models in medicine to read alongside physicians and provide guidance at the point of care. A clinical language model trained on unstructured clinical notes from the electronic health record enhances prediction of clinical and operational events.

• Publisher: Nature Publishing Group UK,Nature Publishing Group
• Subject: 639/705/1042 / 692/308/575 / Area Under Curve / Artificial intelligence / Clinical Decision-Making - methods / Clinical Trials as Topic / Comorbidity / Data processing / Datasets / Decision making / Decisions / Electronic Health Records / Electronic medical records / Engines / Hospital Mortality / Hospital systems / Hospitals / Humanities and Social Sciences / Humans / Insurance Coverage / Language / Large language models / Length of Stay / Modelling / Mortality / multidisciplinary / Natural Language Processing / Patient Readmission / Patients / Physicians / Physiology / Point-of-Care Systems - trends / Prediction models / Predictions / Predictive analytics / Science / Science (multidisciplinary) / Structured data