Journal Article

Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data

Kuhl, Christiane, Khader, Firas, Hamesch, Karim, Tayebi Arasteh, Soroosh, Han, Tianyu, Müller-Franzes, Gustav, Stegmaier, Johannes, Wang, Tianci, Nebelung, Sven, Haarburger, Christoph, Truhn, Daniel, Kather, Jakob Nikolas, Bressem, Keno

2023

Overview

When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evaluates a transformer-based neural network as a novel AI architecture that integrates multimodal patient data, i.e., imaging data (chest radiographs) and non-imaging data (clinical data). We evaluate the performance of our model in a retrospective study with 6,125 patients in intensive care. We show that the combined model (area under the receiver operating characteristic curve [AUROC] of 0.863) is superior to the radiographs-only model (AUROC = 0.811, p < 0.001) and the clinical data-only model (AUROC = 0.785, p < 0.001) when tasked with predicting in-hospital survival per patient. Furthermore, we demonstrate that our proposed model is robust in cases where not all (clinical) data points are available.

• Publisher: Nature Publishing Group UK,Nature Publishing Group,Nature Portfolio
• Subject: 639/705/117 / 692/700/1421 / 692/700/1421/1770 / Area Under Curve / Critical Care / Diagnostic Imaging / Electric Power Supplies / Electronic health records / Hospitals / Humanities and Social Sciences / Humans / Intensive care / Machine learning / Medical prognosis / multidisciplinary / Neural networks / Patients / Performance evaluation / Physicians / Radiography / Retrospective Studies / Science / Science (multidisciplinary) / Sensory integration / Survival