Asset Details
Do you wish to reserve the book?
A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data
Hey, we have placed the reservation for you!
By the way, why not check out events that you can attend while you pick your title.
Oops! Something went wrong.
Looks like we were not able to place the reservation. Kindly try again later.
Are you sure you want to remove the book from the shelf?
A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data
Do you wish to request the book?
A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data
Please be aware that the book you have requested cannot be checked out. If you would like to checkout this book, you can reserve another copy
We have requested the book for you!
Your request is successful and it will be processed during the Library working hours. Please check the status of your request in My Requests.
Oops! Something went wrong.
Looks like we were not able to place your request. Kindly try again later.
Journal Article
A multimodal vision transformer for interpretable fusion of functional and structural neuroimaging data
2024
Overview
Multimodal neuroimaging is an emerging field that leverages multiple sources of information to diagnose specific brain disorders, especially when deep learning‐based AI algorithms are applied. The successful combination of different brain imaging modalities using deep learning remains a challenging yet crucial research topic. The integration of structural and functional modalities is particularly important for the diagnosis of various brain disorders, where structural information plays a crucial role in diseases such as Alzheimer's, while functional imaging is more critical for disorders such as schizophrenia. However, the combination of functional and structural imaging modalities can provide a more comprehensive diagnosis. In this work, we present MultiViT, a novel diagnostic deep learning model that utilizes vision transformers and cross‐attention mechanisms to effectively fuse information from 3D gray matter maps derived from structural MRI with functional network connectivity matrices obtained from functional MRI using the ICA algorithm. MultiViT achieves an AUC of 0.833, outperforming both our unimodal and multimodal baselines, enabling more accurate classification and diagnosis of schizophrenia. In addition, using vision transformer's unique attentional maps in combination with cross‐attentional mechanisms and brain function information, we identify critical brain regions in 3D gray matter space associated with the characteristics of schizophrenia. Our research not only significantly improves the accuracy of AI‐based automated imaging diagnostics for schizophrenia, but also pioneers a rational and advanced data fusion approach by replacing complex, high‐dimensional fMRI information with functional network connectivity, integrating it with representative structural data from 3D gray matter images, and further providing interpretative biomarker localization in a 3D structural space. The MultiViT model combines structural and functional neuroimaging data for the prediction of schizophrenia and integrates vision transformers with cross‐attention layers in order to preserve mutual information. The pipeline generates highly interpretable cross‐attention‐based brain saliency maps and emphasizes functional network connectivity patterns related to the disorder.
Publisher
John Wiley & Sons, Inc
Subject
/ Adult
/ Brain
/ Datasets
/ Female
/ Functional magnetic resonance imaging
/ Functional Neuroimaging - methods
/ Functional Neuroimaging - standards
/ Gray Matter - diagnostic imaging
/ Humans
/ Magnetic Resonance Imaging - methods
/ Magnetic Resonance Imaging - standards
/ Male
/ Multimodal Imaging - methods
/ Schizophrenia - diagnostic imaging
/ Schizophrenia - physiopathology
