Journal Article

Multimodal fusion: advancing medical visual question-answering

Mudgal, Anjali, Jafari, Amir, Kumar, Aditya, Kush, Udbhav

2024

Overview

This paper explores the application of Visual Question-Answering (VQA) technology, which combines computer vision and natural language processing (NLP), in the medical domain, specifically for analyzing radiology scans. VQA can facilitate medical decision-making and improve patient outcomes by accurately interpreting medical imaging, which requires specialized expertise and time. The paper proposes developing an advanced VQA system for medical datasets using the Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (BLIP) architecture from Salesforce, leveraging deep learning and transfer learning techniques to handle the unique challenges of medical/radiology images. The paper discusses the underlying concepts, methodologies, and results of applying the BLIP architecture and fine-tuning approaches for VQA in the medical domain, highlighting their effectiveness in addressing the complexities of VQA tasks for radiology scans. Inspired by the BLIP architecture from Salesforce, we propose a novel multi-modal fusion approach for medical VQA and evaluating its promising potential.

• Publisher: Springer London,Springer Nature B.V
• Subject: Artificial Intelligence / Computational Biology/Bioinformatics / Computational Science and Engineering / Computer Science / Computer vision / Data Mining and Knowledge Discovery / Datasets / Deep learning / Electronic health records / Image Processing and Computer Vision / Language / Literature reviews / Medical imaging / Natural language / Natural language processing / Neural networks / Original Article / Probability and Statistics in Computer Science / Questions / Radiology / Sales management / Task complexity