Journal Article

RS-LLaVA: A Large Vision-Language Model for Joint Captioning and Question Answering in Remote Sensing Imagery

Ricci, Riccardo, Bazi, Yakoub, Al Rahhal, Mohamad Mahmoud, Bashmal, Laila, Melgani, Farid

2024

Overview

In this paper, we delve into the innovative application of large language models (LLMs) and their extension, large vision-language models (LVLMs), in the field of remote sensing (RS) image analysis. We particularly emphasize their multi-tasking potential with a focus on image captioning and visual question answering (VQA). In particular, we introduce an improved version of the Large Language and Vision Assistant Model (LLaVA), specifically adapted for RS imagery through a low-rank adaptation approach. To evaluate the model performance, we create the RS-instructions dataset, a comprehensive benchmark dataset that integrates four diverse single-task datasets related to captioning and VQA. The experimental results confirm the model’s effectiveness, marking a step forward toward the development of efficient multi-task models for RS image analysis.

• Publisher: MDPI AG
• Subject: captioning / Data analysis / data collection / Datasets / Image analysis / Image processing / Image retrieval / instruction tuning / Language / Large Language and Vision Assistant Model (LLaVA) / Large language models / large language models (LLMs) / Machine vision / Methods / model validation / Multitasking / Natural language / Neural networks / Question-answering systems / Questions / Remote sensing / remote sensing (RS) / Semantics / Task analysis / Vision / visual question answering (VQA)