Journal Article

Animal-CLIP: A Dual-Prompt Enhanced Vision-Language Model for Animal Action Recognition

Li, Yongxiang, Ma, Zhanyu, Liang, Kongming, He, Zhongjiang, Jing, Yinuo, Zhang, Ruxu, Sun, Hao

2025

Overview

Animal action recognition has a wide range of applications. With the rise of visual-language pretraining models (VLMs), new possibilities have emerged for action recognition. However, while current VLMs perform well on human-centric videos, they still struggle with animal videos. This is primarily due to the lack of domain-specific knowledge during model training and more pronounced intra-class variations compared to humans. To address these issues, we introduce Animal-CLIP, a specialized and efficient animal action recognition framework built upon existing VLMs. To address the lack of domain-specific knowledge in animal actions, we leverage the extensive expertise of large language models (LLMs) to automatically generate external prompts, thereby expanding the semantic scope of labels and enhancing the model’s generalization capability. To effectively integrate external knowledge into the model, we propose a knowledge-enhanced internal prompt fine-tuning approach. We design a text feature refinement module to reduce potential recognition inconsistencies. Furthermore, to address the high intra-class variation in animal actions, a novel category-specific prompting method is introduced to generate adaptive prompts to optimize the alignment between text and video features, facilitating more precise partitioning of the action space. Experimental results demonstrate that our method outperforms six previous action recognition methods across three large-scale multi-species, multi-action datasets and exhibits strong generalization capability on unseen animals.

• Publisher: Springer US,Springer Nature B.V
• Subject: Accuracy / Activity recognition / Animals / Artificial Intelligence / Automation / Biodiversity / Computer Imaging / Computer Science / Computer vision / Datasets / Ecosystems / Enhanced vision / Image Processing and Computer Vision / Knowledge / Language / Large language models / Pattern Recognition / Pattern Recognition and Graphics / Prompt engineering / Semantics / Video / Vision