Journal Article

Unleashing the Power of Contrastive Learning for Zero-Shot Video Summarization

Pang, Zongshang, Nakashima, Yuta, Nagahara, Hajime, Otani, Mayu

2024

Overview

Video summarization aims to select the most informative subset of frames in a video to facilitate efficient video browsing. Past efforts have invariantly involved training summarization models with annotated summaries or heuristic objectives. In this work, we reveal that features pre-trained on image-level tasks contain rich semantic information that can be readily leveraged to quantify frame-level importance for zero-shot video summarization. Leveraging pre-trained features and contrastive learning, we propose three metrics featuring a desirable keyframe: local dissimilarity, global consistency, and uniqueness. We show that the metrics can well-capture the diversity and representativeness of frames commonly used for the unsupervised generation of video summaries, demonstrating competitive or better performance compared to past methods when no training is needed. We further propose a contrastive learning-based pre-training strategy on unlabeled videos to enhance the quality of the proposed metrics and, thus, improve the evaluated performance on the public benchmarks TVSum and SumMe.

• Publisher: MDPI AG,MDPI
• Subject: Contrastive learning / Deep learning / Frames (data processing) / Image processing / Machine learning / Methods / Neural networks / Performance evaluation / Semantics / Summaries / Video data / Video production / video summarization / visual pre-training