Journal Article

Exploiting local spatio-temporal characteristics for effective video understanding

Liu Tongcun, Wang, Yulong, Liu Haoxin

2021

Overview

The explosive growth in online video streaming presents challenges for video understanding with high accuracy and low computation complexity. Recent methods have realized global video representation without considering the local spatial structures of the videos over time. In this paper, we propose a method called partial channel fusion (PCF), which exploits local spatio-temporal characteristics for video understanding. We also present an agnostic and effective module for PCF which can provide both high efficiency and high performance in a variety of networks. Rather than independently modeling the spatial structure and motion structure of videos, the PCF module enables information exchange among multiple frames by partially fusing channels over the temporal dimension. By inserting the PCF module into different layers of a 2D convolutional network (2D-convNets), the local and global spatio-temporal characteristics of videos can be captured. Experimental results on two challenging datasets demonstrate the superiority of PCF in improving the accuracy of a 2D-convNets, advancing the state-of-the-art without increasing computational complexity.

• Publisher: Springer Nature B.V
• Subject: Accuracy / Complexity / Deep learning / Image retrieval / Modules / Multimedia / Neural networks / Streaming media / Video transmission