Journal Article

Video-driven speaker-listener generation based on Transformer and neural renderer

Shao, Zhengxi, Chen, Jifeng, Liu, Qiong, Yang, Daowu, Jiang, Wen, Yang, Qi

2024

Overview

The traditional speaker-centric synthesis methods prioritize language accuracy but overlook emotional connection and feedback mechanisms with the audience. This paper is dedicated to an in-depth exploration of responsive speaker-listener generation, aiming to enhance communication by providing real-time non-verbal feedback such as head movements and facial expressions. Driven by video, we extract 3DMM coefficients to model facial features and head poses. Combining this with a Transformer speech encoder extracting 45-dimensional acoustic features, we achieve speaker generation at the sentence level. For responsive listener generation, we introduce two attention mechanisms in the Transformer decoder: cross-modal multi-head attention aligning audio-motion modalities and biased causal self-attention suitable for longer audio sequences. Finally, by aligning audio with a behavioral model and optimizing an enhanced neural renderer for facial images, we successfully achieve precise control over facial movements. Extensive experiments demonstrate the superiority of our approach compared to existing technologies.

• Publisher: Springer US,Springer Nature B.V
• Subject: Computer Communication Networks / Computer Science / Data Structures and Information Theory / Feature extraction / Feedback / Head movement / Image enhancement / Listening / Multimedia / Multimedia Information Systems / Neural networks / Real time / Renderers / Speaking / Special Purpose and Application-Based Systems / Speech encoders / Speech recognition / Track 6: Computer Vision for Multimedia Applications / Transformers / Verbal communication