Paper

Intelligent Video Editing: Incorporating Modern Talking Face Generation Algorithms in a Video Editor

Jawahar, C V, Mukhopadhyay, Rudrabha, Gupta, Anchit, Faizan Farooq Khan, Namboodiri, Vinay P

2021

Overview

This paper proposes a video editor based on OpenShot with several state-of-the-art facial video editing algorithms as added functionalities. Our editor provides an easy-to-use interface to apply modern lip-syncing algorithms interactively. Apart from lip-syncing, the editor also uses audio and facial re-enactment to generate expressive talking faces. The manual control improves the overall experience of video editing without missing out on the benefits of modern synthetic video generation algorithms. This control enables us to lip-sync complex dubbed movie scenes, interviews, television shows, and other visual content. Furthermore, our editor provides features that automatically translate lectures from spoken content, lip-sync of the professor, and background content like slides. While doing so, we also tackle the critical aspect of synchronizing background content with the translated speech. We qualitatively evaluate the usefulness of the proposed editor by conducting human evaluations. Our evaluations show a clear improvement in the efficiency of using human editors and an improved video generation quality. We attach demo videos with the supplementary material clearly explaining the tool and also showcasing multiple results.

• Publisher: Cornell University Library, arXiv.org
• Subject: Algorithms / Editing / Editors / Manual control / Synchronism