Journal Article

Restore DeepFakes video frames via identifying individual motion styles

Zhang, Haichao, Feng, Ya‐Pei, Lu, Zhe‐Ming, Luo, Hao

2021

Overview

Recent advance on highly realistic AI‐synthesised video approaches makes it hard to distinguish whether a speaker in video is real. Many existing DeepFakes detection approaches can be invalidated by using them as supervision in adversarial training, which in turn improves the performance of DeepFakes. This makes the problem a dilemma. However, human can recognise the identity of familiar persons by observing their motion styles. Inspired by this, the paper proposes a novel method to recognise original speaker's identity clues in DeepFakes videos by learning individual motion styles. As a biological signature, motion styles can neither be used in the training process of DeepFakes nor be modified to deceive detection methods without reducing realistic performance. Also, this paper proposes a novel pipeline to continuously restore the original frames from DeepFakes videos without knowing the DeepFakes approach in advance. Based on above ideas, our scheme makes it possible to restore DeepFakes videos. Through training the cross‐modal transfer module, the appearance embedding can be inferred from the identity code. Extensive experiments are conducted to reveal the effectiveness of the method, it shows for the first time that the original persons can be identified automatically and the original video from DeepFakes videos can be generated.

• Publisher: John Wiley & Sons, Inc,Wiley
• Subject: Biological activity / Computer vision and image processing techniques / Deception / Deepfake / Frames (data processing) / Human motion / Methods / Motion perception / Neural networks / Optical, image and video signal processing / Performance enhancement / Realism / Sensors / Speech and audio signal processing / Speech processing techniques / Video / Video signal processing