Paper

Near-realtime Facial Animation by Deep 3D Simulation Super-Resolution

Cong, Matthew, Museth, Ken, Park, Hyojoon, Swartz, Jonathan, Sifakis, Eftychios, Sangeetha Grama Srinivasan, Kim, Doyub, Kim, Byungsoo

2024

Overview

We present a neural network-based simulation super-resolution framework that can efficiently and realistically enhance a facial performance produced by a low-cost, realtime physics-based simulation to a level of detail that closely approximates that of a reference-quality off-line simulator with much higher resolution (26x element count in our examples) and accurate physical modeling. Our approach is rooted in our ability to construct - via simulation - a training set of paired frames, from the low- and high-resolution simulators respectively, that are in semantic correspondence with each other. We use face animation as an exemplar of such a simulation domain, where creating this semantic congruence is achieved by simply dialing in the same muscle actuation controls and skeletal pose in the two simulators. Our proposed neural network super-resolution framework generalizes from this training set to unseen expressions, compensates for modeling discrepancies between the two simulations due to limited resolution or cost-cutting approximations in the real-time variant, and does not require any semantic descriptors or parameters to be provided as input, other than the result of the real-time simulation. We evaluate the efficacy of our pipeline on a variety of expressive performances and provide comparisons and ablation experiments for plausible variations and alternatives to our proposed scheme.

• Publisher: Cornell University Library, arXiv.org
• Subject: Ablation / Actuation / Animation / Modelling / Neural networks / Real time / Semantics / Simulation / Simulators / Training