Revising Second Order Terms in Deep Animation Video Coding

Konstantin Schmidt, Thomas Richter · 2025

Read Revising Second Order Terms in Deep Animation Video Coding by Konstantin Schmidt, Thomas Richter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23561.txt.

Introduction

Konstantin Schmidt and Thomas Richter improve a low-bandwidth way of sending talking-head video, where only a few facial keypoints cross the network. The First Order Motion Model rebuilds a face by warping a still source image, which keeps bitrates tiny and computing needs modest. Its weak spot is strong head movement, since the warping math cannot recreate real rotations well.

Focusing on head rotations, the authors replace the model's local Jacobian warps with one global rotation step. On clips with rotating heads, the change improves picture quality while cutting the data sent for predicted frames by forty to eighty percent. They also add modern normalization tricks to the training critic, which steadies the adversarial training. Scores from the LPIPS and DISTS quality measures confirm the gains.

Catalog record

CREATORKonstantin Schmidt, Thomas Richter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23561

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Revising Second Order Terms in Deep Animation Video Coding Thanks: The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU). The hardware is funded by the German Research Foundation (DFG) Konstantin Schmidt, Thomas Richter Abstract First Order Motion Model is a generative model that animates human heads based on very little motion information derived from keypoints. It is a promising solution for video communication because first it operates at very low bitrate and second its computational complexity is moderate compared to other learning based video codecs. However, it has strong limitations by design. Since it generates facial animations by warping source-images, it fails to recreate videos with strong head movements. This works concentrates on one specific kind of head movements, namely head rotations. We show that replacing the Jacobian transformations in FOMM by a global rotation helps the system to perform better on items with head-rotations while saving 40% to 80% of bitrate on P-frames. Moreover, we apply state-of-the-art normalization techniques to the discriminator to stabilize the adversarial training which is essential for generating visually appealing videos. We evaluate the performance by the learned metics LPIPS and DISTS to show the success our optimizations. I Introduction Recently there has been a lot of progress in the development of learning based video codecs for communication applications. These codecs are able to outperform classical engineered codec standards like e.g. H.264, H.265 or H.266 (AVC [ 1 ] , HEVC [ 2 ] , VVC [ 3 ] ) by quite a margin [ 4 , 5 , 6 ] . Among these learning based codecs two main paths of development can be identified. First are codecs that mimic the classical design of video codecs based on motion estimation and residual coding. These codecs use deep networks to replace the hand-engineered predictors and residual coders. The main drawback of this design is the huge computational complexity caused by the decoder running in the encoder. Such codecs are able to generate videos at bitrates more than 30 % lower than H.266 [ 5 ] . Second are codecs based on First Order Motion Model (FOMM) [ 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 ] that are able to generate appealing videos at moderate computational complexity at bitrates of 3 kbps or even lower. FOMM has less than 60 million parameters and 55 G-MACS complexity per P-frame. These codecs are suited for communication applications only (coding of human faces), while the former are able to code general content. Such codecs use a single image (denoted here as I-frame) that is animated by as low as 10 keypoints (KPs) that are transmitted per video-frame to be coded (denoted here as P-frame). While sometimes being able to generate almost artifact-free videos at bitrates below 3 kbps, such codecs have major limitations by design. One such limitation is strong head rotations on the roll-axis of P-frames, since these items do not occur very often in the training set. As a solution to this problem, we propose to amend or replace the Jacobian transformation of the KPs by a single rotation parameter. We show that this improves the quality of generated videos while also reducing the bitrate of P-frames. The second optimization we propose is targeting the adversarial loss and could be applied to any generative facial animation codec. The basic principle of FOMM is to generate videos by warping I-frames based on some warping information coded in the bitstream. Often the P-frame to generate contains elements that are not present in the I-frame. In such cases the warping fails to generate meaningful content and the model relies on an adversarial loss to hallucinate these parts. The adversarial loss is known to be very unstable and as a result often only a small amount is added to the overall loss. We propose two normalization techniques to stabilize the discriminator which allows for higher amount adversarial loss and thus better image quality. First, to our best knowledge we are the first to use Gradient Normalization [ 23 ] to stabilize the discriminator in generative video coding. Second, inspired by self-supervised learning, we let the discriminator learn facial landmarks while training. In summary our contributions are: • Optimized linear transformation of local patches that form the output frame. • Reduced P-frame bitrate compared to the original FOMM. • A more stable adversarial loss that results in higher image fidelity. II Optimizations of the Transformations for Warping I-frames FOMM generates animated P-frames by warping I-frame areas around so called keypoints (KP). These KPs are outputted by a DNN per video from (see Fig. 2 ). They are similar to facial landmarks (which identify facial elements like eyes, nose, etc.) but are learned unsupervisedly. Fig. 1(a) depicts the difference between KPs and landmarks on a video-frame of an item of the training set. The 68 landmarks generated with [ 24 ] are plotted in red and the KPs are plotted as yellow asterisks. These KPs, together with also unsupervisedly learned local linear transformations matrices called Jacobians (JACs) form the bitstream of the P-frames. (a) Landmarks (red) and Keypoints (KP). (b) Warped patches with one additional patch for the background (bottom center). The warping operation used here is based on Jaderberg et. al. [ 25 ] where the warped output is calculated by the per-pixel displacement information given in the matrix WG (please see [ 25 ] for more details on this). This matrix is calculated as: WG=(WG_{neutral}-KP_{p})\cdot JAC_{i}\cdot JAC_{p}^{-1}+KP_{i}, (1) based on the unitary warping grid WG_{neutral} . JAC_{i} and JAC_{p} are local 2x2 linear transformation matrices of I-frame and P-frame that, together with the KPs allow for affine (i.e. first order) transformations of local patches. Fig. 1(b) (right) depicts such patches. Each patch contains a mutually exclusive part of the input image that is warped and merged by a generator DNN (Fig. 2 ) to form the output image. Please note that Eq. 1 is performed for each of the 10 KPs. After inspecting these local matrix-transformations, we conclude that: • Rotations mostly follow global head rotations on the jaw-axis. • Scalings mostly follow global head scaling. • Transformations like shearing often appear random. • Calculating inverse JAC s as in 1 is often unstable. Especially during training an unstable matrix inversion is suboptimal for robust gradient flow. This motivates us to replace the local transformation matrices JAC_{k} for KP k by a single global rotation and a scaling: JAC_{k}:=R\cdot scf=\begin{bmatrix}\cos\phi&-\sin\phi\\ \sin\phi&\cos\phi\end{bmatrix}\cdot scf, (2) or a combination of a single global rotation, scaling and shearing matrices: \displaystyle JAC_{k} \displaystyle:=R\cdot SHR_{k}\cdot scf (3) \displaystyle=\begin{bmatrix}\cos\phi&-\sin\phi\\ \sin\phi&\cos\phi\end{bmatrix}\cdot\begin{bmatrix}1+\lambda\mu_{k}&\lambda_{k}\\ \mu_{k}&1\end{bmatrix}\cdot scf. The rotation matrix R=\big(\begin{smallmatrix}\cos\phi&-\sin\phi\\ \sin\phi&\cos\phi\end{smallmatrix}\big) has a single parameter \phi that is given by the KP-detection DNN (Fig. 2 ) and learned by a supervised loss from a head-pose estimation loss. We use the model in [ 26 ] as head-pose estimation network which gives a robust estimation of jaw, pitch and roll axis of the head. Here, we only use the roll axis which directly gives

Read the complete text