OT-Talk: Animating 3D Talking Head with Optimal Transportation

Xinmu Wang, Xiang Gao, Xiyun Song, Heather Yu, Zongfang Lin, Liang Peng, Xianfeng Gu · 2025

Read OT-Talk: Animating 3D Talking Head with Optimal Transportation by Xinmu Wang, Xiang Gao, Xiyun Song, Heather Yu, Zongfang Lin, Liang Peng, Xianfeng Gu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01932.txt.

Description

A talking avatar must move its lips with the voice, and small errors read as wrongness instantly. Xinmu Wang and co-authors animate 3D heads with help from optimal transport, the mathematics of moving mass cheaply. Their system, OT-Talk, listens with a pretrained speech model and sequences the motion with a transformer. Geometry comes from graph convolutions that respect the mesh's triangles.

The key step treats each face shape as a spread of probability over the surface. A sliced Wasserstein distance then measures how far one expression sits from another. Training against that distance yields smoother, better-synced motion than vertex errors alone. Tests on two public datasets beat prior methods, and viewers in a perception study agreed.

Catalog record

CREATORXinmu Wang, Xiang Gao, Xiyun Song, Heather Yu, Zongfang Lin, Liang Peng, Xianfeng Gu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01932

Browse the full library · About the Tower of Babel