SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model

Yan Li, Ziya Zhou, Zhiqiang Wang, Wei Xue, Wenhan Luo, Yike Guo · 2024

Read SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model by Yan Li, Ziya Zhou, Zhiqiang Wang, Wei Xue, Wenhan Luo, Yike Guo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03430.txt.

Description

Yan Li and co-authors generate singing faces from audio with a spectral diffusion model in a 2024 paper. Talking-head generators stumble on song, where pitch swings wide and expressions swell. Their SINGER model studies the sound in the frequency domain at several scales at once.

A spectral filter module then ties those patterns to the faces singers actually make. The two modules plug into a diffusion backbone for vivid, synced video. Lacking real singing footage, the team also gathered a wild dataset of performances. Tests show mouths and mood track the music far better than adapted talkers.

Catalog record

CREATORYan Li, Ziya Zhou, Zhiqiang Wang, Wei Xue, Wenhan Luo, Yike Guo
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03430

Browse the full library · About the Tower of Babel