SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model
Yan Li, Ziya Zhou, Zhiqiang Wang, Wei Xue, Wenhan Luo, Yike Guo · 2024
Read SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model by Yan Li, Ziya Zhou, Zhiqiang Wang, Wei Xue, Wenhan Luo, Yike Guo (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03430.txt.
Description
Yan Li and co-authors generate singing faces from audio with a spectral diffusion model in a 2024 paper. Talking-head generators stumble on song, where pitch swings wide and expressions swell. Their SINGER model studies the sound in the frequency domain at several scales at once.
A spectral filter module then ties those patterns to the faces singers actually make. The two modules plug into a diffusion backbone for vivid, synced video. Lacking real singing footage, the team also gathered a wild dataset of performances. Tests show mouths and mood track the music far better than adapted talkers.
Catalog record
| CREATOR | Yan Li, Ziya Zhou, Zhiqiang Wang, Wei Xue, Wenhan Luo, Yike Guo |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03430