Mind the Time: Temporally-Controlled Multi-Event Video Generation

Ziyi Wu, Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Yuwei Fang, Varnith Chordia, Igor Gilitschenski, Sergey Tulyakov · 2024

Read Mind the Time: Temporally-Controlled Multi-Event Video Generation by Ziyi Wu, Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Yuwei Fang, Varnith Chordia, Igor Gilitschenski, Sergey Tulyakov (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05263.txt.

Description

Ask a video generator for three events in a row and it may skip one or shuffle their order. Ziyi Wu and co-authors fix that by giving each event its own time slot. Their generator, MinT, binds every caption to the stretch of video where that event belongs. A new positional encoding, built for time, lets the captions guide the right frames through attention.

The team fine-tuned a pretrained video diffusion model on data with events grounded in time. The results were coherent clips with smoothly joined events in the requested order. In tests, MinT beat commercial and open rivals by a wide margin. The authors claim the first real control over event timing in generated video.

Catalog record

CREATORZiyi Wu, Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Yuwei Fang, Varnith Chordia, Igor Gilitschenski, Sergey Tulyakov
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05263

Browse the full library · About the Tower of Babel