Video Motion Transfer with Diffusion Transformers

Alexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, Fabio Pizzati · 2024

Read Video Motion Transfer with Diffusion Transformers by Alexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, Fabio Pizzati (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07776.txt.

Description

Alexander Pondaven and co-authors present a method in 2024 that copies the motion of one video onto a freshly made video. Their tool is built for the newest class of video generators, which use transformer blocks to clean up noise step by step. Motion copy lets a creator keep the action of a reference clip while the scene and actors change.

First they run the reference through a pre trained generator and read its cross frame attention maps. From those maps they extract a patch level motion signal. Then they steer the new video's hidden state so its motion matches that signal, with no training at all. They also tune the model's position codes for a further boost. Tests against recent rival methods favor theirs on several scores and in human judgments.

Catalog record

CREATORAlexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, Fabio Pizzati
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07776

Browse the full library · About the Tower of Babel