Video Motion Transfer with Diffusion Transformers
Alexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, Fabio Pizzati · 2024
Read Video Motion Transfer with Diffusion Transformers by Alexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, Fabio Pizzati (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07776.txt.
Description
Alexander Pondaven and co-authors present a method in 2024 that copies the motion of one video onto a freshly made video. Their tool is built for the newest class of video generators, which use transformer blocks to clean up noise step by step. Motion copy lets a creator keep the action of a reference clip while the scene and actors change.
First they run the reference through a pre trained generator and read its cross frame attention maps. From those maps they extract a patch level motion signal. Then they steer the new video's hidden state so its motion matches that signal, with no training at all. They also tune the model's position codes for a further boost. Tests against recent rival methods favor theirs on several scores and in human judgments.
Catalog record
| CREATOR | Alexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, Fabio Pizzati |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07776