Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation

Hyeonho Jeong 1 , 2 , *, Chun-Hao P. Huang 1, Jong Chul Ye 2, Niloy J. Mitra 1 , 3, Duygu Ceylan 1, 2 KAIST, 3 University College London · 2024

Read Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation by Hyeonho Jeong 1 , 2 , *, Chun-Hao P. Huang 1, Jong Chul Ye 2, Niloy J. Mitra 1 , 3, Duygu Ceylan 1, 2 KAIST, 3 University College London (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06016.txt.

Description

Hyeonho Jeong and co-authors present a 2024 paper on making generated videos hold together over time. Video generators produce rich clips, yet objects often drift, warp, or change identity across frames. The team blames the lack of any training signal that tracks points through the scene.

Their method trains one network to do two jobs at once: generate the video and track points across its frames. The tracking task forces the model's inner features to stay spatially consistent. Only small changes to a standard video model were needed. In wide tests, the joint training cut appearance drift and gave steadier, more coherent clips. The work shows tracking and generation help each other.

Catalog record

CREATORHyeonho Jeong 1 , 2 , *, Chun-Hao P. Huang 1, Jong Chul Ye 2, Niloy J. Mitra 1 , 3, Duygu Ceylan 1, 2 KAIST, 3 University College London
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06016

Browse the full library · About the Tower of Babel