Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation

Yuying Ge Yizhuo Li Yixiao Ge Ying Shan · 2024

Read Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation by Yuying Ge Yizhuo Li Yixiao Ge Ying Shan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04432.txt.

Description

Yuying Ge, Yizhuo Li, Yixiao Ge, and Ying Shan introduced Divot in twenty twenty-four, a video tokenizer powered by diffusion. Language models that both understand and generate video need a tokenizer capturing space and time, whose codes can be decoded back into realistic clips. Their insight inverts the usual training: let a video diffusion model do the judging.

If the diffusion model can denoise clips when conditioned on the tokenizer's features, those features must hold the video's spatial and temporal content. The same diffusion model then serves as the de-tokenizer. Wrapping Divot around a pretrained language model produced competitive results across comprehension and generation benchmarks. An instruction-tuned variant, Divot-Vicuna, handled video storytelling, interleaving narrative text with generated video segments.

Catalog record

CREATORYuying Ge Yizhuo Li Yixiao Ge Ying Shan
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04432

Browse the full library · About the Tower of Babel