Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
Yuying Ge Yizhuo Li Yixiao Ge Ying Shan · 2024
Read Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation by Yuying Ge Yizhuo Li Yixiao Ge Ying Shan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04432.txt.
Description
Yuying Ge, Yizhuo Li, Yixiao Ge, and Ying Shan introduced Divot in twenty twenty-four, a video tokenizer powered by diffusion. Language models that both understand and generate video need a tokenizer capturing space and time, whose codes can be decoded back into realistic clips. Their insight inverts the usual training: let a video diffusion model do the judging.
If the diffusion model can denoise clips when conditioned on the tokenizer's features, those features must hold the video's spatial and temporal content. The same diffusion model then serves as the de-tokenizer. Wrapping Divot around a pretrained language model produced competitive results across comprehension and generation benchmarks. An instruction-tuned variant, Divot-Vicuna, handled video storytelling, interleaving narrative text with generated video segments.
Catalog record
| CREATOR | Yuying Ge Yizhuo Li Yixiao Ge Ying Shan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04432