Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
Mahtab Bigverdi Zelun Luo Cheng-Yu Hsieh, Ethan Shen Dongping Chen Linda G. Shapiro Ranjay Krishna · 2024
Read Perception Tokens Enhance Visual Reasoning in Multimodal Language Models by Mahtab Bigverdi Zelun Luo Cheng-Yu Hsieh, Ethan Shen Dongping Chen Linda G. Shapiro Ranjay Krishna (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03548.txt.
Description
Mahtab Bigverdi and colleagues give multimodal language models a scratchpad of perception tokens: internal tokens that carry depth maps and object boxes, so the model can reason over what it sees. Language alone is a clumsy medium for spatial facts. Fine-tuning generalizes poorly, and calling outside vision tools is heavy. Writing in twenty twenty-four, their training method converts intermediate visual representations into token sequences with a vector-quantized autoencoder. Multi-task training then teaches the model to emit and use these tokens mid-reasoning, like a visual chain of thought.
Counting benchmarks improved by roughly ten points across three suites, with better transfer between data sets than fine-tuning achieved. Relative depth judgments improved as well. The authors argue perception should be a native dialect of the model, not a translation it requests from outside.
Catalog record
| CREATOR | Mahtab Bigverdi Zelun Luo Cheng-Yu Hsieh, Ethan Shen Dongping Chen Linda G. Shapiro Ranjay Krishna |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03548