Perception Tokens Enhance Visual Reasoning in Multimodal Language Models

Mahtab Bigverdi Zelun Luo Cheng-Yu Hsieh, Ethan Shen Dongping Chen Linda G. Shapiro Ranjay Krishna · 2024

Read Perception Tokens Enhance Visual Reasoning in Multimodal Language Models by Mahtab Bigverdi Zelun Luo Cheng-Yu Hsieh, Ethan Shen Dongping Chen Linda G. Shapiro Ranjay Krishna (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03548.txt.

Description

Mahtab Bigverdi and colleagues give multimodal language models a scratchpad of perception tokens: internal tokens that carry depth maps and object boxes, so the model can reason over what it sees. Language alone is a clumsy medium for spatial facts. Fine-tuning generalizes poorly, and calling outside vision tools is heavy. Writing in twenty twenty-four, their training method converts intermediate visual representations into token sequences with a vector-quantized autoencoder. Multi-task training then teaches the model to emit and use these tokens mid-reasoning, like a visual chain of thought.

Counting benchmarks improved by roughly ten points across three suites, with better transfer between data sets than fine-tuning achieved. Relative depth judgments improved as well. The authors argue perception should be a native dialect of the model, not a translation it requests from outside.

Catalog record

CREATORMahtab Bigverdi Zelun Luo Cheng-Yu Hsieh, Ethan Shen Dongping Chen Linda G. Shapiro Ranjay Krishna
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03548

Browse the full library · About the Tower of Babel