When LLaVA Meets Objects: Token Composition for Vision-Language-Models

Soumya Jahagirdar [email protected], Walid Bousselham [email protected], Anna Kukleva [email protected], Hilde Kuehne [email protected] · 2026

Read When LLaVA Meets Objects: Token Composition for Vision-Language-Models by Soumya Jahagirdar [email protected], Walid Bousselham [email protected], Anna Kukleva [email protected], Hilde Kuehne [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04864.txt.

Description

Soumya Jahagirdar and 3 co-authors shrink the visual token load of vision-language models in a 2026 work called Mask-LLaVA. Autoregressive models usually represent images with many tokens, which costs computing power, especially when generating answers.

Their framework combines three levels of visual features: mask-based object representations, global tokens, and local patch tokens. All tokens are used in training. At test time, however, the model can flexibly drop many of the object tokens without retraining and without a significant performance loss. On standard benchmarks, results rivaled current token-efficient methods and approached the original LLaVA baseline while using only a fraction of the visual tokens. The design adapts its cost to the task at inference time.

Catalog record

CREATORSoumya Jahagirdar [email protected], Walid Bousselham [email protected], Anna Kukleva [email protected], Hilde Kuehne [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04864

Browse the full library · About the Tower of Babel