Decoupling Vision and Language: Codebook Anchored Visual Adaptation

Jason Wu, Tianchen Zhao, Chang Liu, Jiarui Cai, Zheng Zhang, Zhuowei Li, Aaditya Singh, Xiang Xu, Mani Srivastava, Jonathan Wu, [0.5em] AWS · 2026

Read Decoupling Vision and Language: Codebook Anchored Visual Adaptation by Jason Wu, Tianchen Zhao, Chang Liu, Jiarui Cai, Zheng Zhang, Zhuowei Li, Aaditya Singh, Xiang Xu, Mani Srivastava, Jonathan Wu, [0.5em] AWS (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19449.txt.

Description

Large vision-language models translate images into representations for reasoning, but their vision encoders often stumble on domain-specific tasks like medical diagnosis or fine-grained classification. Jason Wu and nine co-authors note that representation errors then cascade through the language model into wrong answers. Existing adaptation methods tune the projector between encoder and language model, keeping the two components coupled.

The team introduces CRAFT (Codebook RegulAted Fine-Tuning), a lightweight way to tune the encoder with a discrete codebook anchoring visual representations to a stable token space. Nothing else in the model changes, so domain adaptation no longer needs re-alignment whenever the encoder changes. The adapted encoder can then boost vision-language models with different language models behind them.

Catalog record

CREATORJason Wu, Tianchen Zhao, Chang Liu, Jiarui Cai, Zheng Zhang, Zhuowei Li, Aaditya Singh, Xiang Xu, Mani Srivastava, Jonathan Wu, [0.5em] AWS
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-19449

Browse the full library · About the Tower of Babel