Decoupling Vision and Language: Codebook Anchored Visual Adaptation
Jason Wu, Tianchen Zhao, Chang Liu, Jiarui Cai, Zheng Zhang, Zhuowei Li, Aaditya Singh, Xiang Xu, Mani Srivastava, Jonathan Wu, [0.5em] AWS · 2026
Read Decoupling Vision and Language: Codebook Anchored Visual Adaptation by Jason Wu, Tianchen Zhao, Chang Liu, Jiarui Cai, Zheng Zhang, Zhuowei Li, Aaditya Singh, Xiang Xu, Mani Srivastava, Jonathan Wu, [0.5em] AWS (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-19449.txt.
Description
Large vision-language models translate images into representations for reasoning, but their vision encoders often stumble on domain-specific tasks like medical diagnosis or fine-grained classification. Jason Wu and nine co-authors note that representation errors then cascade through the language model into wrong answers. Existing adaptation methods tune the projector between encoder and language model, keeping the two components coupled.
The team introduces CRAFT (Codebook RegulAted Fine-Tuning), a lightweight way to tune the encoder with a discrete codebook anchoring visual representations to a stable token space. Nothing else in the model changes, so domain adaptation no longer needs re-alignment whenever the encoder changes. The adapted encoder can then boost vision-language models with different language models behind them.
Catalog record
| CREATOR | Jason Wu, Tianchen Zhao, Chang Liu, Jiarui Cai, Zheng Zhang, Zhuowei Li, Aaditya Singh, Xiang Xu, Mani Srivastava, Jonathan Wu, [0.5em] AWS |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-19449