Interpretability Transfer from Language to Vision via Sparse Autoencoders

Alexey Kravets, Da Li, Chuan Li, Da Chen, Vinay P. Namboodiri · 2026

Read Interpretability Transfer from Language to Vision via Sparse Autoencoders by Alexey Kravets, Da Li, Chuan Li, Da Chen, Vinay P. Namboodiri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24946.txt.

Description

Alexey Kravets leads VISTA, a bridge carrying language interpretability into vision without dedicated vision autoencoders. Labeling visual concepts is hard and fuzzy, so past language gains stalled visually. VISTA steers a visual projector in a LLaVA model to map visual tokens into the language model's labeled textual autoencoder space. Regularized by that autoencoder's reconstruction loss, and the matching rate between top textual concepts and image elements roughly triples. The same lens shows DINOv2 localizing space better than rival encoders, and object removal and replacement tests prove alignment.

Catalog record

CREATORAlexey Kravets, Da Li, Chuan Li, Da Chen, Vinay P. Namboodiri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67101
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24946

Browse the full library · About the Tower of Babel