Don’t Blind Your VLA: Aligning Visual Representations for OOD Generalization
Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, Aleksandr I. Panov · 2025
Read Don’t Blind Your VLA: Aligning Visual Representations for OOD Generalization by Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, Aleksandr I. Panov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25616.txt.
Introduction
Nikita Kachaev led four co-authors on vision-language-action models. Mikhail Kolosov, Daniil Zelezetsky, and Alexey K Kovalev helped. Aleksandr I Panov closes the five-author team. Their 2025 paper studies visual feature degradation in robots. Vision-language-action models adapt pretrained vision-language models. Those models promise transferable world knowledge for agents.
They systematically study feature retention during fine-tuning. Naive action fine-tuning leads to visual degradation, they show. Hidden features are probed to characterize effects. Attention maps are studied for feature changes. Targeted tasks contrast action models with vision counterparts. Changes in vision-language capabilities are isolated that way. Strategies for aligning visual features are tested. A simple effective method mitigates the degradation found. It yields better generalization to out-of-distribution scenarios. Trade-offs between action tuning and feature loss are clarified. Useful methods recover inherited vision-language capabilities for agents.
Catalog record
| CREATOR | Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, Aleksandr I. Panov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-25616
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Don’t Blind Your VLA: Aligning Visual Representations for OOD Generalization Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, Aleksandr I. Panov Abstract . The growing success of Vision-Language-Action (VLA) models stems from the promise that pretrained Vision-Language Models (VLMs) can endow agents with transferable world knowledge and vision-language (VL) grounding, laying a foundation for action models with broader generalization. Yet when these VLMs are adapted to the action modality, it remains unclear to what extent their original VL representations and knowledge are preserved. In this work, we conduct a systematic study of representation retention during VLA fine-tuning, showing that naive action fine-tuning leads to degradation of visual representations. To characterize and measure these effects, we probe VLA’s hidden representations and analyze attention maps, further, we design a set of targeted tasks and methods that contrast VLA models with their counterpart VLMs, isolating changes in VL capabilities induced by action fine-tuning. We further evaluate a range of strategies for aligning visual representations and introduce a simple yet effective method that mitigates degradation and yields improved generalization to out-of-distribution (OOD) scenarios. Taken together, our analysis clarifies the trade-off between action fine-tuning and the degradation of VL representations and highlights practical approaches to recover inherited VL capabilities. Code is publicly available: blind-vla-paper.github.io 1. Introduction Vision–Language Models (VLMs) have demonstrated remarkable success due to their ability to integrate large-scale multimodal datasets, thereby acquiring semantic grounding and generalizable visual-language (VL) representations ( NVIDIA et al. 2025a ; Bai et al. 2025 ; Wang et al. 2025b ; Beyer et al. 2024 ; Driess et al. 2023 ; Awadalla et al. 2023 ) . When exposed to novel visual or linguistic contexts, such models exhibit robust cross-modal understanding and compositional perception – properties that underpin their strong zero and few-shot generalization beyond the training distribution. These advancements have naturally inspired the extension of VLMs toward embodied domains. Figure 1. Visual alignment method overview. Mid-level VLA features are projected onto a normalized sphere and aligned with teacher embeddings, preserving visual semantics and improving OOD generalization. Bottom plots show comparison with standard SFT across three generalization axes on the Simpler-based benchmark ( Liu et al. 2025 ) . Vision–Language–Action (VLA) models represent a prominent direction in this research trajectory. They adapt pretrained VLMs to action prediction tasks in robotic settings, with the goal of leveraging the semantic priors and cognition abilities inherited from large-scale vision–language pretraining. The underlying hypothesis is that, if appropriately adapted, VLA models can transfer the visual–semantic representations of their initial VLM to the action domain, enabling generalization to previously unseen scenes, instructions, and scenarios. However, in practice, adapting VLMs to the action modality often introduces new challenges. Several recent studies ( Pugacheva et al. 2025 ; Driess et al. 2025 ; Liu et al. 2023 ; Mees et al. 2022 ; Chen et al. 2023 ) have shown that current VLA models struggle to maintain generalization in visually and linguistically complex tasks, raising questions about whether strong VL capabilities of VLMs truly transfer to embodied settings. This issue becomes the most evident during task-specific fine-tuning, where limited data diversity and datasets frequently lead to overfitting ( Zang et al. 2024 ; Staroverov et al. 2023 ; Ding et al. 2024 ; Pugacheva et al. 2025 ; Cherepanov et al. 2025b ) . Figure 2. Overview of the proposed method. (a, b) Training pipeline with visual alignment loss – no extra overhead, only precomputed teacher features and a lightweight regularization term during SFT. (c) Conceptual illustration of the loss landscape for VL tasks: the core idea is to optimize the model with respect to the action objective while preserving performance on VL understanding. During large-scale robotic pretraining, recent works have attempted to mitigate this degradation by preserving multimodal understanding capabilities. Prior strategies include incorporating auxiliary reasoning objectives ( Chen et al. 2025 ) , applying multimodal co-training on web-scale data ( Yang et al. 2025 ) , or freezing pretrained visual–language backbones to preserve VL representations and improve instruction following ( NVIDIA et al. 2025b ; Driess et al. 2025 ) . While these approaches help retain vision–language knowledge and improve generalization, they often depend on heavy supervision, high computational cost, or constrained model architecture. Yet, despite these advances at the pretraining stage, there remain no effective methods to address representation degradation during task-specific supervised fine-tuning (SFT) – the critical phase where VLA models must adapt to certain robotic domains without losing their semantic grounding and VL abilities. In this work, we adopt a realistic VLA deployment setting: starting from a pretrained VLA and adapting it with limited data for supervised fine-tuning in a chosen embodiment and domain. Under these constraints, we conduct a systematic investigation into the degradation of VL representations and multimodal understanding abilities in VLA models and ask a central question: Can we design a simple yet effective method to recover the inherited VL representations during fine-tuning on robotic actions? To answer this question, we first examined the attention maps and feature activations of the VLA model in comparison to VLM’s across matched image-instruction pairs from the robotics domain. Our analysis of attention maps revealed that: while the pretrained VLM accurately focuses on task-relevant objects, the fine-tuned VLA models often produce diffuse or misplaced activations, failing to attend to key entities under out-of-distribution (OOD) conditions ( Figure 4 ). Next, we conducted a t-SNE ( van der Maaten and Hinton 2008a ) analysis of intermediate representations across VLM’s and VLA’s layers, which exposed a clear representation collapse ( Barbero et al. 2024 ; Arefin et al. 2024 ) in VLA models – indicating that standard action fine-tuning compresses diverse internal features into a narrow representation space, reducing representational diversity and generalization capacity. Next, we propose VL-Think task suite ( section 4 ) to assess transfer of VL knowledge from VLMs to VLA models, benchmark several strong VLMs and compare OpenVLA–7B ( Kim et al. 2024 ) to its pretrained base (PrismaticVLM ( Karamcheti et al. 2024 ) ). We observe systematic, domain-specific forgetting after action fine-tuning, indicating that VLAs lose VL knowledge about domains absent from the robotics fine-tuning data. To address this representational degradation, we introduce a lightweight Visual Representation Alignment method inspired by the Platonic Representation Hypothesis ( Huh et al. 2024 ) . This hypothesis suggests that large vision and language models tend to converge toward a shared latent representation space that encodes general visual and semantic representations across generalist models. Our method explicitly constrains the visual representations of a VLA to remain aligned with a generalist vision model throughout fine-tuning. By maintaining this link, the VLA preserves semantic consistency while adapting its action policy to new tasks. The method adds negligible computational overhead and integrates seamlessly with SFT ( Figure 2 ). Extensive experiments on different variations of Simpler ( Li et al. 2024 ) benchmark demonstrates that this alignment consistently improves out-of-distribution generalization – yielding up to a 10% relative gain over naive SFT ( Table 1 ). Our key contributions are as follows: (1) We systematically demonstrate that naive VLA fine-tuning induces representation collapse and attention sink relative to their initial VLM. (2) We introduce VL-Think, a diagnostic task suite for assessing transfer of VL