VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen · 2026

Read VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models by Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03309.txt.

Introduction

Jianke Zhang and nine co-authors ask a practical question about robots that see, read, and act: how much does the choice of the underlying vision-language model matter for the final policy? Vision-language-action models borrow a pretrained model's competence, and builders assume better general models make better robots. The authors introduce VLM4VLA, a minimal pipeline that converts general models into action policies with only a few new learnable parameters, enabling fair comparison.

Across three benchmarks, the simple pipeline proved surprisingly competitive with sophisticated designs. Vision-language initialization consistently beat training from scratch, but a model's general benchmark competence was a poor predictor of its control performance. The finding challenges a common assumption: broad vision-language skill looks necessary for embodied control, but it is far from sufficient.

Catalog record

CREATORJianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03309

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen Abstract Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLMs) into their policy backbone, are gaining significant attention for their promising generalization capabilities. This paper revisits a fundamental yet seldom systematically studied question: how VLM choice and competence translate to downstream VLA policies performance? We introduce VLM4VLA , a minimal adaptation pipeline that converts general-purpose VLMs into VLA policies using only a small set of new learnable parameters for fair and efficient comparison. Despite its simplicity, VLM4VLA proves surprisingly competitive with more sophisticated network designs. Through extensive empirical studies on various downstream tasks across three benchmarks, we find that while VLM initialization offers a consistent benefit over training from scratch, a VLM’s general capabilities are poor predictors of its downstream task performance. This challenges common assumptions, indicating that standard VLM competence is necessary but insufficient for effective embodied control. We further investigate the impact of specific embodied capabilities by fine-tuning VLMs on seven auxiliary embodied tasks (e.g., embodied QA, visual pointing, depth estimation). Contrary to intuition, improving a VLM’s performance on specific embodied skills does not guarantee better downstream control performance. Finally, modality-level ablations identify the visual module in VLM, rather than the language component, as the primary performance bottleneck. We demonstrate that injecting control-relevant supervision into the vision encoder of the VLM yields consistent gains, even when the encoder remains frozen during downstream fine-tuning. This isolates a persistent domain gap between current VLM pretraining objectives and the requirements of embodied action-planning. Figure 1: An overview of our VLM4VLA framework. (Left) The evaluation pipeline for testing different VLM backbones, which are evaluated on downstream tasks after an optional fine-tuning stage on auxiliary embodied tasks. (Bottom Right) We systematically investigate three factors influencing VLM-to-VLA transfer: the choice of VLM backbone, the impact of fine-tuning on auxiliary embodied tasks, and the influence of different training strategies (frozen vs. fine-tuned different VLM modules, training from scratch). (Top Right) A visualization of inconsistent performance of various VLM backbones across downstream tasks. 1 Introduction Vision-Language-Action (VLA) models ( Brohan et al. 2023 ) have recently emerged as a central research focus, as they leverage the extensive visual-language knowledge from Vision-Language Models (VLMs) as a prior for enhancing the generalization of robotic strategies ( Kim et al. 2024 ; Black et al. 2024 ; Zhang et al. 2024 ; Chen et al. 2025c ) . The majority of existing VLA methods have focused on developing more advanced network architectures ( Li et al. 2023 ; Shi et al. 2025 ) , incorporating additional training paradigms or modalities ( Zheng et al. 2024 ; Chen et al. 2024 ; Zhang et al. 2025a ) , and refining action decoding schemes ( Zhao et al. 2023 ; Pertsch et al. 2025 ; Wen et al. 2025 ) . However, limited attention ( Liu et al. 2025 ) has been given to a fundamental question at the core of VLA: how do the choice and specific capabilities of the underlying VLM affect the performance of VLA policies? In this paper, we revisit this crucial problem. To provide a fair and clean test interface that evaluates the capabilities of VLMs without introducing extraneous variables, we first build the generic VLM4VLA pipeline to convert general-purpose VLMs into VLAs, as shown in Figure 2 . VLM4VLA is a carefully designed network plug-in, introducing fewer than 1% new parameters. To enhance the stability of inference and the robustness of evaluation, we use a simple MLP head with L1/L2 loss rather than a diffusion-based (flow-matching) approach, thus controlling stochasticity and reducing tuning complexity. This network allows us to train the modified VLMs directly using downstream robot data, facilitating alignment between the VLM’s capabilities and the demands of robotic tasks. Despite its simplicity, VLM4VLA proves effective, demonstrating competitive performance against more advanced network designs, such as the flow-matching action expert ( Black et al. 2024 ) , on benchmark tests. This provides a solid foundation for conducting fair and scalable experiments across various VLMs, aligning their capabilities with the demands of embodied tasks. Based on the VLM4VLA pipeline, we conduct a large-scale empirical study across various downstream tasks on three commonly used benchmarks, evaluating 24 different VLMs that are either zero-shot or fine-tuned. Specifically, we investigate the VLM capabilities across three dimensions: • General capability. We first quantify the benefit of VLM pretraining by comparing VLM-initialized policies against each other and training-from-scratch baselines. We then examine whether a VLM’s general-purpose strength (as measured by standard VLM evaluations) predicts its downstream embodied control performance. • Embodied-specific capability. Recent works have sought to enhance VLAs by improving the VLM backbone on a series of VLM tasks. We examine how the embodied-specific capabilities of VLMs correlate with performance on downstream control tasks. • Modality-level Analysis. Finally, we disentangle the contributions of each modality by independently ablating the vision and language encoders. We compare freezing versus fine-tuning each module to assess whether further aligning pretrained representations with control-relevant information improves downstream VLA performance. For general capability, we select nine open-sourced VLMs as the backbone. As shown in Figure 1 (a), we apply the VLM4VLA pipeline directly to these VLMs together with the training from scratch baselines and fine-tune them with robot data. For embodied-specific capability, we collect seven commonly used embodied tasks or pretrained models based on the Qwen2.5-VL ( Bai et al. 2025b ) backbone to conduct ablation studies. As shown in Figure 1 (b), we either fine-tune the VLM using auxiliary datasets or directly use the released fine-tuned models. We then apply the VLM4VLA pipeline to adapt these models to the embodied policies. Lastly, we conduct modality-level ablations to disentangle the contributions of vision and language components (Figure 1 (c)). This involves selectively freezing or fine-tuning the vision and text encoders. Furthermore, we investigate the efficacy of injecting control-specific knowledge directly into the vision encoder via the FAST tokenizer ( Pertsch et al. 2025 ) . We evaluate the fine-tuned models on a range of tasks across three benchmarks including Calvin ( Mees et al. 2022 ) , SimplerEnv ( Li et al. 2024 ) , and Libero ( Liu et al. 2023 ) to assess their embodied capabilities. To our suprise, our findings reveal that while VLM initialization offers a consistent benefit over training from scratch, a VLM’s general capabilities are poor predictors of its downstream task performance, contrary to common assumptions. For instance, we observe that Kosmos ( Peng et al. 2023 ) outperforms Qwen2.5-VL ( Bai et al. 2025b ) and Paligemma ( Steiner et al. 2024 ) in certain environments. Inconsistencies across benchmarks suggest that VLA policies require capabilities beyond those currently pursued by VLMs. Furthermore, the gains by fine-tuning VLMs on specific auxiliary embodied tasks do not transfer to the downstream control tasks. Lastly, our modality-level analysis identifies the vision encoder, rather than the language component, as the primary performance bottleneck. We find that fine-tuning the vision encoder is essential for strong control performance, whereas the language encoder is less critical. The significant performance gains observed after injecting action-relevant information into the vision modules within

Read the complete text