EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training

Yiyang Du, Zhanqiu Guo, Xin Ye, Liu Ren, Chenyan Xiong · 2026

Read EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training by Yiyang Du, Zhanqiu Guo, Xin Ye, Liu Ren, Chenyan Xiong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20012.txt.

Introduction

Du and seven colleagues propose EmbodiedMidtrain, a mid-training step that adapts vision-language models for vision-language-action robot control. Off-the-shelf vision models hold broad knowledge but sit far from action data, limiting downstream robot performance. Their data engine scores candidates from a large vision pool by proximity to action data, producing the EmbodiedMidtrain-1M set for mid-training before fine-tuning. Tests on simulated manipulation and real-world tasks show steady gains across backbones and action heads, even on tasks never used for selection.

Catalog record

CREATORYiyang Du, Zhanqiu Guo, Xin Ye, Liu Ren, Chenyan Xiong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45776
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20012

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training Yiyang Du, Zhanqiu Guo, Xin Ye, Liu Ren, Chenyan Xiong Abstract Vision-Language-Action Models (VLAs) inherit their visual and linguistic capabilities from Vision-Language Models (VLMs), yet most VLAs are built from off-the-shelf VLMs that are not adapted to the embodied domain, limiting their downstream performance. In this work, we propose EmbodiedMidtrain to bridge the gap between VLMs and VLAs. We first characterize the data distribution gap between them, showing that VLA data occupy compact regions that are largely separated from the broader VLM distribution, while the degree of alignment varies substantially both across and within VLM data sources. Then, we build a mid-training data engine that leverages a lightweight learnable proximity estimator to select the most VLA-aligned candidates from a large VLM pool, and mid-trains the VLM on this curated mixture before downstream VLA fine-tuning. Experiments on three robot manipulation benchmarks show that mid-training consistently improves performance across different VLM backbones, achieving results competitive with expert VLAs and off-the-shelf VLMs trained with larger model scale and training budgets. Further analysis reveals that mid-training provides a stronger initialization for VLA fine-tuning, with gains emerging from the earliest steps and widening throughout training. Moreover, the data engine captures both dataset-level and sample-level alignment signals, favoring spatial reasoning over text-centric tasks while preserving the diversity of the VLM data. We will release all code, data and models for future research. 1 1 1 Project page: adu2021.github.io/blog/EmbodiedMidtrain/ 1 Introduction Recent advances in robotics foundation models have given rise to Vision-Language-Action Models (VLAs), which unify vision perception, language understanding, and action generation within a single model to enable generalist robot control across diverse tasks, environments, and embodiments ( Kim et al. 2024 ; Black et al. 2024 ; GR Team 2025 ) . Most VLAs leverage Vision-Language Models (VLMs) as their backbone, inheriting rich visual and linguistic representations that facilitate rapid adaptation to embodied settings ( Kim et al. 2025a ; Physical Intelligence et al. 2025 ; NVIDIA et al. 2025 ) . Despite this progress, a fundamental gap exists: most VLAs initialize from general-purpose off-the-shelf VLMs that are not tailored toward embodied action generation. VLM pretraining ( Liu et al. 2023c ; Li et al. 2023a ; Bai et al. 2023 ) covers broad vision-language tasks such as captioning, visual question answering, and document understanding, whereas VLA training operates on robotic manipulation trajectories grounded in physical interaction. This distributional mismatch means that, even when a VLM backbone provides a strong initialization for language and visual understanding, its internal representations may not be well-suited for the embodied reasoning that effective action generation demands ( Yang et al. 2025 ) . Bridging this gap remains an open challenge: simply fine-tuning VLM on curated embodied data does not reliably translate into better VLA performance ( Zhang et al. 2026 ) , indicating the need for more principled VLM adaptation to the VLA domain. In this work, we present EmbodiedMidtrain , a mid-training framework that aligns VLM data distribution with the VLA domain, enabling VLAs to truly benefit from the abilities of their VLM backbones. We begin with a deep analysis of the VLM-VLA gap and identify a core issue (Figure 1 a): the two domains exhibit a significant data distribution discrepancy. VLA data form compact clusters that are largely separated from the broad, diverse VLM distributions ( Xing et al. 2025 ) ; yet this gap is not uniform, as some VLM samples are inherently more aligned with the VLA domain than others. This reveals that the bulk of VLM data concentrates on capabilities distant from VLA data, leaving downstream VLA fine-tuning to bridge a wide representational gap from a misaligned starting point. (a) VLM-VLA Data Distribution Gap (b) VLM Data Distribution Shift (c) VLM Mid-training (d) Downstream VLA Gains Figure 1: Overview of EmbodiedMidtrain . We analyze the data distribution gap between VLMs and VLAs, and select VLM samples with higher proximity to the VLA domain for mid-training, yielding a stronger initialization for downstream VLA fine-tuning. Leveraging this understanding, we develop a mid-training data engine that reshapes the VLM training distribution toward the VLA domain before fine-tuning begins (Figure 1 b) to bridge the gap. The key idea is that, rather than training on all available VLM data indiscriminately, we can select the subset that is most aligned with the target VLA distribution and use it to mid-train the VLM, producing a stronger initialization for downstream VLA learning. Concretely, we propose a lightweight proximity estimator: a learnable classifier on frozen VLM features that learns to distinguish VLA data from VLM data, and whose predicted scores serve as a continuous measure of each VLM sample’s closeness to the VLA domain. Top-scoring samples are then assembled into a curated mid-training mixture, and the VLM is mid-trained on this distribution-aligned dataset to serve as a better initialization for VLA fine-tuning (Figure 1 c). This pipeline is lightweight, scalable, and requires no architectural changes to the underlying VLM or VLA. Experiments on Calvin ABC-D ( Mees et al. 2022 ) , SimplerEnv Bridge ( Walke et al. 2023 ) , and Libero-10 ( Liu et al. 2023a ) show that our mid-trained 1.1B model achieves a significant performance boost (Figure 1 d). Beyond raw performance gain, our analysis reveals several insights into what makes mid-training effective. First, the same curated data transfers across architectures: applying the mixture selected with InternVL3.5-1B to Qwen3VL-2B yields consistent gains, suggesting that proximity-based selection captures transferable properties across VLMs. Second, the learned proximity estimator substantially outperforms hand-crafted alternatives like feature-space distance and perplexity-based metrics. Third, training dynamics analysis shows that the mid-trained model outperforms the baseline from the earliest fine-tuning checkpoints and the gap widens over time, indicating mid-training provides a fundamentally better initialization rather than a transient head start. Our main contributions are as follows: • A proximity-based mid-training pipeline to bridge the VLM-VLA gap. We propose EmbodiedMidtrain , a mid-training data engine that learns a proximity estimator to score VLM samples by their closeness to the VLA domain, and selects the top-ranked candidates to curate a distribution-aligned mid-training mixture. • Consistent VLA performance gains across benchmarks and backbones. Extensive experiments on Calvin ABC-D, SimplerEnv Bridge, and Libero-10 demonstrate that our mid-trained VLM initialization yields large and consistent improvements over the original backbone, achieving results competitive with substantially larger models. • Insights on better VLM initialization for VLAs. We show that mid-training produces an initialization whose advantages emerge from the earliest fine-tuning steps and amplify over time. We further demonstrate that the learned proximity estimation outperforms hand-crafted alternatives by capturing fine-grained alignment with the VLA domain. 2 Related work Vision-Language-Action Models. VLAs extend foundation models like VLMs to generate robot actions, leveraging the broad visual and language understanding inherited from VLMs ( Kim et al. 2024 ; Black et al. 2024 ; NVIDIA et al. 2025 ) . Existing designs differ in backbone choice and action generation mechanism. Token-based approaches such as OpenVLA ( Kim et al. 2024 ) discretize robot actions into tokens for autoregressive generation; OpenVLA-OFT ( Kim et al. 2025a ) further introduces parallel decoding with learnable action embeddings for efficient continuous action prediction. Flow-matching and diffusion-based approaches such as

Read the complete text