Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation
Jin Shi, Brady Zhang, Yishun Lu · 2026
Read Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation by Jin Shi, Brady Zhang, Yishun Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16241.txt.
Introduction
Shi, Zhang, and Lu present VLA-AD, a distillation framework that turns large vision-language-action teachers into lightweight student policies using offline semantic guidance. Instead of copying low-level actions alone, it adds task phase anchors and multi-frame direction descriptions during training only. With OpenVLA-7B as teacher, the student shrinks 44-fold to 158M parameters while matching the teacher within 0.27 percent on average. The student runs at 12.5 Hz on an RTX 4090, a 3.28-fold inference speedup.
Catalog record
| CREATOR | Jin Shi, Brady Zhang, Yishun Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41027 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16241
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Offline Semantic Guidance for Efficient Vision-Language-Action Policy Distillation Jin Shi, Brady Zhang, Yishun Lu Abstract Billion-parameter Vision-Language-Action (VLA) policies have recently shown impressive performance in robotic manipulation, yet their size and inference cost remain major obstacles for real-time closed-loop control. We introduce VLA-AD , a distillation framework that uses a Vision-Language Model as an offline semantic supervisor to transfer large VLA teachers into lightweight student policies. Instead of relying only on low-level action imitation, VLA-AD augments teacher-provided 7-DoF action targets with high-level semantic guidance, including task phase anchors and multi-frame operating-direction descriptions. These auxiliary signals are used only during training: at test time, the student policy runs independently, with neither the VLA teacher nor the VLM required. We evaluate VLA-AD on three LIBERO benchmark suites. Using OpenVLA-7B as the teacher, our method produces a 158M-parameter student, yielding a 44 × 44\times reduction in model size while matching the teacher with only a 0.27 % 0.27\% average relative gap. The resulting policy runs at 12.5 Hz on an RTX 4090, achieving a 3.28 × 3.28\times inference speedup over OpenVLA-7B. We further show that the same semantic distillation pipeline generalizes to a different π 0.5 \pi_{0.5} -4B teacher, where the student outperforms the teacher on two suites and remains within 0.53 % 0.53\% on libero_goal . Additional analysis indicates that phase-level supervision and multi-frame directional cues make the student less sensitive to noisy teacher actions, such as erroneous high-frequency gripper changes. Overall, VLA-AD demonstrates that offline semantic guidance from VLMs can substantially improve the efficiency, robustness, and deployability of VLA policy distillation. 1 Introduction In recent years, robotic manipulation has undergone a paradigm shift with the emergence of large-scale vision-language-action (VLA) foundation models . By combining transformer architectures with large-scale vision-language pretraining and real-world robotic trajectories, these models have demonstrated strong multi-task generalization and zero-shot instruction-following capabilities ( Brohan et al. 2023 ; Kim et al. 2024 ) . They represent an important step toward generalist robot policies that can execute diverse language-conditioned manipulation tasks in complex, unstructured environments. Despite these advances, deploying billion-parameter VLA models on real robotic platforms remains highly challenging. A major bottleneck is inference latency ( Ahn et al. 2024 ) . Evaluating 7B-parameter or larger models on edge devices commonly used in robotics can require more than one second per control step ( Brohan et al. 2023 ; Lin et al. 2026 ) . A common solution is to distill these large models into compact policies using standard behavioral cloning (BC) ( Ross et al. 2011 ; Mandlekar et al. 2021 ) . However, this introduces a second challenge: vulnerability to compounding errors. Since single-task robotic datasets typically cover only a limited subset of possible states, cloned student policies can suffer from severe distribution shift during closed-loop execution ( Chen et al. 2019 ) . As a result, a student may achieve high offline action-prediction accuracy but fail completely during online evaluation. Small prediction errors can accumulate over hundreds of timesteps, driving the robot into out-of-distribution (OOD) states that were not observed during training. Recent advances suggest that vision-language models (VLMs) can provide a useful source of auxiliary supervision for addressing this problem. Although VLMs do not directly output low-level 7-DoF robot actions, they possess strong scene understanding, spatial reasoning, and commonsense physical priors ( Huang et al. 2023 ; Bai et al. 2025 ) . Prior work has shown that aligning robotic policies with VLM-derived representations can improve semantic awareness and OOD generalization ( Kachaev et al. 2025 ; Karamcheti et al. 2023 ) . This motivates the use of a VLM as an offline “knowledge translator”: it can extract task-relevant visual-semantic information from demonstrations and inject this information into a lightweight control policy, thereby bridging raw visual observations and high-level manipulation logic ( Li et al. 2024 ; Ahn et al. 2022 ) . Motivated by this observation, we introduce VLA-AD , a VLM-assisted distillation framework for converting large VLA teachers into efficient closed-loop policies. VLA-AD uses a VLM as a parallel offline supervisor during training, while removing the VLM entirely at deployment. This allows the student to benefit from high-level semantic supervision without increasing inference latency. We evaluate VLA-AD through closed-loop experiments on the LIBERO benchmark and make the following contributions: • VLM-Assisted Zero-Overhead Distillation. We propose VLA-AD, a parallel-supervision distillation framework that uses a VLM as an offline semantic supervisor for compressing billion-scale VLA teachers into compact student policies. The VLM provides phase and direction descriptions during training, but is removed entirely at deployment, introducing zero additional inference latency. With OpenVLA-7B as the teacher, VLA-AD allows a 158M-parameter student with 44\times parameter compression, matching the teacher within only 0.27\% difference on average across three LIBERO suites. On an RTX 4090, the student reaches 12.5 Hz, giving a 3.28\times speedup over OpenVLA-7B. • Teacher-Agnostic Semantic Supervision Beyond Teacher Performance. We show that VLA-AD is not tied to a specific teacher architecture: the same semantic-supervision pipeline transfers across both OpenVLA-7B and \pi_{0.5} -4B teachers. Despite using a compact 158M-parameter student, VLA-AD matches OpenVLA-7B within only 0.27\% on relative average across three LIBERO suites, and surpasses the \pi_{0.5} -4B teacher on two of the three suites while remaining within 0.53\% on the libero_goal . This indicates that VLM-generated phase and direction anchors capture transferable manipulation structure, allowing the student to learn closed-loop behaviours that are not merely bounded by the raw teacher action distribution. • Phase Anchors and Multi-Frame Direction for Robust Distillation. We introduce a structured semantic-anchor mechanism that combines a 9-class phase descriptor with a multi-frame operating-direction signal. The phase descriptor provides a stable abstraction of the current manipulation stage, while the multi-frame direction signal resolves temporal ambiguity in visually similar operating states. This design gives the student consistent spatio-temporal context, helping it infer manipulation intent rather than overfitting to isolated frame-level observations. Overall, VLA-AD provides a practical route toward deploying compact and robust VLA policies on real robotic platforms. By transferring semantic knowledge only during training, it improves closed-loop robustness while preserving the low-latency inference required for real-time control. 2 Related Work 2.1 Distillation of Large Robotic Policies Deploying large-scale vision-language-action (VLA) foundation models ( Kim et al. 2024 ; Brohan et al. 2023 ) on edge devices requires substantial model compression. Recent work has therefore explored VLA distillation as a way to reduce inference latency. TinyVLA ( Wen et al. 2025 ) focuses on data-efficient architectural miniaturisation for high-frequency control, while CEED-VLA ( Song et al. 2025 ) improves efficiency through early-exit decoding and consistency distillation. Other methods, such as VITA-VLA ( Dong et al. 2025 ) and ActDistill ( Ye et al. 2026 ) , aim to transfer manipulation priors using action-expert distillation or dynamic action-guided routing. Despite these advances, standard behavioural cloning (BC) remains vulnerable to compounding errors and covariate shift during closed-loop execution ( Ross et al. 2011 ; Mandlekar et al. 2021 ) . Since single-task demonstration datasets usually cover only a limited range of states, small prediction errors can quickly drive the student policy into out-of-distribution (OOD) regions. Recent methods partially address this issue, but often rely on auxiliary action experts ( Dong et al. 2025 ) , dynamic routing mechanisms ( Ye et al. 2026 )