PVI: Plug-in Visual Injection for Vision-Language-Action Models

Zezhou Zhang, Songxin Zhang, Xiao Xiong, Junjie Zhang, Zejian Xie, Jingyi Xi, Zunyao Mao, Zan Mao, Zhixin Mai, Zhuoyang Song †, Jiaxing Zhang † · 2026

Read PVI: Plug-in Visual Injection for Vision-Language-Action Models by Zezhou Zhang, Songxin Zhang, Xiao Xiong, Junjie Zhang, Zejian Xie, Jingyi Xi, Zunyao Mao, Zan Mao, Zhixin Mai, Zhuoyang Song †, Jiaxing Zhang † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12772.txt.

Introduction

Zezhou Zhang and co-authors add temporal visual evidence to vision-language-action models for robot manipulation. The vision backbone is tuned for semantic abstraction and static views, so fine geometry and motion cues fade. Their plug-in module injects extra visual features through zero-initialized residual paths. That design keeps pretrained behavior and needs only one fine-tuning stage. Temporal video features beat strong static image features in their tests. Gains were largest on multi-phase tasks, and a real robot folded cloth over long horizons.

Catalog record

CREATORZezhou Zhang, Songxin Zhang, Xiao Xiong, Junjie Zhang, Zejian Xie, Jingyi Xi, Zunyao Mao, Zan Mao, Zhixin Mai, Zhuoyang Song †, Jiaxing Zhang †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48312
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12772

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PVI: Plug-in Visual Injection for Vision-Language-Action Models Zezhou Zhang, Songxin Zhang, Xiao Xiong, Junjie Zhang, Zejian Xie, Jingyi Xi, Zunyao Mao, Zan Mao, Zhixin Mai, Zhuoyang Song †, Jiaxing Zhang † Abstract VLA architectures that pair a pretrained VLM with a flow-matching action expert have emerged as a strong paradigm for language-conditioned manipulation. Yet the VLM, optimized for semantic abstraction and typically conditioned on static visual observations, tends to attenuate fine-grained geometric cues and often lacks explicit temporal evidence for the action expert. Prior work mitigates this by injecting auxiliary visual features, but existing approaches either focus on static spatial representations or require substantial architectural modifications to accommodate temporal inputs, leaving temporal information underexplored. We propose Plug-in Visual Injection (PVI), a lightweight, encoder-agnostic module that attaches to a pretrained action expert and injects auxiliary visual representations via zero-initialized residual pathways, preserving pretrained behavior with only single-stage fine-tuning. Using PVI, we obtain consistent gains over the base policy and a range of competitive alternative injection strategies, and our controlled study shows that temporal video features (V-JEPA2) outperform strong static image features (DINOv2), with the largest gains on multi-phase tasks requiring state tracking and coordination. Real-robot experiments on long-horizon bimanual cloth folding further demonstrate the practicality of PVI beyond simulation. 1 Introduction Vision-Language-Action (VLA) models offer a scalable pathway toward generalist robot manipulation by unifying language comprehension with visuomotor control [ 39 , 18 , 17 , 3 , 48 , 27 ] . A prevailing design pairs a Vision-Language Model (VLM) for semantic understanding and instruction grounding with a diffusion or flow-matching Transformer as the action expert, which generates continuous action sequences conditioned on the VLM’s representations [ 24 , 7 , 6 , 5 , 36 , 37 ] . However, this factorization carries a structural limitation: the VLM’s language backbone, optimized for semantic abstraction rather than geometric fidelity, tends to compress away fine-grained spatial cues—object geometry, edge alignments, and grasp affordances—before they ever reach the action expert [ 39 , 26 , 23 ] , precisely the information that dexterous manipulation demands. A natural response to this bottleneck is to enrich the visual information available to the action network beyond what the VLM pathway preserves. Prior work pursues this in two complementary ways: fusing geometry-grounded or spatially enriched features into the VLM’s visual tokens before semantic processing [ 25 , 1 ] , or bypassing the VLM entirely by injecting auxiliary visual representations directly into the action expert via side branches [ 23 , 21 , 22 , 44 , 40 ] . Both strategies recover meaningful geometric detail and have demonstrated improvements in manipulation precision. Yet both strategies share a common limitation: the injected representations are often static, capturing the spatial state of a scene at a single instant while remaining blind to the temporal dynamics—motion continuity, progressive contact, substep transitions—that closed-loop manipulation inherently demands. Recent work has begun to explore richer temporal and video-based representations within VLA models, demonstrating that motion and contact dynamics provide complementary information beyond static features [ 28 , 46 , 41 , 32 ] . However, these approaches typically require substantial architectural modifications to the pretrained backbone or the VLM-to-action interface to accommodate temporal inputs, raising a more fundamental question: how can we inject temporally aware visual information in a way that is lightweight, minimally invasive, and compatible with existing pretrained VLA weights? Figure 1 : Overview of Plug-in Visual Injection (PVI). Typical VLAs condition the VLM on static images with language, providing limited temporal context; moreover, the VLM’s output representations may under-emphasize fine-grained geometric cues. PVI bypasses this bottleneck by injecting auxiliary visual representations directly into the frozen action expert via a trainable plug-in, with no backbone modification and no re-pretraining required. We address this challenge with Plug-in Visual Injection (PVI), a lightweight module that injects auxiliary visual representations directly into the flow-matching action expert of a pretrained VLA without modifying its architecture or requiring retraining from scratch, as shown in Fig. 1 . PVI introduces a frozen auxiliary visual encoder alongside a trainable copy branch that mirrors the action expert’s architecture and conditioning interface. The copy branch is conditioned on projected auxiliary visual features rather than the VLM representations; its layer-wise hidden states are mapped by zero-initialized injection layers and added residually to the corresponding layers of the frozen main action expert, allowing auxiliary information to influence denoising across the full depth of the policy while preserving pretrained representations. This design enables stable, single-stage fine-tuning through the injection pathway alone. Crucially, the conditioning interface is encoder-agnostic, admitting static, temporal, or combined visual encoders under a unified injection mechanism, decoupling the injection infrastructure from any particular representation choice. We instantiate PVI on GR00T N1.5 [ 5 ] and evaluate it on simulated bimanual manipulation tasks spanning diverse primitives, complemented by real-robot experiments on deformable-object manipulation. Our contributions are threefold: • PVI framework. We propose Plug-in Visual Injection (PVI), a stable, minimally invasive, and encoder-agnostic visual-injection framework for pretrained VLAs. Via a dual-pathway design with layer-wise zero-initialized residual injection, PVI injects auxiliary representations into a frozen action expert while preserving its pretrained behavior, requiring only single-stage fine-tuning of the injection pathway. Its encoder-agnostic interface admits static, temporal, or combined visual encoders under a unified mechanism, decoupling the injection infrastructure from any particular representation choice. • Effective injection via systematic design comparison. We systematically evaluate PVI against a range of plausible injection strategies across diverse bimanual simulation tasks. PVI improves the average success rate from 35.7% to 59.7% (+24.0 percentage points), outperforming competitive alternatives. Real-robot experiments further validate the effectiveness of PVI beyond simulation, where PVI successfully drives long-horizon bimanual cloth folding that demands precise two-arm coordination over a highly deformable object. • Representation study. Leveraging PVI as a controlled testbed, we systematically compare temporal video encoders, static image encoders, and their combinations. Temporal video features (V-JEPA2 [ 2 ] ) consistently yield larger gains than strong static baselines (DINOv2 [ 31 ] ), especially on tasks requiring state tracking and multi-phase coordination, providing direct empirical evidence that temporal dynamics carry complementary information beyond what static representations can supply. 2 Related Works 2.1 VLA Models with Diffusion/Flow-Matching Action Experts Early VLAs cast action generation as autoregressive token prediction, inheriting the semantic generalization of large VLMs but relying on action discretization, which can limit fine-grained continuous control [ 18 , 17 , 48 , 13 , 8 ] . Recent methods therefore increasingly pair a VLM for semantic grounding with a dedicated diffusion or flow-matching action expert for continuous action generation [ 27 , 24 , 7 , 6 , 5 , 36 , 12 ] . While this design improves action modeling, it also creates a structural bottleneck: the action expert receives visual information primarily through compact VLM representations optimized for semantic abstraction, which can attenuate the precise geometric and temporal cues that dexterous manipulation demands. 2.2 Auxiliary Visual Feature Injection To mitigate this bottleneck, one line of work enriches the VLM pathway itself, fusing geometry-grounded or spatially enriched features into VLM visual representations before semantic processing, or strengthening the vision backbone with pretrained geometric and spatiotemporal priors [ 25 , 1 , 28 , 46 , 41 , 45 , 33 ,

Read the complete text