ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Models
Puhao Li, Yingying Wu, Ziheng Xi, Wanlin Li, Yuzhe Huang, Zhiyuan Zhang, Yinghan Chen, Jianan Wang, Song-Chun Zhu, Tengyu Liu, Siyuan Huang, [4pt] Tsinghua University · 2025
Read ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Models by Puhao Li, Yingying Wu, Ziheng Xi, Wanlin Li, Yuzhe Huang, Zhiyuan Zhang, Yinghan Chen, Jianan Wang, Song-Chun Zhu, Tengyu Liu, Siyuan Huang, [4pt] Tsinghua University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16211.txt.
Introduction
Li and a team including Huang propose ControlVLA in twenty twenty-five, a way to adapt robot models with few examples. Large models for vision, language, and action can learn broad skills from many demos. Adapting them to a new task with just a few demos is still hard. Past methods often need simulation data or fixed grasp modules.
ControlVLA adds object focused signals to the base model through new layers that start at zero. Those layers learn slowly, so old skills are not erased at once. In real tests on six tasks, such as pouring cubes and folding clothes, the method hit a high success rate. It needed only ten to twenty demos per task. Past methods needed more than one hundred demos for like results.
Catalog record
| CREATOR | Puhao Li, Yingying Wu, Ziheng Xi, Wanlin Li, Yuzhe Huang, Zhiyuan Zhang, Yinghan Chen, Jianan Wang, Song-Chun Zhu, Tengyu Liu, Siyuan Huang, [4pt] Tsinghua University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-16211
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ControlVLA: Few-shot Object-centric Adaptation for Pre-trained Vision-Language-Action Models Puhao Li, Yingying Wu, Ziheng Xi, Wanlin Li, Yuzhe Huang, Zhiyuan Zhang, Yinghan Chen, Jianan Wang, Song-Chun Zhu, Tengyu Liu, Siyuan Huang, [4pt] Tsinghua University Abstract Learning real-world robotic manipulation is challenging, particularly when limited demonstrations are available. Existing methods for few-shot manipulation often rely on simulation-augmented data or pre-built modules like grasping and pose estimation, which struggle with sim-to-real gaps and lack extensibility. While large-scale imitation pre-training shows promise, adapting these general-purpose policies to specific tasks in data-scarce settings remains unexplored. To achieve this, we propose ControlVLA, a novel framework that bridges pre-trained VLA models with object-centric representations via a ControlNet-style architecture for efficient fine-tuning. Specifically, to introduce object-centric conditions without overwriting prior knowledge, ControlVLA zero-initializes a set of projection layers, allowing them to gradually adapt the pre-trained manipulation policies. In real-world experiments across 6 diverse tasks, including pouring cubes and folding clothes, our method achieves a 76.7% success rate while requiring only 10-20 demonstrations — a significant improvement over traditional approaches that require more than 100 demonstrations to achieve comparable success. Additional experiments highlight ControlVLA’s extensibility to long-horizon tasks and robustness to unseen objects and backgrounds. Keywords: Robotic manipulation, Imitation learning, Few-shot learning 1 Introduction Robotic manipulation in the real world remains a fundamental challenge, particularly when learning skills from limited demonstrations. While recent advances in robotic manipulation [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] have shown promise, current methods still demand extensive training data and struggle to efficiently adapt to new tasks and environments with few demonstrations. This limitation significantly hinders the deployment of robots in diverse real-world scenarios, where large amounts of task-specific training data are often impractical or prohibitively expensive. To tackle this, previous works [ 17 , 18 , 10 , 19 ] augment expert demonstrations in simulation to enhance policy learning. However, these approaches typically assume a priori knowledge of object and environment CAD models, as well as precise 3D pose estimations—requirements often impractical in real-world scenarios. Currently, imitation learning has achieved impressive success in manipulation [ 11 , 1 , 8 , 20 , 3 , 21 , 22 , 23 ] , primarily due to its scalability and capability to acquire skills across a wide range of scenarios without relying on external a priori knowledge. In particular, pre-trained general-purpose vla ( vla ) [ 24 , 11 , 23 , 22 , 3 , 25 ] has emerged as a promising approach for enabling generalizable robot behaviors across various tasks and environments. However, fine-tuning these vla for downstream tasks remains data-intensive, as substantial amounts of task- and environment-specific data are still required to adapt to the visual and action domains of the target task [ 26 , 23 , 22 , 3 ] . On a separate front, object-centric representations has shown potential in improving data efficiency for learning expert policies [ 4 , 27 , 7 ] . By focusing on relevant object properties ( e.g . , shape, size, and position), object-centric representations reduce the complexity of the input observation space. This approach enhances policy robustness to changes in object pose and instance, while also making the policies less susceptible to real-world noise compared to pixel-level features. Despite this, existing methods still require hundreds of demonstrations to learn a task [ 4 , 27 ] , mainly due to the lack of an action prior from vla in data-scarce scenarios. Fig. 1: ControlVLA bridges pre-trained manipulation policies with object-centric representations via ControlNet-style efficient fine-tuning. ControlVLA requires only 10–20 demonstrations to achieve 76.7% task success rate, significantly surpassing baseline’s 20.8% success rate. To this end, we introduce ControlVLA , a novel learning framework that combines pre-trained vla with object-centric representations for efficient few-shot learning. By integrating object-centric representations into a pre-trained vla , our approach leverages both the rich prior knowledge from large-scale pre-training and the data efficiency of object-centric learning. Inspired by Zhang et al . [ 28 ] , ControlVLA introduces additional cross-attention layers with zero-initialized key-value (KV) projection weights , allowing expert policies to acquire task-specific skills while progressively integrating object-centric representations. This design ensures that the policy focuses on task-relevant concepts without compromising the generalization or action quality of the pre-trained vla . The zero-initialization of additional KV projections stabilizes fine-tuning by mitigating the introduction of harmful noise, thereby enabling a seamless integration of task-specific object-centric representations with general-purpose vla pre-training. As a result, ControlVLA significantly reduces data requirements for task-specific adaptation, enhancing the efficiency of robotic manipulation deployment in the real world. We demonstrate the efficacy and efficiency of ControlVLA across 8 diverse real-world tasks , achieving robust performance with only 10-20 demonstrations . The evaluation tasks span diverse manipulation challenges: pick-and-place tasks with rigid, soft, and precision-critical objects, as well as complex manipulations including articulated object operation, object pouring, and deformable cloth folding. Empirically, ControlVLA attains an impressive 76.7% success rate across all tasks with very limited demonstrations, significantly surpassing baseline methods that achieve a mere 20.8% success rate. Additionally, we demonstrate the extensibility of ControlVLA on long-horizon tasks and its robustness on unseen objects and backgrounds . Ablation studies confirm the necessity of three key components: (1) vla pre-training for skill priors, (2) object-centric representation for efficient task grounding and learning, and (3) ControlNet-style conditioning for stable adaptation. In summary, ControlVLA bridges the gap between large-scale vla pre-training and efficient object-centric adaptation , enabling robots to acquire complex skills from minimal demonstrations. 2 Related Works 2.1 Few-shot Learning for Manipulation Reducing reliance on costly demonstrations while ensuring robust performance remains a key challenge in robotic manipulation. Early approaches synthesized training data through simulation-based augmentation [ 17 , 18 ] or combined imitation with reinforcement learning for robustness [ 19 ] , yet these methods depend heavily on accurate object poses and CAD models, limiting real-world applicability. To overcome sim-to-real gaps, especially in contact-rich or deformable tasks, recent methods turn to human video priors , using representations like R3M [ 21 ] , VIP [ 9 ] or Ag2Manip [ 2 ] to guide policy learning. However, these largely rely on 2D cues and struggle with precise spatial reasoning. Concurrently, few-shot learning aims to generalize from minimal demonstrations, with DenseMatcher [ 29 ] and SPOT [ 7 ] addressing transferable skill learning via 3D correspondences and object-centric planning. Yet, most approaches still require handcrafted grasping routines and accurate pose pipelines, and few leverage pre-trained vla as priors. Our work addresses these gaps by introducing ControlNet-style conditioning for object-centric policy modulation, enabling efficient few-shot adaptation in unstructured settings. 2.2 Object-centric Representation Learning Object-centric representation decomposes complex scenes into manipulable entities, enabling more efficient reasoning for robot learning. Early methods represent objects by poses [ 30 , 31 , 32 ] or by bounding boxes [ 33 , 34 ] , but their dependence on known categories or instance labels limits generalization to novel objects and dynamic environments. Unsupervised discovery techniques segment visual inputs into object-like regions without requiring labels [ 35 , 36 ] , yet they struggle in cluttered or