CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models

Wenxuan Song, Han Zhao, Fuhao Li, Ziyang Zhou, Xi Wang, Jing Lyu, Pengxiang Ding, Yan Wang, Donglin Wang, Haoang Li · 2026

Read CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models by Wenxuan Song, Han Zhao, Fuhao Li, Ziyang Zhou, Xi Wang, Jing Lyu, Pengxiang Ding, Yan Wang, Donglin Wang, Haoang Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10903.txt.

Introduction

Wenxuan Song and nine colleagues offer a cheaper way to give vision-language-action robot models the benefits of auxiliary training. Extra training objectives improve such models and cut convergence steps, but their added losses cost heavy compute.

CapVector splits the job in parameter space: two models train on a small task set under different strategies, and the difference between their weights becomes a capability vector. Merging that vector into the pretrained model, plus a light orthogonal penalty during standard fine-tuning, yields the auxiliary gains without the auxiliary cost. Robot-learning teams get stronger adaptation at near-standard training prices.

Catalog record

CREATORWenxuan Song, Han Zhao, Fuhao Li, Ziyang Zhou, Xi Wang, Jing Lyu, Pengxiang Ding, Yan Wang, Donglin Wang, Haoang Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47444
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models Wenxuan Song, Han Zhao, Fuhao Li, Ziyang Zhou, Xi Wang, Jing Lyu, Pengxiang Ding, Yan Wang, Donglin Wang, Haoang Li Abstract This paper proposes a novel approach to address the challenge that pretrained VLA models often fail to effectively improve performance and reduce adaptation costs during standard supervised finetuning (SFT). Some advanced finetuning methods with auxiliary training objectives can improve performance and reduce the number of convergence steps. However, they typically incur significant computational overhead due to the additional losses from auxiliary objectives. To simultaneously achieve the enhanced capabilities of auxiliary training with the simplicity of standard SFT, we decouple the two objectives of auxiliary-objective SFT within the parameter space, namely, enhancing general capabilities and fitting task-specific action distributions. To deliver the goal, we only need to train the model to converge on a small-scale task set using two distinct training strategies, resulting in two finetuned models. The parameters’ difference between the two models can then be interpreted as capability vectors provided by auxiliary objectives. These vectors are then merged with pretrained parameters to form a capability-enhanced meta model. Moreover, when standard SFT is augmented with a lightweight orthogonal regularization loss, the merged model attains performance comparable to auxiliary finetuned baselines with reduced computational overhead. Internal and external experiments demonstrate that our capability vectors (1) are effective and versatile across diverse models, (2) can generalize to novel environments and embodiments out of the box. 1]HKUST (GZ) 2]Zhejiang University 3]Westlake University 4]Tsinghua University 5]Beijing Academy of Artificial Intelligence \contribution [*]Equal Contribution \contribution [†]Project Lead \contribution [‡]Corresponding Author \metadata [Code] https://github.com/OpenHelix-Team/CapVector \metadata [Website] https://capvector.github.io \metadata [Weights (ready to use)] https://huggingface.co/haofuly/capvector_models_collection \correspondence Wenxuan Song at Han Zhao at 1 Introduction Vision–Language–Action (VLA) models have become a dominant paradigm in current research on robotic foundation models. They map multimodal perception into executable robotic control, exhibiting a certain degree of language following and visual generalization ability. Similar to Large Language Models (LLMs) ( 1 ; 42 ) , training VLAs typically consists of two processes: (1) A pre-training process that allows the model to learn the mapping relation between multimodal input and action output. This process is conducted on large-scale robotic datasets and costs thousands of GPU hours. (2) A finetuning process that allows the model to fit the specific task structure. However, recent studies have revealed that pre-trained models do not exhibit the expected strong generalization capability on certain complex downstream tasks. That is, merely collecting a small number of demonstrations and performing standard supervised finetuning (SFT) is often insufficient for the model to quickly adapt to the task and achieve performance significantly superior to training from scratch ( 22 ; 20 ; 9 ; 7 ) . Several approaches aim to augment the standard SFT with an auxiliary objective. By designing auxiliary training objectives ( 45 ; 32 ; 23 ; 4 ; 26 ) aimed at enhancing specific foundational capabilities, this paradigm enables the model to not only fit the target task’s action distribution but also strengthen the corresponding foundational abilities ( e.g. , spatial perception and multimodal reasoning). With appropriately designed auxiliary objectives, models can significantly reduce the number of training steps required for convergence and achieve downstream performance that surpasses that of standard SFT. Figure 1 : Illustration of our CapVector . The grey line indicates the regular finetuning from pretrained models to finetuned models, the yellow line indicates the specifically designed finetuning with auxiliary objectives, and the pink dashed line denotes the capability vectors. The outer circle denotes the region of the parameter space that yields workable performance on the data manifold, while the inner circle represents the subregion associated with superior performance. We observe that by extracting capability vectors from \mathcal{D}_{\text{ext}} and merging them to obtain a capability-enhanced meta model \theta_{\textnormal{meta}} , the performance on \mathcal{D}_{\text{down}} is improved. Variables are defined in Section 2 . Despite the above strengths, these approaches have obvious drawbacks: auxiliary objectives often introduce extra modules and additional forward passes. For example, the 3D Foundation Model in Spatial Forcing ( 23 ) requires additional computation to obtain aligned targets during training, and LaRA-VLA ( 4 ) requires training of the latent chain-of-thought tokens, which incurs extra computational overhead. As the number of downstream tasks and the scale of data grow, this overhead gradually becomes prohibitive (Appendix Section 9 ). This naturally motivates the following question: Can the beneficial properties s , induced by carefully designed finetuning procedures, be transferred into the pretrained model itself, such that the model inherently possesses s ? If so, one could rely solely on standard SFT to inherit the same training efficiency and performance improvements, without incurring additional overhead. The answer to this question is yes . Drawing inspiration from the concept of task vectors ( 17 ) , we posit the following assumption: The two gains obtained during the training process—namely, the improvement of general capabilities and the enhancement of task-specific action fitting accuracy—can be decoupled. Furthermore, the changes in the model after training can be seen as a linear combination of parameter vectors that reflect these two characteristics. Based on the assumption, we can acquire two sets of finetuned model parameters by applying the auxiliary-objective SFT and the standard SFT method to the same downstream task, respectively. The difference between these two sets of parameters can be interpreted as the capability vectors ( CapVector ). These can then be integrated into the pretrained backbone through arithmetic operations, thereby achieving model merging. The whole process is shown in Figure 1 . While prior work in this field has primarily focused on obtaining an off-the-shelf specialist model via merging ( 11 ; 16 ; 41 ) , it remains unclear whether such techniques can be employed to produce a better generalist model that is more suitable for arbitrary downstream finetuning while also delivering superior performance. After capability extraction, a lightweight orthogonal regularization loss is needed during downstream finetuning to prevent forgetting of the capability vectors. The detailed implementation is described in Section 2 . In experiments, we focus on investigating the extractable capabilities and the underlying training mechanism of this approach. Extensive experiments demonstrate that the merged meta model can achieve performance and training efficiency comparable to SFT methods with auxiliary objectives across multiple downstream tasks. Furthermore, we validate the versatility of CapVector through the experiments on diverse VLA architectures and SFT strategies. After validating the effectiveness, we derive empirical conclusions from a series of experiments on what types of downstream tasks are suitable for extracting high-quality capability vectors. Finally, internal and external experiments in the real world demonstrate its practicality and generalization to novel environments and embodiments out of the box. In summary, our core contributions are as follows: • We define and introduce the concept of the capability vector, which represents the gain in general capabilities acquired during finetuning with auxiliary objectives in the form of model parameters. By merging these capability vectors with the pretrained model, we obtain a capability-enhanced meta model. • Based on the meta model, we only need to make minimal modifications to standard SFT by introducing an orthogonal regularization loss to mitigate forgetting. This achieves both the simplicity of standard SFT

Read the complete text