KineVLA: Towards Kinematics-Aware Vision-Language-Action Models with Bi-Level Action Decomposition

Gaoge Han, Zhengqing Gao, Ziwen Li, Jiaxin Huang, Shaoli Huang, Fakhri Karray, Mingming Gong, Tongliang Liu · 2026

Read KineVLA: Towards Kinematics-Aware Vision-Language-Action Models with Bi-Level Action Decomposition by Gaoge Han, Zhengqing Gao, Ziwen Li, Jiaxin Huang, Shaoli Huang, Fakhri Karray, Mingming Gong, Tongliang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17524.txt.

Introduction

Han and co-authors present ExecVLA, a vision-language-action model that follows fine execution detail. Two robots can finish a goal in unlike ways, and instructions often fix the how. Their bi-level action split separates the goal part from the execution part. Extra objectives keep goals stable while execution traits stay distinct. Data come from simulation and a real Realman-75 robot. Constraint following improved sharply over goal-only baselines, they report.

Catalog record

CREATORGaoge Han, Zhengqing Gao, Ziwen Li, Jiaxin Huang, Shaoli Huang, Fakhri Karray, Mingming Gong, Tongliang Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59842
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.17524

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

KineVLA: Towards Kinematics-Aware Vision-Language-Action Models with Bi-Level Action Decomposition Gaoge Han, Zhengqing Gao, Ziwen Li, Jiaxin Huang, Shaoli Huang, Fakhri Karray, Mingming Gong, Tongliang Liu Abstract In this paper, we introduce a novel kinematics-rich vision-language-action (VLA) task, in which language commands densely encode diverse kinematic attributes (such as direction, trajectory, orientation, and relative displacement) from initiation through completion, at key moments, unlike existing action instructions that capture kinematics only coarsely or partially, thereby supporting fine-grained and personalized manipulation. In this setting, where task goals remain invariant while execution trajectories must adapt to instruction-level kinematic specifications. To address this challenge, we propose KineVLA, a vision-language-action framework that explicitly decouples goal-level invariance from kinematics-level variability through a bi-level action representation and bi-level reasoning tokens to serve as explicit, supervised intermediate variables that align language and action. To support this task, we construct the kinematics-aware VLA datasets spanning both simulation and real-world robotic platforms, featuring instruction-level kinematic variations and bi-level annotations. Extensive experiments on LIBERO and a Realman-75 robot demonstrate that KineVLA consistently outperforms strong VLA baselines on kinematics-sensitive benchmarks, achieving more precise, controllable, and generalizable manipulation behaviors. The code and dataset will be released publicly. 1 Introduction Figure 1 : Comparison of Vanilla VLAs vs. KineVLA. Vanilla VLAs ( 13 ; 23 ) , which accept coarse command inputs (e.g., “place the wine bottle on the drawer”) and produce relatively fixed bottle label orientation actions. In contrast, Our KineVLA can processes fine kinematic commands (e.g., “control the wine bottle to face a specific orientation on the cabinet”) and generate robot end-effector actions oriented front, back, left, or right. Vision-Language-Action (VLA) models have recently demonstrated strong performance in goal-oriented robotic manipulation. However, most existing approaches implicitly assume goal equivalence under instruction paraphrasing, meaning that instructions describing the same task goal can be executed with interchangeable motion trajectories. As a result, these models are largely insensitive to fine-grained kinematic specifications expressed in natural language. In real-world human robot interaction, this assumption frequently breaks down. Semantically equivalent goals may require distinct and non-interchangeable trajectories due to explicit kinematic constraints such as position, orientation, distance, and motion direction. For example, as shown in Figure 1 , instructions such as “place the bottle on the drawer” and “place the bottle on the drawer with the label facing right, front, or back” describe an identical task goal, yet they demand fundamentally different execution trajectories and end-effector motions. These trajectories cannot be treated as interchangeable refinements of the same action without violating the instruction semantics. This setting goes beyond conventional fine-grained goal specification and instead characterizes a class of tasks that are goal invariant but kinematics variant. Such tasks are not captured by the implicit assumptions underlying most existing VLA formulations. Figure 2 : A task example from our proposed Kinematics-Rich Datasets : In contrast to coarse task instructions that aim to complete a general goal, our approach captures diverse, fine-grained kinematic variations for an action instruction and their temporal evolution across multiple key action stages. These variations encompass {A} Object Part, {B} Action Constraint, and {C} Target Relations, with corresponding images of key actions displayed from left to right. Our KineVLA method is designed to address this challenge, excelling at perceiving these multi-faceted details to achieve precise manipulation. Despite the rapid progress in recent VLA models such as OpenVLA ( 13 ) and \pi_{0} ( 3 ) , most existing approaches focus on semantic-level grounding, mapping visual observations and language descriptions to discrete action tokens or low-frequency control trajectories. While these models generalize well to diverse manipulation tasks, their generated actions typically reflect task goal rather than motion precision. In other words, current VLAs are optimized to determine what to do (e.g., grasp or place) but not how to do it (e.g., at which position, distance, or orientation). Moreover, although tokenization-based or diffusion-based decoders improve temporal smoothness, they remain agnostic to fine-grained kinematic cues in language. This gap prevents existing policies from achieving physically consistent, user-customized manipulations that depend on explicit kinematic instructions. Naive binning struggles with complex actions, while diffusion-based policies are computationally expensive. We therefore adopt vector-quantized action representations ( 14 ; 23 ; 18 ; 22 ) to capture fine-grained motion and temporal consistency for more stable, sample efficient training. To avoid the entanglement of task identity and kinematic realization in single-level tokenizers, we further introduce a bi-level action representation that decouples goals and kinematics, making it well suited for kinematics-rich tasks. In order to achieve greater sensitivity to sophisticated fine-grained kinematic actions while preserving robustness to coarse-grained actions, KineVLA proposes a bi-level vector-quantized action discretization scheme, which decomposes robot actions into two complementary latent spaces: a goal-level codebook that captures semantic goals and task goal, and a kinematics-level codebook that encodes precise motion parameters such as direction, distance, and velocity. Bi-level action tokens structure the action space but do not by themselves guarantee correct grounding of kinematic language. We therefore introduce bi-level reasoning tokens as explicit intermediate variables to align instruction-level constraints with hierarchical action representations. Bi-level reasoning tokens parse the task goal and kinematic details of the command. Thus resulting in a bi-level chain-of-thought (CoT)-style generation paradigm, in which the model jointly generates textual reasoning ( Reasoning ) and discrete action tokens ( Action ). To ensure consistency between reasoning and control, we maximize the conditional mutual information I(\textit{Reasoning};\textit{Action}\mid C) , which encourages the generated trajectory to remain semantically grounded, thus forming a coherent link between reasoning and execution. Our reasoning tokens are not intended to improve abstract planning ability, but to serve as explicit, supervised intermediate variables that align language and action. To alleviate the data scarcity of this new task, we constructed three Kinematics-Rich Datasets that systematically captures a variety of goals and multi-level kinematic details, covering diverse tabletop organization and object manipulation scenarios. The total number of frames are 378K. We collected paired VLA data using the LIBERO platform ( 15 ) and the Realman-75 robot. The kinematics-rich language instructions are also annotated with reasoning texts at two levels: 1) general coarse-grained goal, and 2) fine-grained kinematics. Our kinematics-rich VLA datasets capture the entire process of action execution, not only coarsely or partially, as illustrated in Figure 2 . The dataset encompasses diverse actions that encode object-part interactions, target spatial relations, and kinematic action constraints. Such a detailed annotation scheme enables the model to learn highly precise motion representations. Extensive experiments show that KineVLA achieves state-of-the-art performance on kinematics-aware benchmarks. Unlike previous models that produce relatively rigid, pre-defined, and goal-directed motions, our model performs flexible, interpretable, and kinematics-aware operations directly from kinematics-rich instructions. In summary, our main contributions are as follows: • We introduce a new kinematic-rich manipulation task, which bridges semantic goals with diverse action execution trajectories, with the aim of meeting the needs of users’ for customized table organization tasks. • We present KineVLA , a VLA model with bi-level vector quantized action representation that decouples task goals from kinematic variations, enabling interpretable and kinematically sensitive control. • We construct three new Kinematics-Aware Manipulation Dataset covering both simulated and real-world scenarios, with detailed spatial, object-level, and motion-constraint annotations to facilitate kinematic reasoning. 2 Related Work 2.1 Vision-Language-Action Models In recent years, VLA models have rapidly progressed from early

Read the complete text