FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies
Xintong Hu Xuhong Huang Jinyu Zhang Yutong Yao Yuchong Sun [2pt] Qiuyue Wang Mingsheng Li Sicheng Xie Yitao Liu Junhao Chen, [2pt] Yixuan Chen Yingming Zheng Shuai Bai Tao Yu [8pt] XLANG Lab, The University of Hong Kong Qwen Team, Alibaba Inc.[4pt] Project Page: https://finevla.xlang.ai/ · 2026
Read FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies by Xintong Hu Xuhong Huang Jinyu Zhang Yutong Yao Yuchong Sun [2pt] Qiuyue Wang Mingsheng Li Sicheng Xie Yitao Liu Junhao Chen, [2pt] Yixuan Chen Yingming Zheng Shuai Bai Tao Yu [8pt] XLANG Lab, The University of Hong Kong Qwen Team, Alibaba Inc.[4pt] Project Page: https://finevla.xlang.ai/ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27284.txt.
Introduction
Xintong Hu and colleagues build FineVLA, a framework for teaching robots from fine-grained instructions about how a task should be done. It unifies nearly a million trajectories from ten robot datasets into a checked set of fine-grained examples, plus a benchmark for testing. Mixing fine-grained and goal-only instructions beats goal-only training, with the largest gains on pose, color, and approach direction.
Catalog record
| CREATOR | Xintong Hu Xuhong Huang Jinyu Zhang Yutong Yao Yuchong Sun [2pt] Qiuyue Wang Mingsheng Li Sicheng Xie Yitao Liu Junhao Chen, [2pt] Yixuan Chen Yingming Zheng Shuai Bai Tao Yu [8pt] XLANG Lab, The University of Hong Kong Qwen Team, Alibaba Inc.[4pt] Project Page: https://finevla.xlang.ai/ |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111608 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27284
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies Xintong Hu Xuhong Huang Jinyu Zhang Yutong Yao Yuchong Sun [2pt] Qiuyue Wang Mingsheng Li Sicheng Xie Yitao Liu Junhao Chen, [2pt] Yixuan Chen Yingming Zheng Shuai Bai Tao Yu [8pt] XLANG Lab, The University of Hong Kong Qwen Team, Alibaba Inc.[4pt] Project Page: https://finevla.xlang.ai/ Abstract Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed. However, existing robot datasets usually pair trajectories with coarse goal-level language, leaving execution-critical details such as active arm, approach direction, and contact region unspecified. This limits steerable policy learning and robotic video understanding. We introduce FineVLA , an open framework for action-aligned fine-grained VLA supervision. The framework includes: (1) a data construction tool that unifies 972,247 trajectories across 85K tasks from 10 open-source robot datasets and builds FineVLA-Data , a human-verified dataset of 47,159 fine-grained trajectories; (2) a held-out benchmark with 500 videos, 10,816 atomic facts, and 1,030 VQA questions; (3) a robotics-specialized VLM annotator for scalable fine-grained annotation; and (4) a steerable VLA policy trained with controlled mixtures of fine-grained and raw goal-level instructions. Our policy experiments yield three findings. First, fine-grained supervision does not sacrifice goal-level task success: fine-grained-only (FG-only) improves over raw-instruction-only (Raw-only) by +1.4 to +8.1 success-rate points across architecture and data-scale settings. Second, fine-grained and raw instructions are complementary: performance follows a consistent inverted-U trend, peaking around FG : Raw = 1 : 2 to 1 : 1. The strongest mixed setting reaches 86.8% / 82.5% in RoboTwin simulation and 62.7 /100 in real-world dual-arm manipulation, compared with 49.9 for Raw-only. Third, fine-grained supervision directly improves steerable control by increasing compliance with language-specified execution factors: in real-world evaluation, the largest gains over Raw-only appear on pose (+23), color (+18), and approach direction (+18)—factors where goal-level instructions provide no guidance. Overall, fine-grained language should augment goal-level instructions: specifying how to execute alongside what to achieve. 0 0 footnotetext: ∗ Equal contribution. † Corresponding authors. 1 Introduction Vision-Language-Action (VLA) models are moving from task-level robot control toward policies that can be steered by human instructions. In this work, we use steerability to mean the ability to execute the same high-level goal in different ways according to user-specified execution constraints, such as which arm to use, which target object to manipulate, how to approach it, where to make contact, which motion or rotation direction to follow, and what final configuration to achieve. Recent robot foundation models such as \pi_{0.7} , LingBot-VLA, GR00T N1.7, and GEN-1 ( 9 , 31 , 20 , 7 ) suggest that future robot policies should not only infer what task to complete, but also follow instructions about how the task should be performed. However, building open, steerable VLA systems remains challenging for three reasons. (i) Heterogeneous data and missing fine-grained annotation infrastructure. Existing open-source robot datasets use diverse action and state representations that cannot be directly unified ( 22 , 17 ) . Within the same task, demonstrations are heavily redundant, and most trajectories carry only a single goal-level description (e.g., “pick up the cup”) while the execution process—actor choice, approach direction, contact region, motion path, and state transitions—remains unspecified. The problem is not only that labels are coarse, but that there is still no open infrastructure for producing action-aligned fine-grained supervision at scale. (ii) Lack of benchmarks and scalable annotators for fine-grained robotic video understanding. Although general video-language models and dense captioning methods have advanced video description, their captions often focus on scene appearance rather than action-relevant execution details such as contact regions, approach directions, and motion paths. Existing embodied benchmarks ( 25 , 18 , 27 ) mainly evaluate spatial reasoning or hand-object dynamics, but do not systematically measure whether VLMs capture process-level manipulation details. There is also a lack of open, robotics-specialized annotators for action-aligned fine-grained captions. (iii) Unknown effectiveness and training recipe. Even if fine-grained data were available, the community lacks systematic evidence on whether action-aligned instructions improve policy learning, and what mixture of fine-grained and goal-level supervision yields the best steerable control. Figure 1 : Overview of FineVLA. FineVLA builds a closed loop for action-instruction alignment, connecting fine-grained data construction, robotic video understanding, scalable annotation, and steerable VLA policy learning. Left : FineVLA-Tool unifies heterogeneous robot trajectories from 10 open-source datasets, removes redundant demonstrations through clustering and sampling, and annotates representative trajectories with action-aligned descriptions across ten fine-grained dimensions. The resulting FineVLA-Data supports both RoboFine-Bench, which evaluates fine-grained robotic video understanding through Grounding VQA, Reasoning VQA, and Caption Evaluation, and RoboFine-VLM, a robotics-specialized VLM trained as a scalable annotator for new trajectories. Right : FineVLA-Policy is trained with mixtures of raw goal-level instructions and fine-grained process-level instructions under two action-decoding architectures, and is evaluated in both RoboTwin simulation and real-world dual-arm manipulation. The steerable-control examples illustrate how fine-grained language specifies execution-sensitive factors such as contact region, target object, active actor, trajectory and orientation, and failure recovery. To address these challenges, we introduce FineVLA , a fully open-source framework for scaling fine-grained VLA data, robotic video understanding, and steerable VLA policies (Figure 1 ). The framework operationalizes this principle through four components, each targeting one of the gaps above. (1) FineVLA-Tool + FineVLA-Data (Gap i). FineVLA-Tool unifies 972,247 trajectories across 85K tasks from 10 open-source datasets, selects representative samples via dynamic time warping (DTW)-based clustering, and annotates them with process-level descriptions across ten fine-grained dimensions (Table 9 ). This produces FineVLA-Data , a human-verified corpus of 47,159 trajectories whose average instruction length increases 10.4 \times (from 9.3 to 96.8 words). (2) RoboFine-Bench (Gap ii). We curate RoboFine-Bench—500 videos with 10,816 human-reviewed atomic facts and 1,030 VQA questions spanning all ten fine-grained dimensions, with complementary VQA and caption tracks. All benchmark trajectories are held out from both VLM fine-tuning and policy training, ensuring an independent evaluation. (3) RoboFine-VLM (Gap ii). We fine-tune Qwen3.5-397B-A17B ( 24 ) on FineVLA-Data to obtain RoboFine-VLM, a VLM specialized for robotic action understanding that serves as a scalable annotator for new trajectories. (4) FineVLA-Policy + training recipe (Gap iii). We train FineVLA-Policy under two action-decoding architectures (StarVLA-OFT and StarVLA-GR00T) and systematically vary the ratio between fine-grained (FG) and raw goal-level (Raw) instructions— keeping trajectories, actions, and visual observations fixed while changing only the paired language—to isolate the effect of action-aligned supervision. Our policy experiments yield three key findings. First, fine-grained supervision does not harm goal-level task success; instead, FG-only outperforms Raw-only across the evaluated simulation settings, with the largest gain on AlohaMix-OFT (+6.5/+4.7 points on Easy/Hard). Second, fine-grained and raw instructions are complementary: success follows an inverted-U trend over the FG:Raw ratio and peaks around 1:2–1:1, reaching 86.8% / 82.5% on AlohaMix-OFT Easy/Hard (+15.0/+11.1 over the Raw-only baseline of 71.8%/71.4%). Third, in real-world dual-arm manipulation, the FG:Raw = 1:1 policy achieves the highest average score ( 62.7 /100 vs. 49.9 for Raw-only), with the largest per-factor gains on execution-sensitive attributes such as pose (+23), color (+18), and approach direction (+18). Under identical visual scenes, varying only the fine-grained instruction produces distinctly different execution behaviors, directly demonstrating steerable control. In addition, RoboFine-VLM achieves the best performance among evaluated VLMs on the held-out RoboFine-Bench, reaching 71.0% VQA accuracy and