Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation

Hongyu Yan 1 , Qiwei Li 1 , Jiaolong Yang 2 , and Yadong Mu 1 · 2026

Read Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation by Hongyu Yan 1 , Qiwei Li 1 , Jiaolong Yang 2 , and Yadong Mu 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27670.txt.

Introduction

Yan and co-authors give robot policies a sense of task progress. Most vision-language-action models lack that awareness. Long tasks with stacked sub-goals suffer most. A progress estimator trains on large video-text robot data. Its error stays low at 0.07 in simulation. It also transfers to real scenes unseen in training. A world model then guides action tokens toward progress. Benchmarks CALVIN and LIBERO show the gains.

Catalog record

CREATORHongyu Yan 1 , Qiwei Li 1 , Jiaolong Yang 2 , and Yadong Mu 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55196
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27670

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation Hongyu Yan 1 , Qiwei Li 1 , Jiaolong Yang 2 , and Yadong Mu 1 Abstract Fig. 1. Illustration of the key ideas in ProgressVLA. (a) We pretrain a vision/language-conditioned progress estimator on Open X-Embodiment (OXE) [ 34 ] and finetune it on LIBERO [ 30 ] / CALVIN [ 32 ] . (b) The estimator serves as a vision-language evaluator and achieves low residual after being finetuned (0.07 on CALVIN and 0.1 on real-world scenarios with a progress scale of [0,1] ). (c) At inference, we use classifier (estimator) guidance in latent action space to steer diffusion toward higher progress. The refined latents are then decoded into action chunks for execution, often producing faster progress. I Abstract Most existing vision-language-action (VLA) models for robotic manipulation lack progress awareness, typically relying on hand-crafted heuristics for task termination. This limitation is particularly severe in long-horizon tasks involving cascaded sub-goals. In this work, we investigate the estimation and integration of task progress, proposing a novel model named ProgressVLA . Our technical contributions are twofold: (1) robust progress estimation : We pre-train a progress estimator on large-scale, unsupervised video-text robotic datasets. This estimator achieves a low prediction residual (0.07 on a scale of [0,1] ) in simulation and demonstrates zero-shot generalization to unseen real-world samples, and (2) differentiable progress guidance : We introduce an inverse dynamics world model that maps predicted action tokens into future latent visual states. These latents are then processed by the progress estimator; by applying a maximal progress regularization, we establish a differentiable pipeline that provides progress-piloted guidance to refine action tokens. Extensive experiments on the CALVIN and LIBERO benchmarks, alongside real-world robot deployment, consistently demonstrate substantial improvements in success rates and generalization over strong baselines. II Introduction Recent vision-language-action (VLA) models have advanced policy learning by scaling to large robotics datasets [ 34 , 42 , 23 , 5 ] . Yet many approaches still rely on dense action supervision [ 25 ] , which limits their scalability, while others depend on implicit and often noisy goal-satisfaction cues. Generative planners including tokenized autoregressive models [ 35 , 6 ] and diffusion policies [ 2 , 19 ] , can produce plausible trajectories. However, their sampling is largely driven by conditioning and typically lacks an explicit, dense notion of task progress. Consequently, long-horizon execution often relies on brittle termination heuristics rather than goal-directed generation. As a motivating fact, we present some empirical validation by us in Table I . It shows that explicitly guiding sampling with a learned progress signal substantially improves progress alignment and reduces the required steps for completing a task on CALVIN [ 32 ] , with a consistent gain in success. TABLE I: Key motivating observation. Progress-guided sampling improves progress alignment and reduces completion steps on CALVIN. Pearson r is the correlation between predicted progress \{\hat{p}_{t}\} and a linear ramp \{t/T\} over a rollout. Avg. steps denotes steps-to-completion, and Success is the task success rate. The baseline here is a standard diffusion policy for robotic manipulation. Progress Guidance Pearson r\uparrow Avg. steps \downarrow Success \uparrow 0.722 90.4 92.7 ✓ 0.934 77.3 93.6 To address these limitations, we propose a progress estimation technique (see Fig. Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation ) and integrate it to guide a diffusion policy, a framework we call ProgressVLA. Central to this approach is a progress estimator that outputs a normalized completion score, conditioned on the language-specified task and current visual observations. Why is progress estimation fundamental to long-horizon robotic manipulation? In vision-language conditioned tasks, a policy must transcend the generation of locally plausible motions; it must continuously evaluate whether its actions effectively advance toward the specified goal [ 18 ] . Without a dense notion of progress, generative policies often squander computation on trajectories that appear visually reasonable but remain task-irrelevant. Furthermore, they lack a principled mechanism for termination, often defaulting to brittle, hand-crafted heuristics. However, learning progress directly from raw pixels is difficult [ 7 ] . Robotic videos exhibit significant nuisance variations, such as camera jitter, background shifts, and distractor objects, which naive learning objectives often entangle with task dynamics. This results in progress signals that are noisy and poorly aligned with actual goal completion. ProgressVLA mitigates this by grounding progress estimation within a pre-trained, object-centric visual feature space. By explicitly conditioning on language, the model ensures the learned signal prioritizes task-relevant state changes over incidental visual dynamics. It is equally important to use progress during control. Rather than treating progress as a post-hoc evaluator, such as for reranking sampled trajectories or as a sparse success classifier [ 48 , 49 , 12 , 14 ] , ProgressVLA embeds progress awareness directly into the action generation process. Specifically, for a given candidate action chunk, we develop an inverse dynamics-based world model to predict the resulting future visual features, while the progress predictor assigns a differentiable score to the predicted outcome. We then backpropagate the progress gradients through the world model to provide classifier-style guidance during each diffusion denoising step. This steers the sampling process toward action chunks predicted to maximize progress toward the goal. By coupling planning with evaluation, progress shapes the generated trajectories and provides a simple threshold-based termination criterion, leading to more goal-directed and reliable long-horizon execution. In summary, our contributions are threefold: (1) a progress estimator grounded in predicted future observations through an inverse dynamics world model, enabling foresight in task evaluation; (2) a progress-guided diffusion sampler that leverages differentiable progress gradients to iteratively optimize action chunks during generation; and (3) extensive empirical validation on the CALVIN and LIBERO benchmarks, complemented by real-robot deployments, demonstrating significant gains in long-horizon success rates and more reliable task termination. Fig. 2: Overview of ProgressVLA. Conditioned on a language instruction and current observation, the diffusion policy first generates a candidate chunk of latent actions. A action-oriented world model then rolls out these actions within a pre-trained visual feature space to project future states, while a progress estimator assigns a completion score to the predicted outcomes. Finally, progress gradients are backpropagated through the world model as classifier guidance, steering the diffusion process toward actions that maximize task advancement. III Related Work III-A Vision-Language-Action Models Large-scale Vision-Language Models (VLMs) have established robust multimodal representations that transfer effectively across diverse perception tasks, including visual question answering and image captioning. Extending these capabilities to control settings has led to the emergence of Vision-Language-Action (VLA) models [ 31 , 38 ] , which map linguistic goals and visual observations to control policies. Current VLA research generally follows three paradigms: (i) autoregressive tokenization, which discretizes continuous control signals into action codebooks [ 25 , 35 , 6 ] ; (ii) direct supervised regression to joint action spaces when dense labels are available [ 50 , 24 , 43 ] ; and (iii) generative trajectory modeling, notably diffusion policies that sample action sequences via iterative denoising [ 13 , 22 , 2 , 19 , 5 ] . While diffusion-based methods produce high-quality, diverse trajectories through stochastic sampling, they often lack an explicit task-level signal to steer generation toward goal completion. Parallel efforts have focused on learning latent, transferable action spaces from video data to facilitate

Read the complete text