Unified Video-Action Joint Denoising for Dexterous Action and Data Generation
Dingrui Wang, YuAn Wang, Jinkun Liu, Yue Zhang, Mattia Piccinini, Yu Sun Johannes Betz · 2026
Read Unified Video-Action Joint Denoising for Dexterous Action and Data Generation by Dingrui Wang, YuAn Wang, Jinkun Liu, Yue Zhang, Mattia Piccinini, Yu Sun Johannes Betz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03868.txt.
Introduction
Wang and co-authors unify video and action generation for dexterous robot hands in one model. World action models borrow video priors for robot control. Most designs narrow that prior into a policy over future actions for one observation. This team keeps the joint space broad instead. Videos and hand paths are modeled together under varied conditioning.
Their model, Donk, denoises video and action jointly. Given language, a first image, and a starting hand state, it samples future video plus two-hand trajectories as a policy. Without the image, the same model samples paired rollouts from text alone. The aligned prior thus doubles as a data engine. Evaluations cover action, video, and text-only generation. Trajectory accuracy improves while video fidelity stays strong. Text-conditioned rollouts come out smooth. Dexterous manipulation gains one recipe for both control and data synthesis.
Catalog record
| CREATOR | Dingrui Wang, YuAn Wang, Jinkun Liu, Yue Zhang, Mattia Piccinini, Yu Sun Johannes Betz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03868
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unified Video-Action Joint Denoising for Dexterous Action and Data Generation Dingrui Wang, YuAn Wang, Jinkun Liu, Yue Zhang, Mattia Piccinini, Yu Sun Johannes Betz Abstract Recent world action models leverage video foundation models by aligning broad visual-dynamics priors with executable robot actions. We revisit this alignment from a distributional perspective. Existing formulations typically narrow the aligned prior into an observation-conditioned policy distribution over future actions. In contrast, we keep the distribution broader by modeling the joint space of interaction videos and executable hand trajectories under multiple conditioning regimes. We propose Donk , a unified video-action denoising model for dexterous hands. With language, an initial image, and the initial hand state, Donk samples future videos and bimanual MANO trajectories as an action policy. Without the image condition, the same denoising architecture samples paired video-action rollouts from a text-conditioned distribution, turning the aligned video prior into a data engine. Across action, video, and text-only generation evaluations, Donk improves dexterous trajectory accuracy, preserves strong video fidelity, and produces smooth text-conditioned action rollouts under the same unified training recipe. Figure 1: Donk unifies video-action generation. Given text alone, Donk generates paired interaction videos and spatio-temporally aligned MANO hand actions; with an observed image, Donk acts as an action policy. 1 Introduction The success of vision-language models (VLMs) has motivated a growing line of vision-language-action (VLA) policies that extend language and visual understanding to robot control. The goal is to build general-purpose embodied agents that can follow language instructions and perform diverse manipulation tasks across objects, scenes, and embodiments. This is particularly challenging for dexterous manipulation: to complete a language-specified task, an agent must infer fine-grained hand-object interactions, reason about contact, anticipate object motion, and produce temporally precise actions. Existing VLA policies have made substantial progress by mapping language and visual observations directly to robot actions 22 ; 35 ; 6 ; 43 ; 51 ; 11 . However, action prediction alone treats actions primarily as output targets and does not explicitly model the physical consequences of those actions. As a result, the policy receives limited supervision about how the scene should evolve under contact, even though such evolution is crucial for dexterous manipulation, where small differences in hand pose, contact timing, and object motion can determine task success. World Action Models (WAMs) address this limitation by building action policies on top of video foundation models (VFMs). Rather than predicting actions alone, WAMs jointly predict future visual observations and actions, thereby coupling motor commands with the visual futures they are expected to produce 53 ; 25 ; 45 ; 4 ; 47 . This formulation is appealing for several reasons. First, VFMs pretrained on large-scale heterogeneous video corpora provide rich spatiotemporal priors over visual fidelity, temporal coherence, semantic controllability, human-object interaction, contact dynamics, and object motion 42 . Second, future video prediction provides a dense supervisory signal beyond sparse action labels, encouraging the model to learn physical regularities implicit in visual dynamics. Third, by aligning actions with predicted visual futures, WAMs shift action learning from pure state-action imitation toward video-action alignment, which can improve learning efficiency and generalization, especially when robot data are limited or heterogeneous. Despite these advantages, existing WAMs are still primarily formulated as observation-conditioned policies. Given a language instruction and the current visual observation, they predict future observations and the corresponding action trajectory. As illustrated in Fig. 1 , in this sense, current WAMs can be viewed as text-image-to-video-action (TI2VA) models: they condition on both text and an initial image to generate aligned future videos and actions. However, this observation-conditioned formulation captures only one instance of a broader conditional video-action generation problem. From a probabilistic perspective, existing WAMs model a conditional distribution over future videos and actions given language and an initial observation, i.e., p(\mathrm{video},\mathrm{action}\mid\mathrm{text},\mathrm{observation}) . However, the initial observation is only one possible conditioning context, rather than an inherent requirement of video-action generation. This suggests a broader formulation in which the same action-aligned generative model can operate under different conditioning contexts. A natural question therefore arises: Can such a model serve not only as an observation-conditioned policy, but also as a language-conditioned generator of robot-relevant video-action experience? We answer this question by formulating a unified text/image-conditioned video-action modeling problem. Given a language instruction, with or without an initial visual observation, the model generates both an interaction video and a spatially aligned action trajectory, namely p(\mathrm{video},\mathrm{action}\mid\mathrm{text},\mathrm{optional\ observation}) . When the initial image is provided, this formulation specializes to TI2VA and serves as an observation-conditioned policy. When the image condition is absent, it becomes text-to-video-action (T2VA), where the model generates paired visual-action supervision directly from language. This unified view turns TI2VA policy learning and T2VA data generation into two conditioning modes of the same end-to-end generative model, rather than two separate pipelines. Such a formulation is especially useful for dexterous manipulation, where paired robot visual-action trajectories are expensive to collect due to the difficulty of teleoperation, calibration, and fine-grained action annotation. By contrast, large-scale human-object interaction videos and text-conditioned video priors are abundant. A text-only T2VA branch can therefore transform the broad interaction priors of video foundation models into structured, action-aligned supervision for robot learning. Realizing this unified formulation is nontrivial. The model must preserve the visual generative capability of the pretrained VFM while also learning to produce action trajectories that are spatially synchronized and semantically consistent with the generated video. Naively injecting action prediction into video generation can interfere with video token representations and degrade visual quality 45 ; 4 . Meanwhile, post-hoc pipelines that first generate or reconstruct videos and then extract actions introduce brittle intermediate representations, temporal misalignment, and error accumulation. The key challenge is therefore to learn video and action generation jointly in a single model, while maintaining visual fidelity, action accuracy, and video-action temporal correspondence. To address this challenge, we propose Donk , a unified video-action joint denoising model for dexterous manipulation. Donk is built on a video diffusion transformer 42 ; 37 and jointly denoises video tokens and action tokens under a flow-matching paradigm 28 . Actions are represented as sequences of MANO hand parameters, providing a structured representation of fine-grained dexterous hand motion. Under image-conditioned inputs, Donk functions as a TI2VA policy, predicting both future visual observations and aligned hand actions from the current scene. Under text-only inputs, the same model functions as a T2VA data engine, generating paired interaction videos and synchronized hand-action trajectories from language instructions. By unifying these two modes within a single joint denoising framework, Donk learns video-action consistency from observed trajectories and reuses the resulting action-aligned generative prior for text-conditioned data generation. Our contributions are threefold: • We formulate text-to-video-action (T2VA) generation for dexterous manipulation, where the goal is to synthesize paired interaction videos and spatially aligned hand-action trajectories from language alone. To the best of our knowledge, this is the first exploration of T2VA as a text-only data engine for dexterous manipulation. • We propose Donk , a unified video-action joint denoising model built on a video diffusion transformer. By jointly denoising video tokens and MANO hand-action tokens within a flow-matching framework, Donk supports both observation-conditioned TI2VA policy learning and text-conditioned T2VA data generation. • We demonstrate that the unified formulation is effective for both policy learning and data generation.