Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation

Chongyang Xu, Yixian Zou ⋆ \star, Ziliang Feng, Fanman Meng, Shuaicheng Liu · 2026

Read Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation by Chongyang Xu, Yixian Zou ⋆ \star, Ziliang Feng, Fanman Meng, Shuaicheng Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11984.txt.

Introduction

Xu and co-authors present Ada3Drift, a policy that lets robots act from point clouds in a single step. Fast one-step action avoids the slow repeat passes that diffusion policies need at control time. Training shifts by degrees from plain action regression to a drifting term that shapes the action spread. Tests cover the Adroit, Meta-World, and RoboTwin suites plus five real-world tasks. Ada3Drift posts the best average success rates among the methods compared, using one pass per action chunk.

Catalog record

CREATORChongyang Xu, Yixian Zou ⋆ \star, Ziliang Feng, Fanman Meng, Shuaicheng Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39218
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11984

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation Chongyang Xu, Yixian Zou ⋆ \star, Ziliang Feng, Fanman Meng, Shuaicheng Liu Abstract Diffusion-based visuomotor policies effectively capture multimodal action distributions through iterative denoising, but their high inference latency limits real-time robotic control. Recent flow matching and consistency-based methods achieve single-step generation, yet sacrifice the ability to preserve distinct action modes, collapsing multimodal behaviors into averaged, often physically infeasible trajectories. We observe that the compute budget asymmetry in robotics (offline training vs. real-time inference) naturally motivates recovering this multimodal fidelity by shifting iterative refinement from inference time to training time. Building on this insight, we propose Ada3Drift, which learns a training-time drifting field that attracts predicted actions toward expert demonstration modes while repelling them from other generated samples, enabling high-fidelity single-step generation (1 NFE) from 3D point cloud observations. To handle the few-shot robotic regime, Ada3Drift further introduces a sigmoid-scheduled loss transition from coarse distribution learning to mode-sharpening refinement, and multi-scale field aggregation that captures action modes at varying spatial granularities. Experiments on three simulation benchmarks (Adroit, Meta-World, and RoboTwin) and real-world robotic manipulation tasks demonstrate that Ada3Drift achieves state-of-the-art performance while requiring 10 × 10\times fewer function evaluations than diffusion-based alternatives. 1 Introduction Learning visuomotor policies from expert demonstrations has achieved remarkable progress [ 3 , 41 , 15 , 9 , 1 , 7 ] . A central challenge is handling the multimodal nature of human demonstrations: for a given observation, multiple valid action strategies may exist. Diffusion Policy [ 5 ] and its 3D extension DP3 [ 37 ] addressed this challenge through iterative denoising, which progressively refines noisy predictions and naturally preserves distinct action modes. This capability has driven a wave of diffusion-based robotic systems [ 20 , 23 , 18 , 2 , 32 ] , yet the underlying multi-step denoising process requires 10–100 function evaluations (NFE) per action chunk, creating a fundamental tension with the 10–50 Hz control frequencies demanded by real-time robotic manipulation. Recent one-step generation methods based on flow matching [ 39 , 28 , 14 ] , consistency distillation [ 25 , 19 ] , and diffusion distillation [ 35 , 16 ] resolve the latency bottleneck by collapsing the generation process into a single forward pass. Yet this speedup comes at a cost: without iterative refinement, their regression-based objectives converge to the conditional expectation of target fields [ 10 ] , averaging over distinct action modes rather than preserving them. In image generation [ 8 ] , such averaging produces blurry but recognizable outputs. In robotic manipulation, by contrast, the averaged trajectory of two valid strategies ( e.g . , reaching left vs. right around an obstacle) may drive the gripper straight into the obstacle ( Fig. 1 ). This qualitative difference elevates mode averaging from a quality issue to a safety concern in embodied systems. We observe that robotic systems possess a structural property that enables recovering multimodal fidelity without sacrificing single-step efficiency: compute budget asymmetry . Training is performed offline with no latency constraints, while inference must satisfy strict real-time requirements. Current one-step methods discard iterative refinement altogether. We propose instead to relocate it: perform all mode-preserving refinement during training, and deploy a single-step generator at inference. The second challenge is adaptivity . Unlike large-scale image generation where a fixed recipe succeeds across millions of samples, the few-shot and multi-task nature of robotic learning demands mechanisms that adjust to varying conditions: the number of demonstrations per task is small (typically 10–50), and the geometry of action distributions varies drastically across tasks. \times (a) Flow Matching mode avg. \triangle (b) Mean Flow biased V(\hat{\mathbf{x}}) ✓ ✓ (c) Adaptive Drifting p_{0} p_{1} generation Figure 1 : Single-step generation under multimodal action distributions. (a) Flow Matching: straight conditional paths collapse distinct modes into a single mode average, producing unsafe averaged actions. (b) Mean Flow: adaptive weighting adjusts paths but still converges to a biased mean between modes. (c) Adaptive Drifting (Ours): the drifting field V(\hat{\mathbf{x}}) steers predictions toward true demonstration modes during training via attraction and repulsion, enabling accurate multimodal recovery with 1 NFE at inference. We address both challenges in Ada3Drift ( Ada ptive 3D Training-Time Drift ing). Ada3Drift employs a training-time drifting field that steers the model’s output distribution toward demonstration modes, while tailoring the mechanism to the robotic regime through sigmoid-scheduled loss transition for few-shot learning, multi-temperature field aggregation for variable action geometries, and a timestep-free architecture enabled by the removal of inference-time iteration. We summarize our contributions as follows: • We analyze the mode-averaging problem that arises when accelerating diffusion-based policies to single-step generation, and show that it is qualitatively more severe in robotic action spaces than in image generation. We propose to recover multimodal fidelity by relocating iterative refinement to training time, guided by the compute budget asymmetry of robotic systems. • We propose Ada3Drift, which realizes this paradigm with designs tailored to the few-shot robotic regime, including sigmoid-scheduled training, multi-scale drifting field aggregation, and a simplified timestep-free architecture. • Extensive experiments on Adroit [ 27 ] , Meta-World [ 21 ] , RoboTwin [ 22 ] , and real-world manipulation tasks show that Ada3Drift achieves state-of-the-art performance with only 1 NFE, matching or exceeding methods that require 10\times more inference compute. 2 Related Work 2.1 Visuomotor Policy Learning Learning robotic manipulation policies from visual observations has evolved rapidly. Early behavior cloning approaches learn direct mappings from images to actions [ 15 , 9 ] , while recent transformer-based architectures scale to diverse tasks via language conditioning [ 3 , 41 ] or multi-task pretraining [ 23 , 2 , 33 ] . Action representation has similarly progressed: explicit regression [ 40 ] struggles with multimodal demonstrations, motivating energy-based [ 9 ] and generative formulations [ 5 ] . The choice of visual representation significantly impacts spatial reasoning capability. While 2D image-based policies [ 3 , 5 ] suffice for planar tasks, they suffer from depth ambiguity and occlusion in 3D manipulation. This has driven a shift toward explicit 3D representations: voxel grids [ 29 ] , neural feature fields [ 36 , 11 ] , and point clouds [ 12 , 37 ] . Among these, point clouds offer a favorable balance of geometric fidelity, computational efficiency, and compatibility with downstream policy architectures. Our work builds on this 3D point cloud paradigm, focusing on the orthogonal challenge of preserving multimodal fidelity under single-step generation. 2.2 Diffusion Models in Robotics Diffusion models [ 13 ] have become a dominant paradigm for representing multimodal action distributions in robot learning. Diffusion Policy [ 5 ] formulated policy learning as a conditional denoising process that naturally captures multimodal demonstration strategies. This formulation has been extended along multiple axes: DP3 [ 37 ] introduced 3D point cloud conditioning, achieving strong results on Meta-World [ 21 ] and Adroit [ 27 ] ; hierarchical variants [ 20 ] decompose long-horizon tasks into sub-goal diffusion; foundation models such as Octo [ 23 ] and RDT-1B [ 18 ] scale diffusion policies to hundreds of thousands of trajectories; and motion planning methods [ 4 ] apply diffusion to trajectory optimization. Despite their expressiveness, all these methods rely on iterative denoising, typically requiring 10–100

Read the complete text