Learning High-Frequency Continuous Action Chunks in Latent Space
Kunyun Wang, Yuhang Zheng, Yupeng Zheng, Jieru Zhao, Wenchao Ding · 2026
Read Learning High-Frequency Continuous Action Chunks in Latent Space by Kunyun Wang, Yuhang Zheng, Yupeng Zheng, Jieru Zhao, Wenchao Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24931.txt.
Introduction
Kunyun Wang leads a fix for robot policies that must act at high rates. Chunking actions helps at moderate speeds but breaks down near sixty cycles per second. At such rates, policies struggle to stay smooth over time and consistent in space. The team moves high-rate action learning into a latent space built with a variational autoencoder. A Reuse-then-Refine step polishes chunk joints under async inference for smooth real-time execution. On three contact-rich tasks, robots run with fewer pauses and less jerky motion.
Catalog record
| CREATOR | Kunyun Wang, Yuhang Zheng, Yupeng Zheng, Jieru Zhao, Wenchao Ding |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57644 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24931
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning High-Frequency Continuous Action Chunks in Latent Space Kunyun Wang, Yuhang Zheng, Yupeng Zheng, Jieru Zhao, Wenchao Ding Abstract Modern robotic policies increasingly rely on action chunking to execute complex tasks in the physical world. While action chunking improves temporal consistency at moderate action frequencies, it becomes insufficient when the action frequency is further increased (e.g., to 60 Hz). At such high frequencies, policies often fail to generate actions that are both temporally smooth and spatially consistent. We address this challenge by shifting high-frequency action learning from the action space to a latent space with variational autoencoder (VAE). This formulation significantly improves both temporal and spatial consistency of high-frequency control. To enable smooth real-time execution, we further introduce Reuse-then-Refine, a chunk-level refine strategy that improves continuity between adjacent action chunks under asynchronous inference. As a result, robots controlled by our policy can execute complex contact-rich tasks continuously, with less pauses and jerky motions. Experiments on three real-world contact-rich robotic tasks show that our approach consistently completes tasks with smooth motions. Our code and data are available at https://github.com/tars-robotics/RTR . † † affiliationnotice: * Work done during an internship at TARS Robotics. \dagger Corresponding authors. Figure 1 : Action frequency shapes execution dynamics in action replay. Low-frequency actions (e.g., 15 Hz) induce stop-and-go motion with obvious velocity drops, whereas high-frequency actions (e.g., 60 Hz) enable continuous motion with stable velocities. 1 Introduction Figure 2 : (a) Original: a policy trained directly in the high-frequency action space, which produces action chunk A given an observation o , resulting in large deviation and high jerk. (b) Latent: a policy trained in a continuous latent space, where a VAE decoder reconstructs a high-frequency action chunk that is both precise and smooth. (c) Reuse-then-Refine (RTR): a method that reuses recently executed actions and refines them via the VAE to ensure continuity between consecutive action chunks under asynchronous inference. Imitation learning has emerged as a central paradigm for robotic manipulation, enabling policies to acquire complex behaviors directly from human demonstrations. A key advance in this direction is action chunking ( 9 ; 33 ) , where policies predict temporally extended action sequences instead of single-step commands, improving the modeling of complex trajectories and long-horizon dependencies. Building on this formulation, recent vision–language–action (VLA) models, such as OpenVLA-OFT ( 14 ) and PI0.5 ( 12 ) , adopt action chunk to jointly learn perception, language grounding, and physical interaction, achieving strong generalization across diverse real-world manipulation tasks. However, the effectiveness of action chunking critically depends on the action frequency at which policies are trained and executed. While action chunking preserves temporal consistency at relatively low frequencies, this property breaks down at high frequencies (e.g., 60 Hz), where policies trained directly in the action space often produce imprecise and highly jittery trajectories. A natural alternative is to train policies at a lower action frequency and interpolate the predicted action chunks to a higher frequency during execution. However, interpolation amplifies small prediction errors and fails to recover the fine-grained motion structure required for high-frequency control, resulting in trajectories that remain imprecise and jittery (Fig. 3 ). Despite these challenges, learning high-frequency actions is desirable. High-frequency actions preserve fine-grained motion details and implicitly encode velocity information, allowing robots to execute trajectories continuously without repeated acceleration and deceleration, avoiding the stop-and-go behavior typical of low-frequency control (Fig. 1 ). A fundamental reason why high-frequency actions are difficult to learn lies in their high temporal information density and fine-grained spatial variation which places a heavy burden on policy function approximation. To address this challenge, we leverage variational autoencoders (VAE) ( 16 ) to compress high-frequency, discrete action chunks into low-frequency, continuous latent representations that are more amenable to learning (Fig. 2 ). Our experiments show that policies trained in the latent space learn smoother and more consistent action chunks than those trained directly in the high-frequency action space (Fig. 3 ). Latent-space learning enables smooth and precise control within individual action chunks, but it does not by itself guarantee continuity across chunks. In real-world deployment, long-horizon tasks require policies to repeatedly generate new action chunks. To achieve real-time execution, prior work introduces asynchronous inference ( 3 ; 31 ; 23 ; 24 ) , overlapping computation with execution to hide inference latency. Under asynchronous execution, however, misalignment between consecutive chunks can induce large discontinuities at chunk boundaries (Fig. 5 (a)), leading to visible stalls and degraded execution quality. Existing approaches, such as RT-C ( 3 ) , attempt to improve chunk-level continuity by conditioning action generation on previous chunk. However, RT-C is tailored to flow-matching or diffusion models and has not been explored in latent space. Our experiments show that directly applying RT-C in the latent space is ineffective and can even degrade continuity (Table 4 ). Conversely, applying RT-C in the original high-frequency action space remains limited by the imprecision of high-frequency action chunk (Table 3 ). To improve chunk-level continuity, we introduce Reuse-then-Refine (RTR) (Fig. 5 ), a training-free method for latent policies. RTR reuses executed actions that overlap with inference, combines them with newly predicted actions, and refines the resulting sequence through the VAE to produce a continuous and smooth action chunk. RTR substantially improves chunk-level continuity (Table 4 ) and reduces execution stalls under asynchronous inference (Table 3 ). By combining latent-space policy learning with RTR, our approach enables smoother, more stable robot control with fewer stalls compared to DP, OFT, and PI0.5. Extensive real-world experiments demonstrate that: (1) learning high-frequency policies in the latent space yields higher precision and smoother trajectories; (2) RTR effectively improves chunk-level continuity and enables real-time smooth execution under asynchronous inference; and (3) smooth and continuous high-frequency action chunks lead to tangible reductions in end-to-end execution latency. Together, these results highlight the importance of representation and execution co-design for high-frequency robotic control and provide a practical pathway toward smooth, less-stall robot execution in real-world settings. Figure 3 : Comparison of action-chunk trajectories under different action representations of OpenVLA-OFT. “Fact” denotes a ground-truth action chunk from the dataset, while “Predict” denotes a policy-generated action chunk. (a) Original trains the policy directly in a high-frequency action space, resulting in poor precision and noticeable jitter. (b) Interpolate trains the policy at a low action frequency and reconstructs high-frequency actions via temporal interpolation, but suffers from reduced precision and smoothness. (c) Latent trains the policy in latent space and decodes the generated latent action using a VAE, producing the most precise and smooth trajectories. 2 Related Work 2.1 Vision–Language–Action models (VLAs) Vision–language–action (VLA) models ( 10 ; 1 ; 29 ; 34 ; 15 ; 6 ; 35 ; 14 ; 12 ; 2 ; 18 ; 7 ; 17 ) build upon large pre-trained vision–language backbones and transfer knowledge from diverse, task-agnostic datasets to robotic manipulation. Trained on large-scale robot manipulation datasets ( 11 ; 27 ; 13 ; 19 ) , these models achieve strong generalization across a wide range of real-world manipulation tasks. Rather than predicting single action at each timestep, several recent VLA approaches ( 14 ; 12 ; 2 ; 18 ; 7 ; 17 ) adopt action chunking to mitigate non-Markovian artifacts in demonstration data, such as brief tremors or pauses. However,