Real-Time Robot Execution with Masked Action Chunking

Haoxuan Wang, Gengyu Zhang, Yan Yan, Yuzhang Shang, Ramana Rao Kompella, Gaowen Liu · 2026

Read Real-Time Robot Execution with Masked Action Chunking by Haoxuan Wang, Gengyu Zhang, Yan Yan, Yuzhang Shang, Ramana Rao Kompella, Gaowen Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20130.txt.

Introduction

Haoxuan Wang and five colleagues work on making robots act in real time. A robot that pauses to think can react too late, since the world keeps moving while it plans. Recent systems use asynchronous inference: the next chunk of actions is predicted while the current chunk still runs.

The authors present a method based on masked action chunking for this setting. The aim is smooth control with no gaps between what the robot does now and what it plans next. Fast, steady action matters for robots that handle objects or move through changing spaces, where even small delays can spoil performance.

Catalog record

CREATORHaoxuan Wang, Gengyu Zhang, Yan Yan, Yuzhang Shang, Ramana Rao Kompella, Gaowen Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20130

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-Time Robot Execution with Masked Action Chunking Haoxuan Wang, Gengyu Zhang, Yan Yan, Yuzhang Shang, Ramana Rao Kompella, Gaowen Liu Abstract Real-time execution is essential for cyber-physical systems such as robots. These systems operate in dynamic real-world environments where even small delays can undermine responsiveness and compromise performance. Asynchronous inference has recently emerged as a system-level paradigm for real-time robot manipulation, enabling the next action chunk to be predicted while the current one is being executed. While this approach achieves real-time responsiveness, naive integration often results in execution failure. Previous methods attributed this failure to inter-chunk discontinuity and developed test-time algorithms to smooth chunk boundaries. In contrast, we identify another critical yet overlooked factor: intra-chunk inconsistency , where the robot’s executed action chunk partially misaligns with its current perception. To address this, we propose REMAC, which learns corrective adjustments on the pretrained policy through masked action chunking, enabling the policy to remain resilient under mismatches between intended actions and actual execution during asynchronous inference. In addition, we introduce a prefix-preserved sampling procedure to reinforce inter-chunk continuity. Overall, our method delivers more reliable policies without incurring additional latency. Extensive experiments in both simulation and real-world settings demonstrate that our method enables faster task execution, maintains robustness across varying delays, and consistently achieves higher completion rates. Video demos are available at this project page . 1 Introduction The deployment of large-scale models has revolutionized AI capabilities across various domains ( 23 ; 9 ; 40 ; 26 ) . In robotics, Vision-Language-Action (VLA) models ( 34 ; 7 ; 6 ; 18 ; 3 ; 14 ; 37 ; 41 ) have emerged as a promising direction for translating human instructions and sensory observations into physical actions. While real-time responsiveness is desirable across all such applications, its importance varies by context. For language or image models, slower responses primarily lead to increased waiting time but won’t result in generation failure. In contrast, for cyber-physical systems like robots operating in dynamic, real-world environments, the absence of real-time responsiveness can mean the difference between successfully filling a cup and catastrophically spilling juice onto the table. A real-time robot control system must ensure a continuous stream of executable actions so that the robot never runs idle ( 11 ) . A straightforward approach is to reduce the time cost of action acquisition—primarily arising from VLA action generation and network communication—such that it falls below the robot’s control period. Although a variety of model-level techniques have been proposed to accelerate model prediction ( 20 ; 5 ; 35 ; 21 ; 31 ; 36 ; 32 ) , these methods typically sacrifice accuracy for speed. Moreover, as model sizes continue to scale, control frequencies increase, and deployment conditions vary in real-world settings, such approaches become increasingly limited in their ability to guarantee real-time performance. Action chunking ( 39 ; 8 ) , where a policy predicts and executes a sequence of actions per inference step, offers a partial solution to real-time control. Although it has achieved state-of-the-art results in dexterous manipulation, action chunking inherently reduces system reactivity to sudden state changes ( 24 ) and introduces discontinuities at chunk boundaries ( 4 ) . These limitations have motivated the search for more general system-level inference frameworks, among which asynchronous inference ( 32 ) has recently emerged as a promising solution for real-time robotic execution. Unlike synchronous inference ( 8 ) , which pauses until future actions are available, asynchronous inference predicts upcoming actions while executing the current ones, ensuring a continuous supply of actions. However, incorporating asynchronous inference into VLA-based control systems is nontrivial: when coupled with action chunking, it amplifies existing limitations and results in substantial performance degradation ( 24 ; 4 ; 32 ) . Previous methods ( 39 ; 24 ; 4 ) addressing this challenge primarily aimed to mitigate inter-chunk discontinuity , seeking a balance between long-term coherence and short-term reactivity. These approaches typically treat the already executed actions as informative priors and apply test-time refinements to the upcoming action chunks. However, such refinements are either heuristic in nature and prone to catastrophic failure ( 39 ) , or exploit the generative and inpainting capabilities of diffusion/flow-matching models ( 33 ) while incurring additional latency ( 24 ; 4 ) . Moreover, these works overlook a critical failure mode: intra-chunk inconsistency , arising from misalignment between observations and the actions executed within a single chunk. In this work, we aim to improve both inter-chunk continuity and intra-chunk consistency when integrating asynchronous inference with chunking policies. We first formulate intra-chunk inconsistency as a partial mismatch between observations and executed actions, leading to a shift in intra-chunk distribution between training and sampling. To address this, we propose R eal-time E xecution with M asked A ction C hunking (REMAC) to learn corrective adjustments to the pretrained policy by masking arbitrary portions of action chunks. In parallel, we enhance inter-chunk continuity by refining the sampling pipeline to incorporate previously executed actions as informative priors. Our method introduces no additional inference delay compared to the pretrained policy ( 3 ; 4 ) , and can be seamlessly integrated into existing VLA frameworks. Extensive experiments across 12 simulated tasks and three real-world settings demonstrate its effectiveness, showing higher success rates, faster task completion, and smoother robot dynamics under varying delay conditions. Moreover, our approach can be combined with existing test-time algorithms, further underscoring its utility to produce stronger backbone policies for asynchronous execution. 2 Related Works 2.1 VLA and Action Chunking An emerging line of robotics research focuses on developing generalist policies capable of performing a broad spectrum of tasks and transfer across different environments and robot embodiments. Vision-Language-Action (VLA) models ( 7 ; 6 ; 34 ; 18 ; 3 ; 2 ; 14 ; 32 ) have become a leading approach in this pursuit. By conditioning on high-level inputs such as video frames, natural language instructions, and proprioceptive signals, VLAs generate the corresponding low-level motor control commands. Among these models, diffusion and flow-matching approaches ( 22 ; 8 ) have gained prominence for their ability to generate high-quality actions efficiently, and they serve as the representative setting for our work. Moreover, scaling to long-horizon manipulation requires temporal abstraction: models must predict not just the next command, but coherent segments of behavior. Inspired by principles of human motor control ( 19 ) , recent systems structure behavior into temporally extended sequences, or action chunks , which are generated by the VLA and executed by a low-level controller. This action-chunking paradigm enables visuomotor policies to operate over meaningful sequences and provides a foundation for alternative strategies of action execution. 2.2 Execution Strategies for Chunked Policies While action chunks are temporally consistent within a single segment, discontinuities and distribution shifts often arise at chunk boundaries. To mitigate this issue, 39 proposed Temporal Ensembling (TE), which aggregates the overlapping portions of consecutive chunks to improve smoothness. 24 highlighted the importance of balancing long-term consistency with short-term reactivity, introducing Bidirectional Decoding (BID), a method that samples multiple candidate predictions and selects the optimal one. More recently, RTC ( 4 ) explored real-time execution strategies under asynchronous inference by framing the task as a test-time inpainting ( 29 ) problem, proposing

Read the complete text