Understanding Asynchronous Inference Methods for Vision-Language-Action Models
Ayoub Agouzoul · 2026
Read Understanding Asynchronous Inference Methods for Vision-Language-Action Models by Ayoub Agouzoul (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08168.txt.
Introduction
Ayoub Agouzoul compares four fixes for a timing problem in robot control with vision-language-action models. These models are slow, so by the time an action runs, the robot's view of the world is stale. Recent methods attack the delay in different places: patching at inference time, simulating delay during training, conditioning on future states, or adding a light residual correction.
Agouzoul rebuilds all four in two shared codebases with matched versions and benchmarks them on the Kinetix suite and on LIBERO manipulation tasks, sweeping delays up to twenty control steps. The controlled comparison shows how each method holds up as latency grows.
Catalog record
| CREATOR | Ayoub Agouzoul |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08168
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding Asynchronous Inference Methods for Vision-Language-Action Models Ayoub Agouzoul Abstract Vision-Language-Action (VLA) models offer a promising path to generalist robot control, but their inference latency causes observation staleness when generated actions are executed asynchronously. Several methods have been proposed concurrently to mitigate this problem: inference-time inpainting (IT-RTC), training-time delay simulation (TT-RTC), future-state-aware conditioning (VLASH), and lightweight residual correction (A2C2). Each takes a fundamentally different approach, but they have so far been evaluated independently with different codebases, base policies, and protocols. We present a systematic comparison of these four methods under controlled conditions. We develop two unified codebases that integrate all methods with harmonized library and dataset versions, and we benchmark them on the Kinetix suite with MLPMixer policies and on the LIBERO manipulation benchmark with SmolVLA, sweeping inference delays up to d = 20 d=20 control steps. A2C2’s per-step residual correction is the most effective method on Kinetix, holding above 90% solve rate up to d = 8 d{=}8 , and also leads on LIBERO from d ≥ 4 d{\geq}4 onwards. IT-RTC is competitive at low delays but degrades sharply under long chunks ( H = 30 H{=}30 ) and high delays. TT-RTC is the most robust training-based method: stable across d max d_{\max} choices, generalizes beyond its training delay distribution, and adds zero inference overhead. VLASH exhibits a clear low-delay vs. high-delay trade-off governed by the fine-tuning delay range [ 0 , d max ] [0,d_{\max}] . Code is available at https://github.com/TheAyos/async-vla-inference . 1 Introduction Vision-Language-Action (VLA) models have emerged as a powerful paradigm for generalist robot control, mapping visual observations and natural-language instructions to robot actions [ 1 , 2 , 3 , 4 , 5 ] . A key design choice in modern diffusion- and flow-matching-based VLA policies is action chunking [ 6 ] : instead of predicting one action per observation, the policy predicts a sequence of H future actions in a single forward pass, reducing compounding errors and amortizing inference cost. However, action chunking interacts poorly with the high inference latency of large VLA models—typically tens to hundreds of milliseconds [ 7 , 8 ] . By the time a chunk is ready, the robot’s state and environment have already moved on, so executed actions are conditioned on a stale observation and may no longer be optimal. Synchronous execution avoids this problem by halting the robot during inference, but caps the control frequency at the policy’s end-to-end inference rate, which is impractical for tasks requiring fast reaction times. Naive asynchronous execution maintains a high control rate by decoupling inference from execution and switching to the new chunk as soon as it is available [ 5 ] , but introduces jitter and discontinuities at chunk boundaries that visibly degrade task performance [ 9 ] . Four methods have recently been proposed to bridge this gap, each targeting a different failure mode: • IT-RTC [ 9 ] : training-free flow-matching guidance that inpaints fresh actions onto a partially frozen chunk at inference time. • TT-RTC [ 10 ] : training-time delay simulation that conditions the policy on action prefixes so it learns to predict only the non-stale postfix, with no inference overhead. • VLASH [ 11 ] : estimates the robot’s future state at execution time using known previous actions, and conditions the policy on this rolled-forward state. • A2C2 [ 12 ] : a lightweight correction head that runs at every control step, producing residual adjustments to the base policy’s stale actions. Despite their shared motivation, these methods have been developed and evaluated in incompatible codebases, on different benchmarks, and with different base policies. This makes direct comparison difficult and obscures the trade-offs between them. Contributions. (i) Two unified codebases ( bt-kinetix in JAX/Flax NNX, bt-libero in PyTorch/LeRobot) that integrate all four methods with harmonized libraries, datasets, and base policies. (ii) A systematic comparison on the Kinetix [ 13 ] continuous-control suite (10 dynamic environments, H\in\{16,30\} , delays up to d{=}15 ) and on the LIBERO [ 14 ] manipulation benchmark with SmolVLA (delays up to d{=}20 ). (iii) Practical insights on training cost, inference overhead, and out-of-distribution delay generalization that inform the choice of asynchronous inference strategy for future VLA design and deployment. 2 Background and Methods 2.1 The Asynchronous Inference Gap Following Black et al. 2025a , we define the inference delay d=\lfloor\delta/\Delta t\rfloor , where \delta is the wall-clock inference time and \Delta t is the control period. At time t the robot captures observation o_{t} and submits it to the policy. During the d control steps that inference takes, the robot must continue to act and so executes the tail of the previous chunk. When the new chunk A_{t}=\pi(o_{t}) is ready at step t{+}d , its first d entries correspond to elapsed timesteps and are obsolete; the robot begins executing from a_{t+d} onward. Naive asynchronous execution thus suffers from three interrelated failure modes: chunk-boundary discontinuity (the new chunk is incompatible with the just-committed actions), observation staleness (the new chunk is conditioned on a d -step-old observation), and open-loop degradation (committed actions cannot adapt during the chunk’s lifetime). 2.2 Four Methods, Four Failure Modes The four methods we compare each primarily target one of these failure modes; their workings are detailed in Appendix A . IT-RTC [ 9 ] addresses chunk-boundary discontinuity by treating the first d actions of a new chunk as known constraints and inpainting the remaining H{-}d actions during flow-matching denoising, drawing on diffusion-based image inpainting [ 15 ] . Each Euler step adds a pseudoinverse-guidance term [ 16 ] that steers the denoised actions toward satisfying the prefix constraint, with a soft exponentially decaying mask blending frozen and free actions. IT-RTC requires no retraining but invokes a backward (VJP) pass per denoising step, tripling the denoising-loop FLOPs (Section 6 ). TT-RTC [ 10 ] attacks the same problem at training time. Through three minimal architecture changes—per-token flow timesteps via adaLN-Zero [ 17 ] , a non-noisy ground-truth prefix during training, and a masked loss that fires only on the postfix—the policy learns to predict postfix actions conditioned on a committed prefix. The training-time delay d is sampled with exponential decay p(d)\propto\lambda^{d} for d\in\{0,\ldots,d_{\max}\} . At inference, TT-RTC is a drop-in replacement for IT-RTC with zero overhead. VLASH [ 11 ] directly targets observation staleness by modifying the state conditioning . When inference begins on o_{t} , VLASH rolls the proprioceptive state forward by \Delta steps using the known committed actions, and conditions the (otherwise unmodified) policy on (o_{t},\hat{s}_{t+\Delta}) rather than (o_{t},s_{t}) . A temporal-offset training augmentation samples \delta\in\{0,\ldots,\Delta_{\max}\} and trains the policy to predict the action chunk starting at t{+}\delta from the original observation o_{t} but the shifted state s_{t+\delta} , encouraging reliance on the (varying) state rather than the (fixed) image. A2C2 [ 12 ] targets open-loop degradation. Rather than modify the base policy, it trains a lightweight correction head \pi_{\text{A2C2}} that runs at every control step and emits a residual \Delta a_{t+k} added to the base action a_{t+k}^{\text{base}} . The head conditions on the latest observation, base-policy hidden features, language, and a sinusoidal time-in-chunk feature \tau_{k}=(\sin(2\pi k/H),\cos(2\pi k/H)) that signals how stale the current base action is. The head is small enough to run inline within the