Geometry Guided Self-Consistency for Physical AI

Yinwei Dai Zhuofu Chen, Lijie Yang Ravi Netravali · 2026

Read Geometry Guided Self-Consistency for Physical AI by Yinwei Dai Zhuofu Chen, Lijie Yang Ravi Netravali (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08638.txt.

Introduction

Yinwei Dai, Zhuofu Chen, Lijie Yang, and Ravi Netravali make robot action generators more reliable by sampling several futures and picking the consensus. Diffusion based physical AI drafts action chunks from noise, so a single draft per round can be brittle, and errors stack across a long task. Their KeyStone method draws parallel candidate chunks, groups them by closeness, and returns the middle member of the biggest group.

Action chunks live in a geometric space where distance truly means physical likeness, so no trained judge is needed. Spare memory bandwidth runs the extra drafts with almost no added delay. Across robot and world models, success rates rise clearly at no training cost.

Catalog record

CREATORYinwei Dai Zhuofu Chen, Lijie Yang Ravi Netravali
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.08638

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Geometry Guided Self-Consistency for Physical AI Yinwei Dai Zhuofu Chen, Lijie Yang Ravi Netravali Abstract State-of-the-art physical AI models generate a chunk of actions per inference through diffusion or flow matching, iteratively refining an initial noise sample into an action trajectory. Because this inference process is inherently stochastic, committing to a single trajectory per round is brittle, and this brittleness compounds across the many sequential rounds that comprise a complete episode. We introduce KeyStone, an inference-time self-consistency method for diffusion-based action generation that draws K K candidate action chunks in parallel from a shared model context, clusters them in continuous action space, and returns the medoid of the largest cluster – no additional model required. Two properties make this practical. First, the compact nature of action trajectories makes diffusion inference memory-bandwidth bound, leaving spare compute capacity to run K K chains in parallel with no additional wall-clock latency. Second, unlike token or pixel spaces where distance carries no semantic meaning and selection requires a learned judge, action chunks are geometrically structured such that Euclidean distance directly reflects physical similarity, making selection principled and judge-free. Across diverse vision-language-action models (VLAs) and world-action models (WAMs), KeyStone improves task success rates by up to 13.3% over single-trajectory sampling with negligible latency overhead, while having on par accuracy with model-based selectors at no training cost. We open source KeyStone at https://github.com/dywsjtu/keystone . 1 Introduction Diffusion and flow-matching models have become the dominant paradigm for action generation in physical AI. They power vision-language-action models (VLAs) ( 3 ; 8 ; 18 ; 20 ; 26 ) as well as world-action models (WAM) ( 25 ; 24 ; 10 ) that couple action generations with a future-observation prediction. Across this family, the action generation paradigm is the same: at each call, the model denoises an initial noise sample into a chunk of actions, which is then executed open-loop on the robot before the model is queried again for the next chunk. This design captures the multimodal action distributions inherent in manipulation tasks and produces a smooth, temporally coherent control over continuous action spaces. A complete physical AI task, therefore, consists of many sequential rounds of action chunk sampling and execution. This sampling-and-execution loop, however, is inherently stochastic. At each round, the policy starts from a fresh noise sample and iteratively refines it into an action chunk; different initial noises lead to different chunks, and not all of them succeed. Some fall into failure regions of the action space—chunks that, when executed, derail the task even when prior rounds were successful (Fig. 1 b). Worse, this brittleness compounds: an episode comprises many rounds of chunk sampling (Fig. 1 a), so even a small per-round probability of landing in a failure region accumulates into a substantial episode-level failure rate. Committing to a single sample per round, the standard practice today, leaves task success at the mercy of sampling noise. Figure 1: Brittleness of single-trajectory sampling in physical AI. (a) A complete episode consists of many rounds of action chunk sampling and open-loop execution. Each round’s stochastic sampling can derail the task. (b) At each round k , we draw many candidate chunks from GR00T N1.6 ( 18 ) on SIMPLER ( 13 ) , execute each to episode end, and label the outcome success or failure (chunks are projected via per-round PCA). success chunks cluster densely while failure ones scatter. The inset shows the corresponding end-effector paths in 3D: success chunks lead to closer 3D trajectories that reach the goal, while a failure one clearly diverges. KeyStone exploits this by sampling multiple action chunks and selecting the medoid of the largest cluster at each round ( green stars ); chained together, these selections form the KeyStone trajectory . A natural response to sampling stochasticity is test-time scaling: draw multiple candidates and select (e.g., self-consistency ( 21 ) ), as studied extensively in language models ( 21 ; 5 ; 14 ) . Recent work brings this paradigm to physical AI ( 12 ; 23 ) , sampling multiple action candidates and selecting among them with a learned selector trained on the physical AI model’s fine-tuning dataset to score how well a candidate matches the in-distribution success modes. This adds non-trivial costs: an extra model to train per task and embodiment (hundreds of H100 hours in TACO 23 for \pi_{0.5} on LIBERO 15 , which exceeds \pi_{0.5} ’s fine-tuning itself), plus an extra forward pass per round at inference. Our key observation is that the geometry of sampled action chunks is surprisingly powerful: it directly encodes the model’s concentration on in-distribution behaviors, making the overheads associated with explicit learned selectors largely unwarranted. Action chunks live in a continuous space whose structure is constrained – every action is bounded by the robot’s physical degrees of freedom – so the Euclidean distance between two action chunks directly reflects whether they would drive the robot to similar execution trajectories. Unlike token or pixel spaces where Euclidean distance carries no semantic meaning (necessitating judges to bridge the gap), action spaces are physically constrained and low-dimensional, making distance a direct proxy for behavioral similarity. The model’s concentration on in-distribution behaviors therefore manifests directly as a dense cluster of geometrically proximate candidates; Fig. 1 b shows this structure empirically. Capitalizing on these insights, we present KeyStone , an inference-time self-consistency method that, at each round, draws K candidate action chunks in parallel from a single shared context, clusters them in the continuous action space, and returns the medoid of the largest cluster as the chunk to execute. Importantly, because diffusion inference for action generation is memory-bandwidth bound, KeyStone’s parallel sampling fits within spare compute capacity and adds no additional wall-clock latency per round. This is because, unlike image and video diffusion, where the diffusion processes are compute-bound due to the large input dimensions, action diffusion operates over compact trajectories (typically O(100) dimensions), shifting the bottleneck from compute to memory bandwidth and leaving substantial parallel compute headroom unused. We evaluate across diverse VLAs and WAMs, and workloads, setting K to the largest value that preserves per-round latency, and find that KeyStone improves task success rates by up to 13.3% over single-trajectory sampling and is comparable with the model-based test-time scaling approach TACO at zero training cost. More broadly, KeyStone shows that physical AI models inherently possess everything needed for successful self-consistency: the geometry to select without a judge, and the compute headroom to sample without added latency. 2 Background and Related Work Action generation via diffusion and flow matching. Physical AI models map observational inputs—camera images, robot states, task instructions—to actions, low-level motor commands such as gripper signals or joint positions. To meet real-time constraints, state-of-the-art models output action chunks , or short sequences of future actions generated by iteratively refining an initial noise sample via diffusion or flow matching. This paradigm underlies VLAs such as \pi_{0} ( 8 ) , GR00T ( 18 ) , and SmolVLA ( 20 ) , as well as world-action models (WAMs) such as FastWAM ( 25 ) , Cosmos ( 10 ) , and DreamZero ( 24 ) , which additionally predict future observations such as video frames. Test-time scaling in text and image generation. Sampling multiple candidates and

Read the complete text