Reasoning-aware Speculative Decoding for Efficient Vision-Language-Action Models in Autonomous Driving

Anh-Dung Dinh, Simon Khan, Flora D. Salim · 2026

Read Reasoning-aware Speculative Decoding for Efficient Vision-Language-Action Models in Autonomous Driving by Anh-Dung Dinh, Simon Khan, Flora D. Salim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31160.txt.

Introduction

Dinh, Khan, and Salim speed up the reasoning step of vision-language-action planners for self-driving cars. These planners first write a short chain-of-causation, a plain-language account of why the car should act, and only then produce a trajectory. The text is generated token by token, so it dominates the running time. The trajectory head, by contrast, is parallel and cheap.

The team splits reasoning into two paths. A routine reasoner handles the predictable continuation, reading the car's trajectory history. The full planner handles only tokens that encode fresh commitments, where new visual evidence matters. A trick called FlatRoPE steers the routine reasoner's attention away from image tokens and onto history. A post-training stage with an action-quality reward aligns the split. Together, the two reasoners cut the reasoning step's running time by about a factor of four on the Alpamayo planner.

Catalog record

CREATORAnh-Dung Dinh, Simon Khan, Flora D. Salim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31160

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reasoning-aware Speculative Decoding for Efficient Vision-Language-Action Models in Autonomous Driving Anh-Dung Dinh, Simon Khan, Flora D. Salim Abstract Modern Vision-Language-Action (VLA) planners for autonomous driving emit a chain-of-causation (CoC) reasoning step before producing a trajectory. The reasoning is autoregressive and dominates inference latency, while the trajectory head is parallel and cheap. Latency is an operational constraint in autonomous driving, so accelerating the reasoning step is the central problem we address. We observe that CoC reasoning has two qualitatively different needs: most tokens continue routine setup that follows naturally from the ego-trajectory history, and a small fraction encode commitments that require fresh visual evidence about an unexpected situation. We split this reasoning into two specialized paths: a routine reasoner that handles the predictable continuation by attending to trajectory history, and a deliberative reasoner (the unmodified VLA target) that handles novel cases by attending to current visual evidence, using the speculative decoding framework as the architectural template for how the two paths cooperate. Unlike standard speculative decoding, our routine reasoner is not a smaller replica of the target; the two reasoners are deliberately specialized to read different parts of the prompt. We propose two techniques to realize this. First, we introduce FlatRoPE , a 1D rotary positional embedding in the draft that breaks the rotational symmetry of the target’s 3D M-RoPE, redirecting attention away from visual tokens and onto trajectory-history tokens. Second, we introduce Action-aware RL (AARL) , a post-training stage that uses an action-quality reward together with a static-reference KL anchor. Together, our two-reasoner system reduces the reasoning-step running time by approximately 4 × 4\times relative to the original Alpamayo planner. 1 Introduction Modern Vision-Language-Action (VLA) planners for autonomous driving [ 27 , 11 , 23 ] are sequential by construction: the model first emits a chain-of-causation (CoC) reasoning text (15–20 tokens) and then produces a trajectory. The reasoning is autoregressive through an 8B+ multimodal backbone; the trajectory is produced in parallel by a flow-matching action head conditioned on the backbone’s KV cache. The reasoning step is the bottleneck: it consumes an order of magnitude more wall-clock budget than the trajectory itself, and latency in driving translates directly into how soon a control commitment can be issued. Accelerating the reasoning step is therefore the central problem we address. We observe that CoC reasoning has two different purposes. Most tokens continue routine setup. The next token follows naturally from the ego trajectory history (recent ego dynamics, partially-emitted CoC). A small fraction encode edge cases that require fresh visual evidence, such as a pedestrian half-stepping off a curb, an oncoming car crossing a center line – where the next token cannot be predicted from history alone. In this work we split the reasoning step into two paths specialized for these two needs: a routine reasoner that handles predictable continuation by attending to trajectory history, and a deliberative reasoner that handles the novel cases by attending to current visual evidence. The routine reasoner is fast and the deliberative reasoner is heavier; we use the speculative decoding framework [ 14 , 4 ] as the architectural template for how they cooperate – the routine reasoner proposes the next K tokens, the deliberative reasoner verifies them in one pass, and the longest matching prefix is committed. Critically, our routine reasoner is not a reduced-parameter distillation of the target model. The two reasoners are deliberately specialized to read different parts of the prompt: the routine reasoner attends primarily to trajectory-history positions where the next routine token can be predicted, while the deliberative reasoner is the unmodified VLA target and retains its full visual attention to detect novelty. The role of each is set by what kind of token-prediction it is good at, not by parameter count. The mechanism that produces this specialization on the draft side is a position-encoding change we call FlatRoPE : the natural choice for a multimodal draft – mirroring the target’s 3D M-RoPE. This couples the draft’s attention to the visual token cluster and diverts the attention budget away from the trajectory-history positions where routine signal lives (Section 4.2 ). FlatRoPE replaces 3D M-RoPE with plain 1D rotary, breaking the rotational symmetry of adjacent vision patches and redirecting the attention budget onto trajectory history. We further introduce a reinforcement-learning post-training stage (Section 4.3 ) with a three-component reward and a static-reference KL anchor that fixes a self-removal pathology in the standard policy anchor. Our contributions: 1. FlatRoPE : plain-1D rotary in the draft, motivated by an attention-entropy diagnostic showing that 3D M-RoPE clusters vision tokens and disproportionately reduces attention mass on trajectory-history tokens. FlatRoPE is the mechanism that produces the routine-reasoner-vs-deliberative-reasoner specialization on the draft side. Across two draft architectures and two open VLA targets, FlatRoPE consistently outperforms the 3D-mirroring baseline on accepted length. 2. AARL post-training with a static-reference KL anchor. The standard policy anchor self-removes as the draft drifts; the static reference does not. The result is the first post-training method we are aware of that improves both validation and held-out test accepted length on an open AV VLA. 3. Empirical validation that this specialization fits autonomous driving. We measure per-token attention budget at the draft’s input fusion (showing FlatRoPE redirects attention to trajectory history as designed), provide a token-level routing diagnostic on real CoC traces (showing the routine reasoner handles setup tokens autonomously while the deliberative reasoner is selectively invoked at visually-grounded decision points), and quantify end-to-end latency reduction on val, on-shelf, and off-shelf test splits. In summary, these three contributions produce a two-mode decoder whose routine path handles the majority of CoC tokens without invoking the deliberative path. With L=4.6 at block size 16 the routine path handles \sim 78\% of CoC tokens, giving a theoretical wall-clock speedup of 3.5\times at iso-quality on Alpamayo-R1. 2 Related work Vision-language-action models for driving. End-to-end VLA models for autonomous driving have rapidly converged on a common recipe: a vision-language backbone consumes multi-camera images and ego state, and an action head produces a future trajectory. EMMA [ 11 ] demonstrated end-to-end driving from a Gemini-class backbone; DriveLM [ 23 ] introduced graph-structured visual question answering as a planning interface. DriveGPT4 [ 31 ] showed that a multimodal LLM can be fine-tuned to emit interpretable driving rationales. Figure 1: Reasoning-Aware Speculative Decoding Framework : The target model encodes visual features and trajectory data into tokens, which a compact draft model then uses to generate reasoning proposals. While the target model maintains a broad visual focus, the draft model prioritizes historical trajectory patterns to perform lightweight, routine-based reasoning. In the event of a novel scenario, any disagreement between the models triggers a rejection by the target, initiating a new round of proposals to ensure accuracy. World-model-grounded VLAs such as GAIA-1 [ 8 ] and Wayve’s LINGO/UniAD-style stacks [ 28 , 9 ] couple language reasoning with imagined futures. RoboTron-Drive [ 10 ] and OmniDrive [ 24 ] focus on perception-grounded reasoning, while OpenEMMA [ 30 ] and MiniDrive [ 34 ] aim at compact, deployable variants. Alpamayo-R1 [ 27 ] and AutoVLA [ 35 ] – the two open targets we evaluate against – additionally emit a chain-of-causation (CoC) reasoning step before invoking a continuous flow-matching or diffusion trajectory head, making the autoregressive language portion

Read the complete text