Action Emergence from Streaming Intent

Pengfei Jing, Victor Shea-Jay Huang, Hengtong Lu, [0.3em] Jifeng Dai, Yan Xie, Benjin Zhu, [0.3em] Li Auto, [0.1em] Equal contribution · 2026

Read Action Emergence from Streaming Intent by Pengfei Jing, Victor Shea-Jay Huang, Hengtong Lu, [0.3em] Jifeng Dai, Yan Xie, Benjin Zhu, [0.3em] Li Auto, [0.1em] Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12622.txt.

Introduction

Jing and five colleagues define action emergence for end-to-end self-driving. The goal is feasible, sensible and safe actions in any long-tail scenes from scene-based thought, not from fetching learned pairs. Step-by-step decoders collapse many modes into one mean output. Diffusion and flow makers express many modes but ignore thought-out intent. Their Streaming Intent tool streams intent in meaning through a steady chain of thought derived from scene reading. Over time across clips, promises stay steady along the driving horizon.

Catalog record

CREATORPengfei Jing, Victor Shea-Jay Huang, Hengtong Lu, [0.3em] Jifeng Dai, Yan Xie, Benjin Zhu, [0.3em] Li Auto, [0.1em] Equal contribution
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52011
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.12622

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Action Emergence from Streaming Intent Pengfei Jing, Victor Shea-Jay Huang, Hengtong Lu, [0.3em] Jifeng Dai, Yan Xie, Benjin Zhu, [0.3em] Li Auto, [0.1em] Equal contribution Abstract We formalize action emergence as a target capability for end-to-end autonomous driving: the ability to generate physically feasible, semantically appropriate, and safety-compliant actions in arbitrary, long-tail traffic scenes through scene-conditioned reasoning rather than retrieval or interpolation of learned scene-action mappings. We show that previous paradigms cannot deliver action emergence: autoregressive trajectory decoders collapse the inherently multimodal future into a single averaged output, while diffusion and flow-matching generators express multimodality but are not steerable by reasoned intent. We propose Streaming Intent as a concrete way to approach action emergence: a mechanism that makes driving intent (i) semantically streamed through a continuous chain-of-thought that causally derives the intent from scene understanding, and (ii) temporally streamed across clips so that intent commitments remain coherent along the driving horizon. We realize Streaming Intent in a VLA model we call SI ( Streaming Intent ). SI autoregressively decodes a four-step chain-of-thought and emits an intent token; the decoded intent then drives classifier-free guidance (CFG) on a flow-matching action head, requiring only two denoising steps to generate the final trajectory. On the Waymo End-to-End benchmark, SI achieves competitive aggregate performance, with an RFS score of 7.96 on the validation set and 7.74 on the test set. Beyond aggregate metrics, the model demonstrates – to our knowledge for the first time in a fully end-to-end VLA – intent-faithful controllability : for a fixed scene, varying the intent class at inference yields qualitatively distinct yet consistently high-quality plans, arising purely from data-driven learning without any pre-built trajectory bank or hand-coded post-hoc selector. 1 Introduction Despite rapid progress on aggregate planning benchmarks, end-to-end autonomous driving systems remain brittle on the long tail: rare junction geometries, unprotected turns, dense merging, and ambiguous yielding scenarios continue to drive the bulk of human-takeover events in deployed fleets. We argue that a core missing capability is action emergence : the ability to produce physically feasible, semantically appropriate, and safety-compliant actions in arbitrary scenes through on-the-fly reasoning over perceptual and contextual inputs, rather than through retrieval or interpolation of previously learned scene-action mappings. Central to this capability is driving intent : a discrete high-level commitment -- yield, merge, turn, cruise -- that mediates between scene understanding and trajectory generation. Without an explicit intent representation, an agent has no structured basis for committing to one future among equally plausible alternatives; the long-tail failure modes of current systems are, in large part, failures of intent. 1 1 1 Unlike scale-driven LLM emergence ( 23 ) , our action emergence is a scale-independent, application-level behavior realizable by any scene-conditioned reasoning architecture. Prior trajectory generators cannot deliver action emergence. Prior end-to-end trajectory generators fall into two families that each fail to provide the intent commitment required for action emergence ( Figure 1 ). (i) Autoregressive (AR) trajectory models ( 30 ; 18 ; 14 ; 2 ) decode future waypoints token by token, which tends to collapse the inherently multimodal future into a single averaged trajectory. At an ambiguous junction, this can produce a physically unrealizable compromise between “turn left”, “continue straight”, and “yield”. (ii) Diffusion and flow-matching (FM) trajectory models ( 7 ; 12 ; 11 ; 29 ; 25 ; 27 ; 10 ) can represent multimodal trajectory distributions, but without an explicit conditioning signal indicating which mode to commit to, sampling is dominated by the data prior and remains weakly steerable by reasoned intent. Neither family delivers intent-faithful controllability – the property that a specific, explicitly reasoned intent determines the output trajectory – which we identify as a necessary condition for action emergence: without a mechanism that binds reasoned intent to executed trajectory, an agent cannot commit to a plan in scenes where multiple intents are plausible. Figure 1: Trajectory diversity under ambiguous intent. Given the same intersection scene, AR models collapse to a single averaged future, diffusion/FM models sample a narrow prior-dominated trajectory bundle, whereas SI produces intent-faithful trajectories. Our approach: SI and Streaming Intent. We propose SI ( Streaming Intent ), a VLA model built around the concept of Streaming Intent – a mechanism that approaches action emergence by making intent both semantically grounded in scene reasoning and temporally coherent across the driving horizon. SI comprises three tightly integrated components. (a) Single-backbone language–action alignment. SI operates on a single shared transformer backbone ( 20 ) that serves both the autoregressive (AR) language decoder and the flow-matching (FM) action head ( 12 ; 13 ) . The AR branch decodes the chain-of-thought and emits an intent token; the decoded intent class then drives classifier-free guidance (CFG) ( 8 ) on the FM head. Because both objectives train the same representation end-to-end, language reasoning and trajectory denoising are structurally coupled: language–action alignment is achieved by construction rather than by an auxiliary loss or a post-hoc bridging module. (b) Intent-driven CFG for intent-faithful trajectory generation. The decoded intent directly conditions the FM denoising process via CFG ( 8 ) , steering the generated trajectory toward the committed maneuver rather than defaulting to the statistically dominant mode. At inference, supplying different intent classes to the same trained model yields geometrically and behaviorally distinct trajectories for the same scene – constituting the intent-faithful controllability that prior models lack and that we identify as a necessary condition for action emergence. (c) Streaming Intent: semantic and temporal continuity. Streaming Intent makes intent continuous along two dimensions. Semantic streaming : intent is not predicted as an isolated label, but emerges from a four-step chain-of-thought ( 24 ) ( Perceive \rightarrow Predict \rightarrow Judge \rightarrow Plan ) before being emitted as the intent token; dense CoT annotation makes intent a scene-grounded intermediate representation rather than an independent classifier output. Temporal streaming : the current clip’s intent token and LLM hidden state are compressed into a compact memory token and carried to the next clip, so each intent prediction is conditioned on accumulated episode history without recomputing the full backbone. Together, these two forms of streaming make intent a dynamically evolving, causally grounded, and temporally coherent commitment that bridges VLA reasoning and FM action generation toward action emergence. Results. On the Waymo End-to-End benchmark ( 26 ) , SI achieves an RFS of \mathbf{7.96} on the validation split, and an RFS of \mathbf{7.74} on the test split ( subsection 3.2 , Table 2 , Table 2 ). Beyond aggregate numbers, SI demonstrates two capabilities that, to our knowledge, no prior end-to-end VLA has shown from a single trained model: (i) action emergence on long-tail scenes, where SI’s intent-conditioned trajectories span the plausible action repertoire while strong single-mode baselines collapse onto the dominant forward-cruising mode ( subsection 3.3 , Figure 3 ); and (ii) intent-faithful controllability , where varying the intent for a fixed scene at inference yields geometrically distinct yet uniformly high-quality plans that align with the human-rated RFS alternatives rather than exhibit random variance ( subsection 3.4 , Figure 4 ). Critically, this controllable diversity arises from a single end-to-end-trained model on structurally aligned language–action representations – it is not stitched from a pre-built trajectory bank, nor selected by a hand-tuned post-hoc preference module, as in

Read the complete text