CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving
Ruiyang Zhu, Yuehan He, Boyuan Zheng, Zesen Zhao, Ahmad Chalhoub, Qingzhao Zhang, Z. Morley Mao · 2026
Read CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving by Ruiyang Zhu, Yuehan He, Boyuan Zheng, Zesen Zhao, Ahmad Chalhoub, Qingzhao Zhang, Z. Morley Mao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17284.txt.
Introduction
Zhu and colleagues address rare, safety-critical driving scenes that defeat end-to-end vision-language-action models, like construction zones and tricky yields. Their CLAP framework adds per-roadblock soft prompts to a frozen driving model, fetched on demand over vehicle-to-everything links. A two-stage method first finds a hard-scene direction, then tunes prompts along it, cutting planning error on hard scenes 24 percent with no loss on normal ones.
Catalog record
| CREATOR | Ruiyang Zhu, Yuehan He, Boyuan Zheng, Zesen Zhao, Ahmad Chalhoub, Qingzhao Zhang, Z. Morley Mao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47767 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17284
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CLAP: Contrastive Latent-space Prompt Optimization for End-to-end Autonomous Driving Ruiyang Zhu, Yuehan He, Boyuan Zheng, Zesen Zhao, Ahmad Chalhoub, Qingzhao Zhang, Z. Morley Mao Abstract End-to-end autonomous driving systems powered by Vision-Language-Action (VLA) models achieve strong performance on common driving scenarios, yet remain brittle in rare but safety-critical long-tail situations such as active construction zones and complex yielding geometries. In this paper, we present a method that addresses the long-tail challenging scenes beyond data scaling and model training. We introduce CLAP ( C ontrastive La tent-space P rompt optimization), a location-aware adaptation framework that augments a frozen VLA driving model with per-roadblock soft prompts, optimized from crowdsourced data and retrieved on demand via Vehicle-to-Everything (V2X) communication. Our approach rests on two observations from VLAs’ latent space: (i) at the VLA’s hidden-state layer, scenarios from the same roadblock cluster tightly and occupy compact regions of the latent space; and (ii) within a single roadblock, long-tail and normal frames are heavily intermixed in the latent representation, making it difficult to improve one without disturbing the other. CLAP addresses this via a two-stage pipeline: supervised contrastive learning to discover a roadblock-specific hard-scene direction, followed by directionally regularized prompt optimization that selectively improves challenging frames while preserving normal frame performance. On the NAVSIM benchmark with various state-of-the-art VLA backbones, CLAP reduces challenging scenario planning error by 24% with no regression on normal frames, significantly improving planning performance. 1 Introduction The promise of end-to-end autonomous driving (AD) lies in replacing hand-engineered perception-planning stacks [ 2 , 1 ] with unified models that jointly learn perception, prediction, and control from raw sensor streams [ 8 , 13 , 46 , 24 , 45 ] . Recent Vision-Language Models (VLMs), pre-trained on Internet-scale vision-text corpora, bring additional benefits: they encode rich common-sense knowledge about traffic rules and safety margins that are difficult to express in purely geometric representations. Building on this foundation, recent work has extended VLMs into Vision-Language-Action (VLA) planners with large-scale driving datasets [ 22 , 21 , 38 , 23 , 49 ] , in both pre-training and fine-tuning stages, achieving substantial gains on end-to-end planning benchmarks. Despite these advances, VLA-based AD systems share a well-known Achilles heel: long-tail scenario failures . A long-tail scenario is a driving situation that is (i) often rare in the training distribution but (ii) safety-critical in deployment. Canonical examples include active road construction with partial lane closures (Figure 1(a) ), yielding under complex geometries and semantics (Figure 1(b) ), and adverse weather combinations not well-represented in training. These cases cause state-of-the-art VLA planners to produce erratic trajectories or resort to overly conservative behaviors [ 5 , 12 ] . The dominant approach to closing this gap is scaling data collection and fine-tuning the model, an axis along which recent VLA work continues to make substantial progress [ 22 , 38 , 21 , 49 , 43 , 23 ] . These gains, however, become progressively harder to extend along the long-tail: rare events are sparse by construction [ 25 ] , the combinatorial space of within-category variation (layouts, weather, time of day, surrounding traffic) makes exhaustive coverage intractable [ 4 , 7 ] , and frequent fine-tuning on niche data risks catastrophic forgetting of previously learned competencies [ 28 ] . (a) The model drive the vehicle to collide with a temporary workzone. (b) The model fails to detour when blocked by a parked vehicle. Figure 1 : Examples of robustness issues of RecogDrive [ 22 ] tested on NAVSIM [ 10 ] . Red trajectories are produced by the driving model while green trajectories are human driver produced ground truth. Key insight: long-tail failures can be patched locally. Rather than repairing a VLA planner for all possible long-tail scenarios, we observe that many failures are tied to specific road regions, such as recurring construction zones, complex intersections, or frequently blocked lanes. For such cases, the goal does not need to be category-level generalization across all construction zones or all blocked lanes. Instead, it is sufficient to improve behavior across repeated traversals of the same location. This locality is reflected in the model’s latent space. Figure 2 shows a t-SNE projection of three state-of-the-art VLA planners’ (RecogDrive [ 22 ] , Alpamayo [ 38 ] , and DriveVLA-W0 [ 21 ] ) hidden states over four spatial regions of the NAVSIM [ 10 ] road topology: two construction zones and two stop-sign intersections. Across all three models, scenarios from the same region cluster tightly, reflecting shared visual structure (road geometry, landmarks, lane layout, etc. ). On the other hand, different regions occupy distinguishable clusters rather than collapsing into a single category-level region ( e.g., the two stop-sign regions occupy distinct parts of the latent space). This suggests a natural unit of adaptation: we call each such region a roadblock : a contiguous spatial region of the road whose traversals share a stable visual and geometric identity (same lane topology, dominant static obstacles and surrounding scene structure). A roadblock is an operational abstraction and can be instantiated in several ways, including (i) HD-map segments [ 37 ] bounded by change points in lane-topology, (ii) geofenced polygons around persistent obstacles ( e.g., active construction), or (iii) unsupervised clusters of GPS traces with similar visual embeddings [ 3 ] . This reframes the problem: rather than patching the model globally, a manifold too large for fine-tuning to cover, we can patch it per road , where the manifold is small enough for a targeted intervention to cover cleanly. (a) RecogDrive [ 22 ] . (b) Alpamayo 1.5 [ 38 ] . (c) DriveVLA-W0 [ 21 ] . Figure 2 : 2D t-SNE [ 36 ] of hidden states from three state-of-the-art VLA planners on four NAVSIM regions. Across all three models, scenarios from the same region form clusters in the latent space. Figure 3 : Overview of CLAP. Connected vehicles upload challenging traces over V2X; the cloud optimizes a per-roadblock prompt pair (P_{A},P_{B}) ; other vehicles retrieve and apply them to the frozen VLA when entering the same location. Key method: soft prompt optimization through contrastive latent-space learning. We adopt soft-prompt tuning [ 18 , 42 ] as the adaptation mechanism, which optimizes continuous prompt embeddings while keeping the backbone model frozen. This is particularly suitable for per-road adaptation: prompts can be learned, stored, and deployed independently across locations, avoiding interference between patches and eliminating the cost and risk of repeated fine-tuning. Soft-prompt tuning, however, brings its own challenge in this setting: a prompt optimized for challenging frames will also perturb normal frames on the same road, as challenging and normal frames are intermixed in latent space [ 29 ] (§ 3.2 ). To address this, we design CLAP (Figure 3 ), a two-stage prompt optimization framework over a frozen VLA, instantiated per roadblock : Stage 1 — direction discovery via supervised contrastive learning. We optimize an auxiliary prompt P_{A} with a supervised contrastive (SupCon) [ 16 ] objective that pushes hidden states of challenging frames away from those of normal frames. The principal direction of the separated representation yields d^{\star} , a compact descriptor of the challenging-scene direction . Stage 2 — directionally regularized prompt optimization. We then optimize a