Dreaming Across Towns: Semantic Rollout and Town-Adversarial Regularization for Zero-Shot Held-Out-Town Fixed-Route Driving in CARLA
Feeza Khan Khanzada, Jaerock Kwon · 2026
Read Dreaming Across Towns: Semantic Rollout and Town-Adversarial Regularization for Zero-Shot Held-Out-Town Fixed-Route Driving in CARLA by Feeza Khan Khanzada, Jaerock Kwon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27994.txt.
Introduction
Khanzada and Kwon tackle driving agents that fail in new simulated towns because road shapes and lane layouts change. Their training asks the agent to predict the meaning of future camera views while ignoring clues that reveal the training town, so it learns features that travel. Tested in two unseen CARLA towns after training in two others, the method beat matched world-model baselines in both places.
Catalog record
| CREATOR | Feeza Khan Khanzada, Jaerock Kwon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74822 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27994
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dreaming Across Towns: Semantic Rollout and Town-Adversarial Regularization for Zero-Shot Held-Out-Town Fixed-Route Driving in CARLA Feeza Khan Khanzada, Jaerock Kwon Abstract Learned driving agents often degrade when deployed in unseen environments. This paper studies a deliberately bounded instance of that problem in the CARLA simulator: zero-shot transfer of a closed-loop fixed-route driving agent from Town05 and Town06 to unseen Town03 and Town04. The study isolates structural town shift by keeping weather fixed to ClearNoon and removing traffic and pedestrians. We build on a Dreamer-style latent world-model agent and add two training-only auxiliary losses: multi-horizon prediction of future visual-semantic embeddings along imagined rollouts and town-adversarial supervision on a semantic projection of the recurrent latent state. A causal context feature conditions the semantic rollout predictor, while the actor and critic retain the standard control feature. The policy receives no navigation command, route polyline, goal pose, or map input; the reference route is used only by the environment for reward, progress, success, and termination. Across the evaluated held-out towns, the proposed model achieves the highest mean success rate among the included Dreamer-family methods. Secondary safety and lane-keeping metrics are mixed across towns. These results support a bounded conclusion: in this controlled fixed-weather CARLA setting, semantic rollout supervision combined with town-adversarial regularization improves mean held-out-town route completion. I Introduction Learned driving agents often degrade when deployed in environments that differ from those seen during training. This paper studies a deliberately bounded instance of that problem in the CARLA simulator [ 16 ] i.e zero-shot held-out-town transfer for route-unconditioned fixed-route closed-loop driving. Training uses only Town05 and Town06, and evaluation is performed without target-domain adaptation on unseen Town03 and Town04 under fixed ClearNoon weather and without traffic or pedestrians. The scope is deliberately restricted. Training is performed on source towns only, and evaluation is performed in held-out towns without target-domain adaptation. Weather is held fixed, and the environment contains only the ego vehicle and its attached sensors, without traffic or pedestrians. The goal is therefore not to claim broad robustness to all forms of autonomous-driving distribution shift, but to test whether a learned route-following agent can retain route-relevant competence when town layout and lane topology change. Existing strategies only partially address this problem. Image augmentation can improve invariance to low-level appearance changes, but it does not explicitly regularize imagined future representations. Domain-adversarial learning [ 18 ] can reduce town-predictive information, and semantic supervision can encourage route-relevant structure, but either mechanism alone may be insufficient. This is therefore a route-unconditioned fixed-route setting rather than a command-conditioned or map-conditioned urban driving benchmark. We study this question within a DreamerV3-style [ 22 ] world-model reinforcement-learning framework. We augment the agent with two training-time auxiliary objectives: multi-horizon prediction of frozen OpenCLIP [ 9 ] image embeddings along imagined rollouts, and town-adversarial supervision on a semantic projection of the deterministic RSSM state. A history-conditioned context feature conditions the semantic rollout predictor. In the primary model, the actor and critic retain the standard Dreamer feature, so the semantic/context branches act as auxiliary regularizers rather than replacing the control input. These auxiliary signals are used only during training; deployment in held-out towns is zero-shot and uses no further adaptation. Our claims are deliberately bounded. We do not claim robustness to weather, traffic, pedestrians, sensor corruption, or real-world deployment. Instead, we evaluate whether these training-time auxiliaries improve held-out-town transfer in a fixed-weather CARLA route-following setting using task-completion, safety, lane-keeping, and speed metrics. The contributions of this paper are threefold. • We define a controlled CARLA protocol for zero-shot held-out-town fixed-route driving that isolates structural shift in town layout and lane topology by fixing weather and removing traffic and pedestrians. • We augment a Dreamer-style latent world-model agent with two training-only auxiliaries: multi-horizon prediction of future visual-semantic embeddings and town-adversarial regularization on a semantic latent projection. • Within this benchmark, the auxiliary-only semantic design achieves the highest mean held-out-town success among the included Dreamer-family methods, while direct semantic-policy control is weaker. The comparisons below are descriptive mean ± standard deviation across three independently trained seeds; the paper does not claim formal statistical significance from these runs alone. II Related Work This section reviews prior work most relevant to the present study: learning-based autonomous driving, world-model-based driving, domain-invariant representation learning, structured representations for transfer, vision-language supervision, and CARLA evaluation protocols. II-A Learning-based Autonomous Driving Policies End-to-end driving policies learn a direct mapping from observations to control, often with additional conditioning or structure to mitigate ambiguity. Conditional Imitation Learning (CIL) conditions policies on high-level commands to reduce multimodality in demonstrations and improve controllability across maneuvers [ 11 ] . Learning by Cheating (LbC) leverages privileged information available in simulation to train a privileged “teacher” and distill behavior into a vision-based student, aiming to inherit robustness while deploying only standard sensory inputs [ 7 ] . Related efforts seek to reduce reliance on dense labels by learning compact visual abstractions that support driving control with less annotation burden [ 2 ] . In parallel, RL-based driving has been explored as an alternative to imitation when rewards and interaction are available, including recent end-to-end RL formulations in urban driving settings [ 13 ] . Hybrids that combine imitation with reinforcement learning objectives have also been proposed to exploit demonstration priors while still improving through interaction [ 5 ] . These works primarily improve the policy mapping from perception to control, using supervision, RL, or hybrid objectives. In contrast, our paper studies how latent structure inside a world-model agent affects zero-shot transfer under held-out-town shift. II-B World Models and Latent Planning for Driving World models aim to learn a compact latent dynamics representation that enables planning, imagination, or model-based value learning from high-dimensional observations. Early latent dynamics approaches for planning from pixels established the core idea: learn a probabilistic latent transition model and plan in latent space [ 21 ] . Dreamer-style agents operationalize this by training a world model and optimizing behavior via imagined rollouts, enabling model-based RL without explicit online planning at every step [ 20 ] . DreamerV3 scales this paradigm across diverse domains through architectural and optimization refinements, strengthening the case for general-purpose world-model agents [ 22 ] . In autonomous driving, Think2Drive [ 28 ] formulates CARLA-v2 driving as model-based reinforcement learning with a latent world model used as a neural simulator for planning, illustrating the relevance of latent imagination for driving control. Within autonomous driving, multiple works explore how to instantiate world-model ideas under the constraints of closed-loop control, safety-critical dynamics, and rich visual scenes. Learning to Drive from a World on Rails uses learned predictive structure to support driving-relevant reasoning under constrained dynamics and perception [ 6 ] . MILE jointly learns a world model and a driving policy for urban driving and reports transfer to unseen towns and weather, making it a close driving-specific precedent for world-model-based transfer [ 24 ] . SEM2 likewise argues for filtering world-model features toward driving-relevant semantics through a semantic masked recurrent world model [ 19 ] . More recent driving-focused world-model systems argue for aligning learned dynamics with driving semantics and controllable factors; for example, Raw2Drive formulates end-to-end autonomous driving in a model-based RL framework and emphasizes aligned world models for sample efficiency and performance