Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models
Arian Mousakhan, Sudhanshu Mittal, Silvio Galesso, Karim Farid, Thomas Brox · 2025
Read Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models by Arian Mousakhan, Sudhanshu Mittal, Silvio Galesso, Karim Farid, Thomas Brox (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13162.txt.
Introduction
Arian Mousakhan and four co-authors built Orbis in twenty twenty-five, a world model that predicts driving video far into the future from a single camera. Existing driving world models degrade over long horizons and stumble in hard scenarios, and many lean on maps, depth, or multiple cameras. Orbis uses none of those extras — just simple design choices.
Despite only about four hundred seventy million parameters and two hundred eighty hours of training video, it reached state-of-the-art prediction quality, standing out in turning maneuvers and urban traffic. The team also ran a controlled comparison: one hybrid tokenizer serving both a discrete-token model and a continuous flow-matching model. The continuous autoregressive model won — less brittle to design choices and more powerful — a useful verdict for world model builders.
Catalog record
| CREATOR | Arian Mousakhan, Sudhanshu Mittal, Silvio Galesso, Karim Farid, Thomas Brox |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-13162
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models Arian Mousakhan, Sudhanshu Mittal, Silvio Galesso, Karim Farid, Thomas Brox Abstract Existing world models for autonomous driving struggle with long-horizon generation and generalization to challenging scenarios. In this work, we develop a model using simple design choices, and without additional supervision or sensors, such as maps, depth, or multiple cameras. We show that our model yields state-of-the-art performance, despite having only 469M parameters and being trained on 280h of video data. It particularly stands out in difficult scenarios like turning maneuvers and urban traffic. We test whether discrete token models possibly have advantages over continuous models based on flow matching. To this end, we set up a hybrid tokenizer that is compatible with both approaches and allows for a side-by-side comparison. Our study concludes in favor of the continuous autoregressive model, which is less brittle on individual design choices and more powerful than the model built on discrete tokens. Code, models and qualitative results are publicly available at https://lmb-freiburg.github.io/orbis.github.io/ . University of Freiburg, Germany 1 Introduction Intelligent agents operate in complex environments by simulating plausible future states based on past observations. This capacity for imagination allows them to plan toward long-term goals bar2024navigation ; ha2018world . Humans naturally acquire this ability through passive observation and minimal interactions, enabling them to adapt quickly to new and unseen scenarios. Emphasizing the passive observation component of such world models has become particularly popular for the driving world, since large amounts of data exists for this domain, and it is attractive to circumvent the manual setup of many perception components by rather learning the visual representation for decision making via the predictive loss of a world model. Recent driving world models agarwal2025cosmos ; gao2024vista ; hassan2024gem built on video diffusion models blattmann2023stable have made major strides towards generating detailed content in high definition and at high frame-rates. However, Fig. 2 reveals that these models only work well for few frames, especially in case of maneuvers that require generating new content, such as turning. This indicates limitations in how these models capture state transitions – the key feature of a world model. We quantified this shortcoming via a metric that compares the trajectory of the real continuation with the trajectories of the generated content. Our evaluation reveals consistent problems with all public driving world models. Consequently, a relevant question is whether world models should rely on continuous-space modeling or predict discrete tokens (similar to LLMs) chang2022maskgit ; esser2021taming ; yu2023magvit . The current trend for visual generation goes towards diffusion-based (continuous) models blattmann2023stable ; polyak2024movie ; girdhar2024factorizing . On the other hand, driving world models based on discrete representations and LLM-like objectives seem to have the edge in terms of rollout duration hu2023gaia ; hu2024drivingworld . Among these, the proprietary GAIA-1 model showed no issues with turns and long drives. This observation prompts the question of whether the discrete paradigm is really superior to the other for long-term generation, and whether the continuous space is the reason for the observed shortcomings in the current state of the art. Figure 1: Comparison of model scale, training data volume, and FVD performance of various approaches on the NuPlan dataset. *DrivingWorld is trained on the test dataset nuPlan. To address these questions, we introduce a hybrid discrete-continuous tokenizer that is compatible with both types of modeling approaches to be able to compare the two strategies on the same ground. For the quantized-token model we developed a frame-wise autoregressive model based on MaskGIT chang2022maskgit , whereas for the continuous-token model we developed an autoregressive model based on flow matching lipman2022flow ; liu2022flow . Both models were trained from scratch. We also put effort into optimizing the details of the tokenizer. Indeed we find that many of these details are important for the performance of the model acting on quantized tokens. Surprisingly, these details are of little relevance for the continuous modeling approach. Both our models can handle long roll-outs, but the continuous approach yields significantly better results and sets the state of the art by a large margin; see Fig. 1 . Unlike many prior approaches, our world model is trained using only raw video data without using any extra low-level regularization objectives, such as structural consistency or pseudo-depth supervision. All implicit perception is learned directly from the presented videos. This makes the approach more scalable and establishes a strong foundation for the development of more controllable models. We also demonstrate that our model can be modified easily to allow ego-motion control via adaptive layer normalization perez2018film . To this end, we evaluate the trajectories produced by our world model also in ego-motion-control-conditioned settings, where we propose a set of metrics to evaluate realism and coverage of the requested trajectories. To summarize, (1) we highlight shortcomings of contemporary driving world models and propose additional benchmarking metrics to make these shortcomings more explicit. (2) We propose a hybrid discrete-continuous tokenizer that is compatible with both discrete and continuous prediction losses and allows us to compare both modeling approaches side-by-side. (3) On its basis, we compare continuous and discrete prediction losses on a fair common ground and find a clear advantage in favor of continuous modeling. (4) As demonstrated in Figure 1 , the resulting model is much more economical in terms of training data and model size than existing world models. Using only 280 hours of front camera video data, our 469M parameter flow-matching model Orbis already produces state-of-the-art performance on long-horizon rollouts with realistic and diverse trajectories. It excels particularly in challenging driving scenarios. Find visualizations embedded in the attached html file. 2 Related Work World Models. The ability of world models to do real-world simulation can be useful for policy learning he2024large ; piergiovanni2019learning , sample efficient RL hafner2019dream ; hafner2020mastering ; li2024think2drive ; wu2023pre , and representation learning zhao2024drivedreamer4d . Previous world models have been limited to gaming ha2018recurrent ; hafner2019dream ; hafner2020mastering and other simulated environments dosovitskiy2017carla . Recent breakthroughs in video generative modeling blattmann2023stable ; yang2024cogvideox have led to future video prediction models - an essential building block for world models. Multiple driving world models wang2024drivedreamer ; Wen_2024_CVPR ; li2024bevformer use BEV (Bird’s-Eye-View) annotations like depth maps, 3D bounding boxes, road maps to generate new scenarios. DriveDreamer wang2024drivedreamer incorporated multi-modal input, such as traffic conditions, text prompts, and driving actions, for future frames and action generation. Many other works Wen_2024_CVPR ; zhao2024drive ; wang2024driving extended this idea to multi-view video generation. Some recent works zhao2024drive ; jia2023adriver also use LLMs and VLMs liu2023visual for spatial reasoning. Although these models show high quality generation, they rely on heavy external knowledge. Such heavy reliance limits the model’s ability to generalize to new environments. In this work, we train a generalizable world model using unannotated front-camera videos and only fine-tune for ego-motion control. Recent driving world models kim2021drivegan ; yang2024genad ; gao2024vista ; agarwal2025cosmos ; hassan2024gem ; hu2023gaia ; russell2025gaia trained predominantly on raw driving video data have shown the ability to simulate realistic future scenes in unseen environments. DriveGAN kim2021drivegan , among the first works to train on real-world driving data, showed realistic future generation with ego-motion and environment controllability. GAIA-1 hu2023gaia further enhanced the quality of