Control-ITRA: Controlling the Behavior of a Driving Model
Vasileios Lioutas, Adam Ścibior, Matthew Niedoba, Berend Zwartsenberg, Frank Wood · 2025
Read Control-ITRA: Controlling the Behavior of a Driving Model by Vasileios Lioutas, Adam Ścibior, Matthew Niedoba, Berend Zwartsenberg, Frank Wood (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12408.txt.
Introduction
Vasileios Lioutas and co-authors give researchers a steering wheel for simulated drivers. Their base model, ITRA, already generates believable multi-agent traffic. The new method, Control-ITRA, adds two levers: waypoints that agents aim for, and a target speed that quietly sets how boldly they drive.
The team compared ways of feeding those levers in during training. The controlled agents followed the set routes and speeds, kept their driving clean of infractions, and still looked natural, in both familiar places and places the model had never seen. The authors argue that such control matters as much as realism itself. Safety teams need to replay exact scenarios and stress odd cases. They need to ask what-if questions of self-driving systems before those systems meet real roads.
Catalog record
| CREATOR | Vasileios Lioutas, Adam Ścibior, Matthew Niedoba, Berend Zwartsenberg, Frank Wood |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12408
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Control-ITRA: Controlling the Behavior of a Driving Model Vasileios Lioutas, Adam Ścibior, Matthew Niedoba, Berend Zwartsenberg, Frank Wood Abstract Simulating realistic driving behavior is crucial for developing and testing autonomous systems in complex traffic environments. Equally important is the ability to control the behavior of simulated agents to tailor scenarios to specific research needs and safety considerations. This paper extends the general-purpose multi-agent driving behavior model ITRA (Ścibior et al., 2021), by introducing a method called Control-ITRA to influence agent behavior through waypoint assignment and target speed modulation. By conditioning agents on these two aspects, we provide a mechanism for them to adhere to specific trajectories and indirectly adjust their aggressiveness. We compare different approaches for integrating these conditions during training and demonstrate that our method can generate controllable, infraction-free trajectories while preserving realism in both seen and unseen locations. 1 Introduction The simulation of realistic driving behavior is a cornerstone in the development and validation of autonomous driving systems. As autonomous vehicles (AVs) increasingly integrate into real-world traffic, the necessity for robust, reliable, and diverse simulation environments becomes paramount. These environments enable the testing of AVs in complex, high-stakes scenarios that would be difficult or dangerous to replicate in real-world conditions. Moreover, the ability to simulate realistic multi-agent interactions is critical for ensuring that AVs can navigate and respond appropriately to the unpredictable behavior of human drivers and other road users. One of the key challenges in multi-agent driving simulations is the balance between realism and control. State-of-the-art models ( Ścibior et al., 2021 ; Suo et al., 2021 ; Nayakanti et al., 2023 ; Gulino et al., 2023 ; Seff et al., 2023 ; Wu et al., 2024 ) aim to replicate the nuances of human driving behavior but often lack the flexibility to adapt to specific research needs or safety protocols. The ability to control the behavior of simulated agents is essential for tailoring scenarios to investigate particular driving conditions, test edge cases, or enforce safety standards. However, introducing control mechanisms without sacrificing realism remains a significant challenge in the field. Conceptually, human driving behavior encompasses numerous unobserved variables, ranging from high-level goals such as “going to the grocery store across the roundabout,” to intermediate behavioral traits like aggressiveness, down to low-level controls such as setting acceleration and steering values. An ideal driving simulator would allow conditioning on any of these variables, enabling targeted scenario design. However, achieving such comprehensive control is challenging due to the difficulty of precisely defining different behaviors or measuring the degree to which conditions are met. In this work, we introduce Control-ITRA, a model that enables the control of agent behavior through two primary methods: by specifying waypoints for the agent to follow and by setting a target speed for it to reach. Waypoints provide a natural mechanism for guiding agents along a desired path, while target speeds offer a way to influence the agent’s aggressiveness indirectly. Specifically, we build upon the ITRA framework ( Ścibior et al., 2021 ) , a state-of-the-art model that leverages rasterized overhead birds-eye view representations to perceive its environment. We selected ITRA as our foundation, as birdviews offer an intuitive means of spatially placing waypoints. Additionally, we developed a mechanism to assign target speeds per agent, supporting both conditional and unconditional control execution. We further explore two strategies for selecting conditions during training, demonstrating that our approach, Control-ITRA, enables the model to meet specified conditions while preserving the realism of the driving behavior. Finally, we evaluate our conditional model on unseen, out-of-domain locations using TorchDriveEnv ( Lavington et al., 2024 ) , a reinforcement learning environment with simulated traffic, and show that our method outperforms traditional reinforcement learning baselines in the benchmark validation scenarios from TorchDriveEnv. 2 Related Work Trajectory Prediction: Numerous advanced autonomous vehicle simulators have been proposed in recent years ( Dosovitskiy et al., 2017 ; Santara et al., 2021 ; Zhao et al., 2024 ) , reflecting the community’s growing recognition of simulation as an essential element for achieving Level 5 autonomous driving ( On-Road Automated Driving Committee(2021), ORAD ) . In this paper, we focus on trajectory prediction models that can simulate realistic traffic behavior. The primary task of trajectory prediction models is to predict future trajectories based on observed environmental behavior. Broadly, trajectory models can be classified into physics-based and learning-based models. Physics-based methods leverage physical models to generate trajectories with relatively low computational resources, often using kinematic and dynamic models ( Lin & Ulsoy, 1995 ; Lytrivis et al., 2008 ; Brännström et al., 2010 ) combined with inference techniques like Kalman Filters (KF) ( Ammoun & Nashashibi, 2009 ; Jin et al., 2015 ; Lefkopoulos et al., 2021 ) and Monte Carlo methods ( Althoff & Mergel, 2011 ; Okamoto et al., 2017 ; Wang et al., 2019 ) . These traditional methods are generally suitable only for simple prediction tasks and environments. Recently, deep learning-based methods have gained popularity due to their ability to model complex physical, road-related, and agent-interactive factors, making them adaptable to more realistic environments. Predicting future states is inherently probabilistic, and methods like those in Cui et al. (2019) ; Chai et al. (2020) forecast multiple possible trajectories for each agent. Djuric et al. (2020) employs rasterized ego-centric and ego-rotated birdview representations to depict an agent’s current and past states, using a CNN to predict future trajectories. Similarly, ITRA ( Ścibior et al., 2021 ) uses ego-centric birdview representations to perceive the environment, modeling each agent as a variational recurrent network ( Chung et al., 2015 ) . Tang & Salakhutdinov (2019) applies a discrete latent model with a fixed number of future trajectories per agent, utilizing a different representation with separate modules for map encoding and individual RNN networks for encoding agent states. Casas et al. (2020) leverages spatially-aware graph neural networks to model agent interactions in the latent space. Transformer-based approaches ( Liu et al., 2021 ; Huang et al., 2022 ; Seff et al., 2023 ; Niedoba et al., 2023 ; Wu et al., 2024 ) have also been widely adopted to encode interactions between agent states. Goal-conditioned Models: In the literature, conditioning on waypoints is typically framed as a goal-conditioning task, often addressed through inverse planning. Here, trajectory prediction is divided into first predicting candidate waypoints and then generating trajectories based on these waypoints. PRECOG ( Rhinehart et al., 2019 ) introduces a probabilistic forecasting model conditioned on agent positions. PECNet ( Mangalam et al., 2020 ) generates endpoints for pedestrian trajectory prediction in a two-step process, where the proposed endpoints guide pedestrian trajectory sequences. Graph-TERN ( Bae & Jeon, 2023 ) divides pedestrian future paths into three sections, inferring a goal point for each section using mixture density networks. MUSE-VAE ( Lee et al., 2022 ) uses a conditional VAE model to generate short-term and long-term goal heatmaps, from which the agent trajectory is then conditioned. DenseTNT ( Gu et al., 2021 ) predicts a dense goal probability distribution over the road ahead and uses a goal set prediction model to determine the final trajectory goals. Y-net ( Mangalam et al., 2021 ) generates goal position heatmaps using a convolution-based approach, sampling final endpoints from the resulting goal distribution.