Reproducibility in the Control of Autonomous Mobility-on-Demand Systems

Xinling Li, Meshal Alharbi, Daniele Gammelli, James Harrison, Filipe Rodrigues, Maximilian Schiffer, Marco Pavone, Emilio Frazzoli, Jinhua Zhao, Gioele Zardini · 2025

Read Reproducibility in the Control of Autonomous Mobility-on-Demand Systems by Xinling Li, Meshal Alharbi, Daniele Gammelli, James Harrison, Filipe Rodrigues, Maximilian Schiffer, Marco Pavone, Emilio Frazzoli, Jinhua Zhao, Gioele Zardini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07345.txt.

Introduction

Xinling Li and co-authors study why results in self-driving fleet control research often fail to reproduce, in work from twenty twenty-five. Fleets of robot taxis promise fast, personal city travel at scale. Central control can tune fleet moves for better service overall. The field grows fast, but shared reporting habits lag behind. Hidden assumptions in models, setups, and code block repeat runs.

The study walks the full research pipeline step by step. Modelling, control tasks, simulation design, and scoring each harbour gaps. Common habits in past papers are surveyed for weak spots. A structured frame plus a checklist are proposed for future work. Clearer reporting should make results replicable and comparable across labs. The lessons extend to wider networked autonomy fields as well. Transparent habits are framed as core infrastructure for the field.

Catalog record

CREATORXinling Li, Meshal Alharbi, Daniele Gammelli, James Harrison, Filipe Rodrigues, Maximilian Schiffer, Marco Pavone, Emilio Frazzoli, Jinhua Zhao, Gioele Zardini
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07345

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reproducibility in the Control of Autonomous Mobility-on-Demand Systems Xinling Li, Meshal Alharbi, Daniele Gammelli, James Harrison, Filipe Rodrigues, Maximilian Schiffer, Marco Pavone, Emilio Frazzoli, Jinhua Zhao, Gioele Zardini Abstract Autonomous Mobility-on-Demand (AMoD)systems, powered by advances in robotics, control, and Machine Learning (ML), offer a promising paradigm for future urban transportation. AMoDoffers fast and personalized travel services by leveraging centralized control of autonomous vehicle fleets to optimize operations and enhance service performance. However, the rapid growth of this field has outpaced the development of standardized practices for evaluating and reporting results, leading to significant challenges in reproducibility. As AMoDcontrol algorithms become increasingly complex and data-driven, a lack of transparency in modeling assumptions, experimental setups, and algorithmic implementation hinders scientific progress and undermines confidence in the results. This paper presents a systematic study of reproducibility in AMoDresearch. We identify key components across the research pipeline, spanning system modeling, control problems, simulation design, algorithm specification, and evaluation, and analyze common sources of irreproducibility. We survey prevalent practices in the literature, highlight gaps, and propose a structured framework to assess and improve reproducibility. Specifically, concrete guidelines are offered, along with a “reproducibility checklist”, to support future work in achieving replicable, comparable, and extensible results. While focused on AMoD, the principles and practices we advocate generalize to a broader class of cyber-physical systems that rely on networked autonomy and data-driven control. This work aims to lay the foundation for a more transparent and reproducible research culture in the design and deployment of intelligent mobility systems. I Introduction The rapid urbanization over the past decades has led to a significant rise in private automobile use. While private vehicles offer convenient, door-to-door service at any time, their proliferation has resulted in widespread traffic congestion, increased emissions, and a substantial loss of productive hours [ schrank20112011 ] . In response, Mobility-on-Demand (MoD)services have emerged as a compelling alternative. By dynamically allocating vehicles based on real-time demand, MoDsystems provide personalized travel options while reducing dependence on private car ownership. As such, they retain many of the advantages of private vehicles, offering a more sustainable and scalable mobility solution. With advances in autonomous driving technologies, AMoDhas gained traction as a key enabler of next-generation transportation systems [ Pavone2015 ] . AMoDsystems (i.e., fleets of robo-taxis performing on-demand mobility tasks) autonomously manage vehicle fleets to respond to passenger requests and satisfy transportation demand effectively. Such systems are inherently complex: they involve high degrees of interaction and require coordination across spatial and temporal scales [ zardini2022analysis ] . As a result, different studies often focus on distinct operational challenges, each adopting specific assumptions regarding system dynamics and passenger behavior. Such methodological diversity complicates efforts to ensure research reproducibility and benchmark comparability. Nevertheless, reproducibility remains a cornerstone of rigorous scientific practice. It increases credibility, accelerates progress, and enables meaningful comparisons across research contributions [ munafo2017manifesto ] . In disciplines such as computational science, robotics, and management science, reproducibility is widely promoted via standardized evaluation procedures, common benchmarks, and the routine of sharing code and data [ peng2011reproducible , leitner2017acrv , baines2024need , fivsar2024reproducibility ] . By contrast, while the importance of reproducibility is gaining recognition in transportation research [ wu2024reproducibility , riehl2025revisiting ] , consistent practices and community standards are still scarce. This paper aims to address this gap by identifying and analyzing the key components that determine the reproducibility of AMoDcontrol research. We propose that these components, ranging from modeling choices to experimental setup, be explicitly documented in research outputs. While the focus is on AMoD, the principles discussed are applicable to a broad range of networked control and optimization problems, including those in aviation, maritime transport, and other complex cyber-physical systems. The term reproducibility has been used to describe varying degrees of rigor in how scientific findings are communicated and validated [ goodman2016does , national2019reproducibility ] . At its most basic level, reproducible research requires sufficient transparency and methodological clarity to enable others to recreate the results and verify the underlying logic. This minimal standard typically assumes access to the same input data and requires that the described methods and conditions are detailed enough to reproduce the original outcomes. A more rigorous standard is achieved when independent researchers, using comparable techniques and assumptions and not necessarily the same data, can arrive at similar findings and inferences. This higher level, which is sometimes referred to as inferential reproducibility or replicability, is critical not only for confirming the robustness of scientific claims but also for enabling future research to build upon and extend the original work. In this article, we strive to provide recommendations that strengthen both levels of reproducibility of AMoDresearch, though we do not draw a strict distinction between the terms. I-A Holistic, System-level Reproducibility for AMoD AMoDsystems operate fleets of Autonomous Vehicles (AVs)to fulfill passenger requests within a transportation network. Users request trips via mobile applications, and the system must coordinate vehicles in real time to deliver a high-quality service. The AMoDcontrol problem, therefore, centers on optimizing fleet management, with performance metrics including profitability, service rate, passenger waiting time, and environmental impact. Fig. 1 : The reproducibility scheme. The graph includes key components to improve AMoDcontrol research reproducibility. Fig. 1 presents a scheme of the system-level reproducibility of AMoDsystems, highlighting the essential components needed to improve reproducibility in this context. Specifically, it includes not only the modeling and algorithmic aspects of specific research efforts, but also key experimental and contextual details, such as assumptions, constraints, and data sources, often under-reported. Together, these components provide a comprehensive blueprint for communicating research transparently and facilitating reproducibility. To define an AMoDcontrol problem, three key elements must be specified: (1) the system model, (2) the target operation, and (3) the control objective. The system model provides a tractable mathematical representation of the real-world AMoDnetwork and serves as the foundation for problem formulation. Based on such a model, researchers define the operation of interest, typically some aspect of fleet control constrained by infrastructure properties (e.g., road capacity), and select an appropriate objective function, which often reflects operational efficiency, profitability, or sustainability. Crucially, any proposed solution is meaningful only in the precise context of its assumptions. These include both explicit modeling assumptions (e.g., demand elasticity, congestion models, penetration rates of new technologies), and implicit design choices (e.g., simulation fidelity). These assumptions directly affect the applicability, generalizability, and replicability of the results. Once a solution is developed, it should be evaluated through experiments that benchmark performance, demonstrate robustness, and assess computational tractability. Clear reporting of all steps in this pipeline is essential to support reproducible research. While prior surveys on AMoDcontrol have focused on specific operational problems or solution approaches [ 9924240 , zardini2022analysis ] , this is, to the best of our knowledge, the first perspective centered on reproducibility. In the remainder of this work, we elaborate on the components reported in Fig. 1 , discussing current practices, identifying common pitfalls, and proposing guidelines for improvement. I-B Organization of the Manuscript The remainder of this paper is organized as follows. Section II presents modeling approaches for AMoDsystems. Section III discusses key operational challenges. reviews algorithmic techniques used to address these challenges. outlines foundational assumptions about system dynamics and their implications for research applicability. Furthermore, details best

Read the complete text