BETTY Dataset: A Multi-modal Dataset for Full-Stack Autonomy
Micah Nye, Ayoub Raji, Andrew Saba, Eidan Erlich, Robert Exley, Aragya Goyal, Alexander Matros, Ritesh Misra, Matthew Sivaprakasam, Marko Bertogna, Deva Ramanan, Sebastian Scherer · 2025
Read BETTY Dataset: A Multi-modal Dataset for Full-Stack Autonomy by Micah Nye, Ayoub Raji, Andrew Saba, Eidan Erlich, Robert Exley, Aragya Goyal, Alexander Matros, Ritesh Misra, Matthew Sivaprakasam, Marko Bertogna, Deva Ramanan, Sebastian Scherer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07266.txt.
Introduction
Micah Nye and co-authors release BETTY, a racing dataset for full-stack self-driving research, in a 2025 paper. Most driving datasets serve perception alone, while autonomy needs every layer tested together. Four years of runs on autonomous race cars fill the archive.
Six venues range from ovals to tight, GPS-denied tracks. The cars hit stability limits, lost tire grip, and crashed at speed, all on record. Every sensor stream, the software stacks own outputs, and ground truth ride along. Teams can train and grade estimation, dynamics, forecasting, and control on the same wild miles.
Catalog record
| CREATOR | Micah Nye, Ayoub Raji, Andrew Saba, Eidan Erlich, Robert Exley, Aragya Goyal, Alexander Matros, Ritesh Misra, Matthew Sivaprakasam, Marko Bertogna, Deva Ramanan, Sebastian Scherer |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07266
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BETTY Dataset: A Multi-modal Dataset for Full-Stack Autonomy Micah Nye, Ayoub Raji, Andrew Saba, Eidan Erlich, Robert Exley, Aragya Goyal, Alexander Matros, Ritesh Misra, Matthew Sivaprakasam, Marko Bertogna, Deva Ramanan, Sebastian Scherer Abstract We present the BETTY dataset, a large-scale, multi-modal dataset collected on several autonomous racing vehicles, targeting supervised and self-supervised state estimation, dynamics modeling, motion forecasting, perception, and more. Existing large-scale datasets, especially autonomous vehicle datasets, focus primarily on supervised perception, planning, and motion forecasting tasks. Our work enables multi-modal, data-driven methods by including all sensor inputs and the outputs from the software stack, along with semantic metadata and ground truth information. The dataset encompasses 4 years of data, currently comprising over 13 hours and 32 TB, collected on autonomous racing vehicle platforms. This data spans 6 diverse racing environments, including high-speed oval courses, for single and multi-agent algorithm evaluation in feature-sparse scenarios, as well as high-speed road courses with high longitudinal and lateral accelerations and tight, GPS-denied environments. It captures highly dynamic states, such as 63 m/s crashes, loss of tire traction, and operation at the limit of stability. By offering a large breadth of cross-modal and dynamic data, the BETTY dataset enables the training and testing of full autonomy stack pipelines, pushing the performance of all algorithms to the limits. The current dataset is available at https://pitt-mit-iac.github.io/betty-dataset/ . I Introduction The most challenging robotics tasks involve the integration and interdependence of various modules, such as perception, state estimation, and control. However, without access to a physical robot for development and testing, these tasks cannot be fully developed or validated using available datasets, which typically only offer perception-related data. Unlike the influential KITTI dataset [ 1 ] , which only has a few sensor modalities, modern datasets include annotations, HD maps, closed-loop planning benchmarks [ 2 , 3 ] , ground truth segmentation & depth [ 4 ] , control signals [ 5 , 6 , 7 ] , and more. Though these datasets enable modern learning-based methods, they alone are still insufficient for addressing the full scope of robotics challenges. Data Category Modality KITTI [ 1 ] NuScenes [ 8 ] NuPlan [ 2 ] ArgoVerse 2 [ 9 ] Waymo Open [ 10 ] TartanDrive 2.0 [ 5 ] REVS [ 6 ] RACECAR [ 11 ] Ours Perception Camera ✔ ✔ ✔ ✔ ✔ ✔ ✗ ✔ ✔ LiDAR ✔ ✔ ✔ ✔ ✔ ✔ ✗ ✔ ✔ Radar ✗ ✔ ✗ ✗ ✗ ✗ ✗ ✔ ✔ State Estimation IMU ✔ ✔ ✔ ✗ ✗ ✔ ✔ ✔ ✔ GNSS ✔ ✔† ✔† ✗ ✗ ✔† ✔ ✔† ✔† Odometry ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Labels & Metadata 2D Map ✔ ✔ ✔ ✗ ✔ ✗ ✗ ✗ ✔ 3D LiDAR Map ✗ ✔ ✔ ✔ ✔ ✔ ✗ ✗ ✔ Metadata ✗ ✗ ✗ ✗ ✗ ✔ ✗ ✗ ✔ 2D Camera Labels ✔ ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✔ 3D Lidar Labels ✔ ✔ ✔ ✔ ✔ ✗ ✗ ✗ ✔ Non-ego Odom ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✔ ✔ Trajectory ✗ ✗ ✔ ✗ ✗ ✔ ✔ ✗ ✔ Actuation Throttle ✗ ✗ ✗ ✗ ✗ ✔ ✔ ✗ ✔ Brake ✗ ✗ ✗ ✗ ✗ ✗ ✔ ✗ ✔ Steering ✗ ✗ ✗ ✗ ✗ ✔ ✔ ✗ ✔ Gear ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✔ Tire State Engine RPM ✗ ✗ ✗ ✗ ✗ ✗ ✔ ✗ ✔ Wheel Torque ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✔ Wheel Speed ✗ ✗ ✗ ✗ ✗ ✔ ✗ ✗ ✔ Suspension ✗ ✗ ✗ ✗ ✗ ✔ ✔ ✗ ✔ Tire Temp ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✔ Tire Pressure ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✔ Sideslip angle ✗ ✗ ✗ ✗ ✗ ✗ ✔ ✗ ✔ TABLE I: Comparison of representative population for autonomous driving datasets across urban driving, offroad driving, and racing domains, with full-stack sensors considered. (†) Denotes RTK GNSS. Furthermore, existing datasets typically target discrete sets of problems. For example, Argoverse 2 [ 9 ] targets supervised autonomous vehicle perception, self-supervised LiDAR perception tasks, and motion forecasting. All of these problems focus on building a better understanding of the world, detecting & tracking the agents in it, and forecasting how the world will evolve. However, this fails to address the challenges of understanding the interaction between the environment and the ego vehicle, particularly in terms of dynamics and state estimation. This issue is even more critical in the car racing domain—both in traditional motorsports and autonomous racing—where vehicles are costly, and data is typically kept confidential, making the field available to limited and privileged research communities. In [ 12 ] , the authors provided a large scale dataset of human- and autonomously-driven racecars in simulation, but is limited to vehicle dynamics and control information, while in the RACECAR dataset [ 11 ] , only the perception data of an autonomous racecar is provided. The BETTY dataset covers these problems, and more, by including the inputs and outputs of the software stack running on an Autonomous Racing Vehicle (ARV). By having the sensor inputs and autonomy stack outputs, researchers can evaluate new offline algorithms with a broad range of input sensory data and supervisory signals. To illustrate, [ 13 ] presents a method that uses LiDAR inertial odometry to estimate the amount of slip experienced by the tires for an ARV, which is used downstream in the dynamics model of a model-based controller. Many evaluations and experiments are not possible with any other dataset due to the lack of cross-functional modalities. For example, with the BETTY dataset, researchers can evaluate how the performance of the localization impacts the slip estimation, which in turn impacts the dynamics model and controller outputs. Additionally, ground truth trajectories, slip sensor readings, and commanded & actual actuator control signals are included, providing additional metrics to evaluate against. To summarize, the main contributions of this work are: 1. A dataset for developing and evaluating tasks including perception, state estimation, dynamics modeling, motion forecasting, and more using supervised or self-supervised methods. 2. A dataset with raw sensor data, processed inputs and outputs, metadata, and annotations, that enables evaluating modules independently and chained together in an end-to-end manner. 3. The most diverse autonomous racing dataset with the largest environment diversity, highest accelerations, and most sensor modalities, by far. II Related Work There is a wealth of robotics datasets for many operational domains and environments, including for autonomous vehicles, off-road autonomy, SLAM, vehicle dynamics, and GNSS datasets. The KITTI dataset [ 1 ] inspired what all modern AV and mobile robotics datasets are today by providing the first large-scale dataset for perception tasks and a suite of evaluation benchmarks. II- 1 Autonomous Vehicle Perception Datasets Works such as Argoverse 2, NuScenes, Waymo Open, and ONCE all provide extensive datasets for perception tasks including 3D object detection and tracking and motion forecasting in structured, on-road environments with prescribed traffic rules [ 14 , 9 , 8 , 15 , 10 , 16 ] . Many more datasets follow suit at a smaller scale to address similar problems or