Correcting and Quantifying Systematic Errors in 3D Box Annotations for Autonomous Driving

Alexandre Justo Miro, Ludvig af Klinteberg, Bogdan Timus, Aron Asefaw, Ajinkya Khoche, Thomas Gustafsson, Sina Sharif Mansouri, Masoud Daneshtalab, sina.sharif.mansouri}@se.traton.com, {ludvig.af.klinteberg, masoud.daneshtalab}@mdu.se · 2026

Read Correcting and Quantifying Systematic Errors in 3D Box Annotations for Autonomous Driving by Alexandre Justo Miro, Ludvig af Klinteberg, Bogdan Timus, Aron Asefaw, Ajinkya Khoche, Thomas Gustafsson, Sina Sharif Mansouri, Masoud Daneshtalab, sina.sharif.mansouri}@se.traton.com, {ludvig.af.klinteberg, masoud.daneshtalab}@mdu.se (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14038.txt.

Introduction

Alexandre Justo Miro and co-authors fix errors in the 3D boxes used to train driverless cars in 2026. Accurate ground truth annotations are critical to supervised learning for vehicles. Cars use active sensors like LiDAR, which scan the surroundings in set patterns. In dynamic scenes, objects are observed at different timestamps and positions.

That timing spread makes 3D box annotation challenging and error-prone. The team works on correcting and quantifying systematic errors in these boxes. Known error sizes let teams judge how much to trust a dataset. Safer self-driving needs labels that match the real world.

Catalog record

CREATORAlexandre Justo Miro, Ludvig af Klinteberg, Bogdan Timus, Aron Asefaw, Ajinkya Khoche, Thomas Gustafsson, Sina Sharif Mansouri, Masoud Daneshtalab, sina.sharif.mansouri}@se.traton.com, {ludvig.af.klinteberg, masoud.daneshtalab}@mdu.se
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-14038

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Correcting and Quantifying Systematic Errors in 3D Box Annotations for Autonomous Driving Alexandre Justo Miro, Ludvig af Klinteberg, Bogdan Timus, Aron Asefaw, Ajinkya Khoche, Thomas Gustafsson, Sina Sharif Mansouri, Masoud Daneshtalab, sina.sharif.mansouri}@se.traton.com, {ludvig.af.klinteberg, masoud.daneshtalab}@mdu.se Abstract Accurate ground truth annotations are critical to supervised learning and evaluating the performance of autonomous vehicle systems. These vehicles are typically equipped with active sensors, such as LiDAR, which scan the environment in predefined patterns. 3D box annotation based on data from such sensors is challenging in dynamic scenarios, where objects are observed at different timestamps, hence different positions. Without proper handling of this phenomenon, systematic errors are prone to being introduced in the box annotations. Our work is the first to discover such annotation errors in widely used, publicly available datasets. Through our novel offline estimation method, we correct the annotations so that they follow physically feasible trajectories and achieve spatial and temporal consistency with the sensor data. For the first time, we define metrics for this problem; and we evaluate our method on the Argoverse 2, MAN TruckScenes, and our proprietary datasets. Our approach increases the quality of box annotations by more than 17% in these datasets. Furthermore, we quantify the annotation errors in them and find that the original annotations are misplaced by up to 2.5 m 2.5\text{\,}\mathrm{m} , with highly dynamic objects being the most affected. Finally, we test the impact of the errors in benchmarking and find that the impact is larger than the improvements that state-of-the-art methods typically achieve w.r.t. the previous state-of-the-art methods; showing that accurate annotations are essential for correct interpretation of performance. Our code is available at https://github.com/alexandre-justo-miro/annotation-correction-3D-boxes . 1 Introduction Accurate ground truth annotations are a cornerstone in the development, evaluation, and testing of autonomous driving systems. Modern perception pipelines are often powered by deep neural networks trained in a supervised manner and so rely on high-quality labeled data. Annotations are estimations of ground truth, and can be obtained in different ways, e.g . manually by humans, via an automatic labeling method, or a combination. The error of these estimations in autonomous driving datasets is widely assumed to be orders of magnitude below the estimation error of the downstream tasks that are benchmarked on the annotations. However, this assumption is seldom revisited, and the accuracy of annotations and their resemblance to the ground truth have been highly overlooked. For object detection, typical datasets [ 12 , 4 , 20 , 23 , 1 , 11 ] provide bounding box positions at a specific reference timestamp within a 100\text{\,}\mathrm{ms} interval, i.e . annotation sample, which is a typical scanning interval for a rotating Light Detection and Ranging (LiDAR) sensor. Annotation is typically done by fitting a 3D bounding box to the LiDAR data associated with the edges of the object. However, a rotating LiDAR may detect an object any time within the mentioned 100\text{\,}\mathrm{ms} interval. Specifically, during 100\text{\,}\mathrm{ms} , objects can move significantly in highly dynamic scenarios, e.g . 3\text{\,}\mathrm{m} for a typical highway speed of 30\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1} . Hence, in general, the ground truth pose of the object differs between the actual time when the LiDAR detected the object and the common reference time at which all boxes are annotated. This discrepancy may lead to large box annotation errors. These are more obviously visible in multi- LiDAR datasets (see Fig. 1 ), but are present in single- LiDAR datasets as well [ 17 ] . (a) Argoverse 2. (b) MAN TruckScenes. (c) Our proprietary dataset. Figure 1 : Examples of 3D annotation errors in various datasets. The original LiDAR points are colored by capture time offset relative to the reference time at which the boxes were annotated (see color bar at the right of the figure). The translucent, blue points show where these LiDAR detections would have been at the annotation reference time. Our proposed annotations (dashed-line boxes) are aligned with points with zero time offset, while the original annotations (solid-line boxes) are aligned with point clouds with large time offsets. The position offset between them is an indication of the annotation error that we aim to correct. For example, Fig. 1(a) shows a 3D box annotation in the Argoverse 2 [ 23 ] dataset. There, the object is moving towards the right of the image, along its x -axis. However, the box is aligned with the LiDAR detections at 30\text{\,}\mathrm{ms} offset w.r.t. the box annotation timestamp, hence significantly more to the right than where the object really was at the box annotation timestamp. In turn, Fig. 1(b) shows a similar situation in the MAN TruckScenes dataset. The position of the box corresponds to the location where the object was at about -35\text{\,}\mathrm{ms} offset, hence significantly more to the left than where the object really was at the annotation timestamp. Finally, Fig. 1(c) shows an equivalent setting in our proprietary dataset. The box is aligned with LiDAR detections offset -20\text{\,}\mathrm{ms} , hence significantly more to the left than where the object was at the annotation timestamp. In all these cases, the boxes should have been placed at the location where hypothetical LiDAR detections offset with 0\text{\,}\mathrm{ms} w.r.t. the annotation timestamp (the translucent blue points in Fig. 1 ) would have hit the moving objects, i.e . where the dashed-line boxes are drawn. From inspecting these and more annotations in the datasets, we hypothesize that the original annotations were made by arbitrarily choosing one reference set of sensor detections to fit the boxes to, without taking into consideration neither the time at which those points were captured nor the dynamics of the objects. An accurate annotation would either place the boxes where the objects were at the annotation timestamp or should have indicated the exact timestamp when each object’s position was annotated. We focus on the former, as annotating multiple boxes at a common, pre-defined sample timestamp has been the standard in the autonomous driving field. However, the latter, i.e . annotating each individual box at the exact location and timestamp that matches the sensor data, has the potential to be more accurate and lead to better performance in downstream tasks, as the detectors would only need to find a box around existing sensor data, instead of implicitly estimating motion and compensating the sensor data to a predefined timestamp. In the case we focus on, the position of the box shall be extrapolated to the sample annotation timestamp using the object’s dynamics. Since the object dynamics are not known apriori, such mechanism requires estimating several states of the object dynamics, including speed, yaw rate, and acceleration; by using several frames from the sequence. As exemplified above, this estimation may imply a large box displacement, and in general it may come across as if the original detections were not fitting the extrapolated box. The novelty of our work is three-fold. First, we introduce a principled, motion-model-based annotation correction framework that accounts for actors’ motion, enabling physically plausible 3D box placement, a capability absent in current datasets and prior methods. Second, we define, for the first time, quantitative metrics tailored to assess the quality of dynamic object annotations, allowing for systematic diagnosis of annotation errors and benchmarking of correction techniques. Finally, we demonstrate the significant sensitivity of commonly accepted

Read the complete text