Beyond In-Distribution Performance: A Cross-Dataset Study of Trajectory Prediction Robustness

Yue Yao, Daniel Goehring, Joerg Reichardt · 2025

Read Beyond In-Distribution Performance: A Cross-Dataset Study of Trajectory Prediction Robustness by Yue Yao, Daniel Goehring, Joerg Reichardt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15842.txt.

Introduction

Models that predict where cars and people will move next often shine on their home data, then stumble elsewhere. Yue Yao and co-authors test three top trajectory models across two big driving data sets. Each model trains on one set and is graded on the other, in both directions.

The smallest model, built with the strongest built-in assumptions, traveled best between data sets. That held when training on the smaller set and testing on the larger. Training on the larger set and testing down, though, every model struggled. The team weighs in during 2025 on how design choices, data size, and augmentation shape honest tests of these predictors.

Catalog record

CREATORYue Yao, Daniel Goehring, Joerg Reichardt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-15842

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond In-Distribution Performance: A Cross-Dataset Study of Trajectory Prediction Robustness Yue Yao, Daniel Goehring, Joerg Reichardt Abstract Zusammenfassung We study the Out-of-Distribution (OoD) generalization ability of three SotA trajectory prediction models with comparable In-Distribution (ID) performance but different model designs. We investigate the influence of inductive bias, size of training data and data augmentation strategy by training the models on Argoverse 2 (A2) and testing on Waymo Open Motion (WO) and vice versa. We find that the smallest model with highest inductive bias exhibits the best OoD generalization across different augmentation strategies when trained on the smaller A2 dataset and tested on the large WO dataset. In the converse setting, training all models on the larger WO dataset and testing on the smaller A2 dataset, we find that all models generalize poorly, even though the model with the highest inductive bias still exhibits the best generalization ability. We discuss possible reasons for this surprising finding and draw conclusions about the design and test of trajectory prediction models and benchmarks. Keywords: trajectory prediction, out-of-distribution test, evaluation metrics 1 Introduction The robustness of trajectory prediction is essential for practical applications in autonomous driving. The advancement of trajectory prediction models is catalyzed through public motion datasets and associated competitions, such as Argoverse 2 (A2) [ 1 ] , and Waymo Open Motion (WO) [ 2 ] . These competitions establish standardized metrics and test protocols and score predictions on test data that is withheld from all competitors and hosted on protected evaluation servers only. This is intended to objectively compare the generalization ability of models to unseen data. However, these withheld test examples still share similarities with the training samples, such as sensor setup, map representation, post-processing, geographic, and scenario selection biases employed during dataset creation. Consequently, the test scores reported in each competition are examples of In-Distribution (ID) testing. To effectively evaluate model generalization, it is essential to test models on truly Out-of-Distribution (OoD) test samples, such as those from different motion datasets. We investigate model generalization across two large-scale motion datasets [ 3 ] : Argoverse 2 (A2) and Waymo Open Motion (WO). The WO dataset, with 576k scenarios, is more than twice the size of A2, which contains 250k scenarios. As model generalization ability is best illustrated by limiting the amount of training data [ 4 ] , we first train on the smaller dataset A2 and test on OoD samples from the larger dataset WO [ 3 ] . Our results demonstrate that model generalization improves through two key factors: introducing inductive bias via polynomial representation (cf. Section 2.1 ) and implementing proper data augmentation (cf. Section 2.2 ). This naturally raises the question: how does model generalization change when the experimental setup is reversed, i.e., when models are trained on the larger WO dataset and tested on OoD samples from the smaller A2 dataset? Given the increased amount of training data, one would hypothesize that all models will exhibit improved generalization in OoD tests. In this study, we investigate model generalization under this reversed experimental setting to assess whether the hypothesized improvements in robustness are realized. Our paper is organized as follows: We first detail the two design choices influencing model generalization – data representation and data augmentation – identified in our prior work [ 3 ] . From the body of related work, we introduce the two benchmark models along with our proposed model and detail their model designs. We then revisit our dataset homogenization protocol for OoD testing and outline the new experimental settings for training on the WO dataset. Subsequently, we present ID and OoD testing results for all models and analyze the new OoD results with respect to data representation, augmentation strategies and compare them to our expectations. Finally, we conclude our work by arguing for establishing OoD testing as an equally important performance metric alongside traditional ID evaluation. 2 Related Work and Benchmark Models 2.1 Data Representation The structure and rules of individual datasets and accompanying competitions heavily influence the design choices of trajectory prediction models in the literature. Deep learning-based models often directly ingest data in the format the dataset is stored in. Recent studies employ sequence-based representation , e.g., sequences of data points, as the model’s input and output [ 5 , 6 ] . This approach effectively captures diverse information, including agent trajectories and road geometries, while aligning with dataset measurement formats. However, it has notable drawbacks: the representation introduces high redundancy, and its computational cost scales with the length and temporal/spatial resolution of the trajectories and road geometries. Furthermore, the high flexibility of this approach can capture measurement noise and outliers, potentially leading to physically infeasible predictions. Some previous works explored the possibility of using polynomial representations for predictions [ 7 , 8 ] . Su et al. [ 8 ] highlight the temporal continuity of this representation, i.e., the ability to provide arbitrary temporal resolution. Reichardt [ 9 ] argues for using polynomial representations to integrate trajectory tracking and prediction into a filtering problem. Polynomial representations restrict the kind of trajectories that can be represented and introduce bias into prediction systems. This limited flexibility is generally associated with greater computational efficiency, smaller model capacity, and hence better generalization. In our previous study, we proposed our model EP with polynomial inputs and outputs as key innovations [ 3 ] . Our EP model demonstrates improved robustness against OoD samples compared to sequence-based SotA models. 2.2 Data Augmentation In competition settings, one or more agents in a scenario are typically designated as focal agents , and predictions are scored exclusively for these agents. However, training the model exclusively with the focal agent’s behavior fails to exploit all available data. To address this, predicting the future motion of non-focal agents is a typical augmentation strategy for training. As there are many more non-focal agents than focal agents, another important design decision involves determining how to balance the contributions of focal and non-focal agent data. We select two open-sourced and thoroughly documented SotA models: Forecast-MAE [ 5 ] (FMAE) and QCNet [ 6 ] , with nearly 1900k and 7600k parameters, respectively. As summarized in Table 1 , both models employ sequence-based representation but employ different strategies in dealing with non-focal agents: Heterogeneous Augmentation : FMAE follows the prediction competition protocol and prioritizes focal agent prediction. Thus, agent history and map information are computed within the focal agent’s coordinate frame. Compared to the multi-modal prediction of the focal agent, FMAE only outputs uni-modal prediction for non-focal agents. The error of focal agent predictions is weighed higher than for non-focal agents in loss function. Homogeneous Augmentation : QCNet does not focus on the selected focal agent and proposes a more generalized approach. It encodes the information of agents and map elements in each agent’s individual coordinate frame. It outputs multi-modal predictions for focal and non-focal agents alike. This approach ensures a consistent prediction task for all agents and enhanced model generalization. The prediction error of focal and non-focal agent predictions is weighed equally in the loss functions. Figure 1 illustrates the augmentation strategies of our benchmark models FMAE and QCNet. Abbildung 1: The two augmentation strategies for non-focal agent data employed in benchmark

Read the complete text