Realistic Adversarial Attacks for Robustness Evaluation of Trajectory Prediction Models via Future State Perturbation

Julian F. Schumann, Jeroen Hagenus, Frederik Baymler Mathiesen, Arkady Zgonnikov · 2025

Read Realistic Adversarial Attacks for Robustness Evaluation of Trajectory Prediction Models via Future State Perturbation by Julian F. Schumann, Jeroen Hagenus, Frederik Baymler Mathiesen, Arkady Zgonnikov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06134.txt.

Introduction

Julian Schumann and three co-authors attack self-driving trajectory predictors more realistically by tampering with the future, not just the past. Safety evaluations usually perturb where neighboring road users have been, which can create impossible scenarios and miss real weaknesses. Overly optimistic robustness scores are the danger.

Their attacks perturb future states too, while respecting dynamic constraints and preserving believable tactical behavior. New metrics score both the realism of an attack and its damage. Against a state-of-the-art prediction model, the attacks raised prediction errors and collision rates substantially. Qualitative inspection showed failures that past-only attacks never exposed. The authors argue robustness claims should be earned against this harder standard.

Catalog record

CREATORJulian F. Schumann, Jeroen Hagenus, Frederik Baymler Mathiesen, Arkady Zgonnikov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06134

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Realistic Adversarial Attacks for Robustness Evaluation of Trajectory Prediction Models via Future State Perturbation Julian F. Schumann, Jeroen Hagenus, Frederik Baymler Mathiesen, Arkady Zgonnikov Abstract . Trajectory prediction is a key element of autonomous vehicle systems, enabling them to anticipate and react to the movements of other road users. Evaluating the robustness of prediction models against adversarial attacks is essential to ensure their reliability in real-world traffic. However, current approaches tend to focus on perturbing the past positions of surrounding agents, which can generate unrealistic scenarios and overlook critical vulnerabilities. This limitation may result in overly optimistic assessments of model performance in real-world conditions. In this work, we demonstrate that perturbing not just past but also future states of adversarial agents can uncover previously undetected weaknesses and thereby provide a more rigorous evaluation of model robustness. Our novel approach incorporates dynamic constraints and preserves tactical behaviors, enabling more effective and realistic adversarial attacks. We introduce new performance measures to assess the realism and impact of these adversarial trajectories. Testing our method on a state-of-the-art prediction model revealed significant increases in prediction errors and collision rates under adversarial conditions. Qualitative analysis further showed that our attacks can expose critical weaknesses, such as the inability of the model to detect potential collisions in what appear to be safe predictions. These results underscore the need for more comprehensive adversarial testing to better evaluate and improve the reliability of trajectory prediction models for autonomous vehicles. 1. Introduction Figure 1. Scenarios of adversarial attacks by the target agent on trajectory prediction model of the ego agent (AV). A) In the baseline, non-adversarial scenario, based on the observed past trajectory X=\{X_{\text{ego}},X_{\text{tar}}\} of the target agent, the ego agent makes a prediction \widehat{Y} that aims to closely capture the (unknown) future states Y . B) Adversarial (target) agent aims to attack the trajectory prediction model of the AV with the goal for that model to produce an erroneous prediction of the target agent. Existing approaches in the literature typically generate adversarial attacks by perturbing past states of the target vehicle \widetilde{X} in such a way that the corresponding prediction \widehat{\widetilde{Y}} deviates as much as possible from the ground-truth future states Y . However, existing methods do not guarantee that these future states Y actually lie in the reachable set of the target agent after the perturbation, which can result in an overall unsuccessful attack where \widehat{\widetilde{Y}} is actually consistent with the actual future of the adversarial agent. C) In our proposed approach, we generate (and constrain) perturbations not only for the past states \widetilde{X} but also future states \widetilde{Y} of the target agent. This ensures that the observed future Y stays dynamically feasible, which ensures that prediction errors resulting from the perturbation are meaningful. D) Our approach allows us to generate a novel type of adversarial attacks: false negative collision attack , where the adversarial agent takes a future trajectory that collides with the AV but deceives the AV into predicting a non-colliding trajectory. In B) - D), the yellow area represent the dynamically reachable set starting from the first observation, while the blue region represents the same starting from the prediction point. The green region represents a simple distance constraint often employed in adversarial attacks in the literature. Trajectory prediction – the task of forecasting the future positions of traffic participants over time – is a critical component of autonomous vehicle ( AV ) systems, allowing them to adapt to the actions of nearby road users (Figure 1 A). As AVs become more prevalent, ensuring the robustness and reliability of the prediction models is essential for public safety ( Hagenus et al. 2024 ; Llorca et al. 2024 ) . The use of adversarial trajectories – specifically generated to deceive the model prediction as much as possible – has emerged as a method for evaluating robustness of trajectory prediction models, helping to uncover vulnerabilities that could lead to dangerous situations on the road, particularly in the presence of adversarial agents ( Cao et al. 2022 ; Saadatnejad et al. 2022 ; Cao et al. 2023 ; Zhang et al. 2022 ; Jiao et al. 2023 ; Tan et al. 2022 ) . The ability of a model to withstand such attacks is known as adversarial robustness ( Tocchetti et al. 2025 ) . Recent advances in adversarial attack generation have shown promise in exposing system vulnerabilities ( Zhang et al. 2022 ; Cao et al. 2022 ; Yin et al. 2024 ; Tan et al. 2022 ; Fan et al. 2024a ) . Existing attack generation methods have traditionally focused on perturbing the observed position of the target vehicle. These methods can create scenarios that challenge trajectory prediction models, leading to prediction errors or simulated collisions (see Section 2 ). State-of-the-art approaches demonstrate two key capabilities. First, they generate trajectories that are dynamically feasible, ensuring that attacks can actually be reproduced in a real-world setting ( Zhang et al. 2022 ) . Second, the generated trajectories adhere to traffic norms, ensuring that resulting attacks cannot be trivially detected ( Cao et al. 2022 ; Zhang et al. 2022 ) , and ensuring that evaluations of prediction models on the perturbed data are meaningful. However, a key limitation of existing methods is their exclusive focus on perturbing observed (past) states of the target vehicle, disregarding the effect of perturbations on the vehicle’s future states. This means that existing attack methods can result in adversarial trajectories that restrict the agent to a different tactical behavior (maneuver) than the original trajectory, potentially negating the real-life effect of the attack. As an example, consider an intersection scenario where an adversarial agent intends to turn left, but aims to deceive the ego vehicle (AV across the intersection) into believing that it would not turn left (Figure 1 B). If the future states of the adversarial agent are not taken into account when generating the attack (as is the case for existing methods), the perturbation of the past trajectory might put the agent in the state from which the left turn is kinematically infeasible. As a result, the adversarial prediction would be consistent with the actual behavior of the agent, meaning that the attack would not actually deceive the ego agent (Figure 1 B). In summary, disregarding future states when generating trajectory perturbations limits the ability of existing attack methods to represent adversarial behaviors that AV s could encounter in the real world. This in turn limits the effectiveness of adversarial testing of trajectory prediction models. The challenge thus lies in generating adversarial trajectories that not only deceive trajectory prediction models, but also remain consistent with the target agent’s original tactical behavior throughout the entire trajectory, i.e., for both past and future states. In this paper, we present a novel method for generating adversarial attacks that accounts for both observed and future states. Our approach improves on existing techniques by introducing constraints on future states, ensuring that the entire adversarial trajectory maintains realistic driving behavior (Figure 1 C). This allows us to create more subtle and potentially dangerous attack scenarios, such as slight modifications to the path of a vehicle that lead to unexpected collisions while appearing benign to the prediction model

Read the complete text