Digital-Twin Losses for Lane-Compliant Trajectory Prediction at Urban Intersections
Kuo-Yi Chao 1 , Erik Leo Haß 1 , Melina Gegg 2 , Jiajie Zhang 1 , Ralph Raßhofer 2 , Alois Christian Knoll 1, Email: [email protected] · 2026
Read Digital-Twin Losses for Lane-Compliant Trajectory Prediction at Urban Intersections by Kuo-Yi Chao 1 , Erik Leo Haß 1 , Melina Gegg 2 , Jiajie Zhang 1 , Ralph Raßhofer 2 , Alois Christian Knoll 1, Email: [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05546.txt.
Introduction
Chao and team build a digital twin-driven V2X chain forecasting multi-agent motion at intersections from shared vehicle and infrastructure sensing. Their model pairs a Bi-LSTM generator with mean squared error and twin losses. The twin loss captures infrastructure limits, collision dodging, mode range, and rule-based priors. The MSE term keeps point-wise results right, while the twin loss counts rule breaks, forecast collisions, and mode collapse against it. Tests on real urban V2X data show serious breaks fall a lot with similar accuracy.
Catalog record
| CREATOR | Kuo-Yi Chao 1 , Erik Leo Haß 1 , Melina Gegg 2 , Jiajie Zhang 1 , Ralph Raßhofer 2 , Alois Christian Knoll 1, Email: [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26593 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05546
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Digital-Twin Losses for Lane-Compliant Trajectory Prediction at Urban Intersections Kuo-Yi Chao 1 , Erik Leo Haß 1 , Melina Gegg 2 , Jiajie Zhang 1 , Ralph Raßhofer 2 , Alois Christian Knoll 1, Email: [email protected] Abstract Accurate and safety-conscious trajectory prediction is a key technology for intelligent transportation systems, especially in V2X-enabled urban environments with complex multi-agent interactions. In this paper, we created a digital twin-driven V2X trajectory prediction pipeline that jointly leverages cooperative perception from vehicles and infrastructure to forecast multi-agent motion at signalized intersections. The proposed model combines a Bi-LSTM-based generator with a structured training objective consisting of a standard mean squared error (MSE) loss and a novel twin loss. The twin loss encodes infrastructure constraints, collision avoidance, diversity across predicted modes, and rule-based priors derived from the digital twin. While the MSE term ensures point-wise accuracy, the twin loss penalizes traffic rule violations, predicted collisions, and mode collapse, guiding the model toward scene-consistent and safety-compliant predictions. We train and evaluate our approach on real-world V2X data sent from the intersection to the vehicle and collected in urban corridors. In addition to standard trajectory metrics (ADE, FDE), we introduce ITS-relevant safety indicators, including infrastructure and rule violation rates. Experimental results demonstrate that the proposed training scheme significantly reduces critical violations while maintaining comparable prediction accuracy and real-time performance, highlighting the potential of digital twin-driven multi-loss learning for V2X-enabled intelligent transportation systems. Fig. 1: TUM intersection scene: OpenDRIVE lane map (grey, 2,233 segments) overlaid with recorded agent trajectories (946 objects, coloured by class) from the first 30,000 frames. The lane geometry serves as the infrastructure prior for the proposed twin-consistency loss. I Introduction Safe autonomous driving at urban intersections requires precise predictions of the future movements of all road users over time horizons at least 1.6 seconds [ 11 ] for safety reasons, and usually of 1 to 5 seconds. Unlike scenarios on highways or in pedestrian environments, movement at intersections is largely determined by discrete turning maneuvers, compliance with traffic rules, and interactions between heterogeneous classes of agents, such as cars, trucks, motorcycles, bicycles, and pedestrians. These factors render simple physical models inadequate and also pose significant challenges for purely data-driven approaches that ignore road structure. Classic motion models [ 7 ] [ 9 ] [ 15 ] continue to be widely used in real-time stacks of autonomous vehicles because they are simple and robust. The constant velocity (CV) model [ 12 ] incorporates the last observed forward velocity over time and requires no training. The Kalman filter with a Singer constant acceleration process model complements a correlated acceleration prior, improving accuracy for short prediction horizons. However, both models exhibit divergent behavior over longer time horizons, as they cannot represent the curvature of intersection geometries. Deep learning approaches have recently achieved state-of-the-art performance on pedestrian benchmarks of ETH/UCY and on large-scale driving datasets like nuScenes and Argoverse. However, the transition from simulation performance to real-world performance remains challenging. The main difficulties lie in generalizing to previously unknown intersection geometries, in the effort required to construct scene graphs to model social interactions, and in the high computational costs of multimodal generative models [ 9 ] . This work explores a pragmatic middle ground: a standard LSTM encoder-decoder trained with map-aware auxiliary losses derived from an HD map with lane centerlines. This approach requires no additional network parameters beyond the base model, no scene graph construction, and no modification to inference latency. Map information is only considered during training, shaping the loss landscape so that the decoder naturally generates lane-compliant trajectories. An important implementation consideration, which is rarely discussed in the literature, concerns the coordinate frame of auxiliary losses. Trajectory models are typically trained in an anchored coordinate system (relative to the anchor), in which all positions are expressed as offsets from the last observed position (the anchor), typically within a range of \pm 30 m for history and 0 to 70 m for five-second predictions. If an infrastructure loss is applied directly to absolute HD map coordinates (e.g., in local tangent plane (ENU) meters), constant map offsets cause the loss to provide virtually no informative gradient. We document this problem, propose a solution in principle, and quantify its effects. In addition, two systematic errors in infrastructure and collision assessment measurements are identified and corrected: 1. calculating infrastructure violation as the absolute mean size of the ENU position instead of the minimum distance between traces 2. calculating self-intersections within a single predicted trajectory instead of the conflicting interaction between multiple agents. Our contributions are: • A fully reproducible pipeline for data preprocessing, training, and evaluation of intersection trajectory prediction based on a recording of over. The dataset contains approximately 90 minutes of continuous sensor data at 15 Hz and captures around 20.000 individual objects with their trajectories. After preprocessing, this results in approximately 1.14 million sliding-window samples ( \approx 797k training, 171k validation, and 171k test samples), evaluated across five prediction horizons ( 1 to 5 s). • An anchor-relative data representation with explicit anchor files, enabling a clear separation between the relative training space and the absolute ENU evaluation space. • Infrastructure and collision loss functions operating in absolute ENU coordinates via anchor recovery, thereby avoiding coordinate-frame inconsistencies. • Evaluation metrics for infrastructure compliance and trajectory self-intersection, with clearly defined semantics. • A systematic ablation study covering five model variants across five prediction horizons (25 models in total), along with two classical baseline methods. II Related Work II-A Classical Motion Models Classic motion models [ 15 ] like velocity (CV), constant acceleration (CA), or Kalman filter (KF) models [ 19 ] [ 18 ] , are simple, robust, and still widely used. The CV model propagates the last observed velocity forward, which is equivalent to a zero-noise Kalman filter for position. Under moderate process noise, it becomes a random walk model in velocity space. The Singer model [ 5 ] introduces a Gaussian-Markov acceleration process with a maneuver time constant adjusted to typical vehicle acceleration profiles. The CV and Singer models [ 6 ] both serve as solid baselines for horizons \leq 1\,\text{s} , but diverge on curved trajectories [ 15 ] . II-B Recurrent Trajectory Models Sequence-to-sequence LSTM models [ 14 ] were among the first deep architectures to outperform long-term CV on pedestrian data. Social-LSTM [ 1 ] extended this approach by summarizing the hidden states of spatially adjacent agents on a social grid, enabling flexible modeling of interactions. Social-GAN [ 4 ] replaced the deterministic decoder with a GAN to generate multi-modal trajectories. These models operate in the pixel space of the camera or aerial image without HD map conditioning. II-C Map-Aware Trajectory Prediction The integration of HD maps as model inputs has become standard practice in vehicle trajectory prediction. VectorNet [ 3 ] encodes both the trajectories of the agents and the polylines of the map as subgraphs and performs cross-attention between them. LaneGCN [ 8 ] creates a graph of lane segments with four connection types (predecessor, successor, left/right neighbors) and propagates information along the graph before merging it with the agents’ features. Trajectron++ [ 10 ] integrates dynamic maps and multi-agent interaction in a conditional variational autoencoder framework with a physically feasible