ART: Adaptive Relational Transformer for Pedestrian Trajectory Prediction with Temporal-Aware Relations

Ruochen Li, Ziyi Chang, Junyan Hu, Jiannan Li, Amir Atapour-Abarghouei, Hubert P. H. Shum · 2026

Read ART: Adaptive Relational Transformer for Pedestrian Trajectory Prediction with Temporal-Aware Relations by Ruochen Li, Ziyi Chang, Junyan Hu, Jiannan Li, Amir Atapour-Abarghouei, Hubert P. H. Shum (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03649.txt.

Introduction

Ruochen Li and colleagues built a model that forecasts where walkers will go next, a skill robots need to move safely among people. Their Adaptive Relational Transformer tracks how pairs of people relate to each other over time, then drops needless interaction work to save computing power. Tests on two walking and sports movement collections found it beat earlier methods while running efficiently.

Catalog record

CREATORRuochen Li, Ziyi Chang, Junyan Hu, Jiannan Li, Amir Atapour-Abarghouei, Hubert P. H. Shum
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24773
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03649

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ART: Adaptive Relational Transformer for Pedestrian Trajectory Prediction with Temporal-Aware Relations Ruochen Li, Ziyi Chang, Junyan Hu, Jiannan Li, Amir Atapour-Abarghouei, Hubert P. H. Shum Abstract Accurate prediction of real-world pedestrian trajectories is crucial for a wide range of robot-related applications. Recent approaches typically adopt graph-based or transformer-based frameworks to model interactions. Despite their effectiveness, these methods either introduce unnecessary computational overhead or struggle to represent the diverse and time-varying characteristics of human interactions. In this work, we present an Adaptive Relational Transformer (ART), which introduces a Temporal-Aware Relation Graph (TARG) to explicitly capture the evolution of pairwise interactions and an Adaptive Interaction Pruning (AIP) mechanism to reduce redundant computations efficiently. Extensive evaluations on ETH/UCY and NBA benchmarks show that ART delivers state-of-the-art accuracy with high computational efficiency. I INTRODUCTION Pedestrian trajectory prediction aims to forecast future locations of pedestrians given observed trajectories. It is a key component of human–robot interaction [ 1 , 16 ] , autonomous driving systems [ 6 , 25 ] , and smart surveillance infrastructure [ 21 , 19 ] . Accurate prediction of real-world pedestrian trajectories is crucial for robot-related applications, including human-aware robot navigation [ 7 , 10 ] and collision avoidance systems [ 38 , 22 ] . However, pedestrian trajectory prediction remains challenging due to the inherent stochasticity of motion behaviors and social interactions. Early approaches to pedestrian trajectory prediction modeled social interactions using pooling-based strategies that aggregate information from nearby pedestrians through fixed spatial windows [ 2 , 14 ] or occupancy maps [ 37 , 13 ] . To enable explicit and expressive interaction modeling, two main directions have emerged. On the one hand, graph-based methods encode pedestrians as nodes and model their pairwise interactions as edges, enabling structured message-passing via graph neural networks [ 20 , 21 , 35 , 23 ] . Specifically, these methods propagate interaction features via a message-passing paradigm over graph structures to model social influence among pedestrians. On the other hand, transformer-based approaches [ 17 , 30 , 39 , 40 ] leverage self-attention mechanisms [ 31 ] to model interactions by dynamically weighting the influence of surrounding agents across both spatial and temporal dimensions, providing a flexible framework for capturing long-range dependencies and complex motion patterns. Despite these advances, previous methods either incur redundant computation or fail to capture the heterogeneous and evolving nature of pedestrian interactions. Pedestrian trajectories contain rich spatial-temporal information, yet existing methods [ 17 , 35 , 30 , 29 ] construct pair-wise relations based on temporally aggregated representations, where trajectory sequences are compressed into static node features and relational reasoning is performed on these representations, thereby overlooking the dynamic evolution of interactions and time-varying strength. In addition, many methods construct social graphs using uniform neighbor selection, such as fully connected graphs [ 17 , 28 ] or top- k based neighbor selection [ 21 , 35 ] , which ignore the heterogeneous nature of pedestrian interactions. Fig. 1: Framework overview. Framework overview. The relation between pedestrians is inferred from the temporal evolution of their pairwise interactions over the observed history. Fig. 2: Overview of ART. Left: Temporal-Aware Relation Graph (TARG) leverages pairwise attention to model agent interactions across time steps, assigning higher weights to informative moments. Right: Adaptive Interaction Pruning (AIP) uses top- p filtering to adaptively retain informative neighbors based on cumulative interaction strength, producing a sparsified graph for trajectory prediction. In this work, we propose an Adaptive Relational Transformer (ART), a lightweight Transformer-based framework for pedestrian trajectory prediction. First, to explicitly capture the temporal evolution of pairwise interactions, we introduce a Temporal-Aware Relation Graph (TARG). Unlike existing methods that compress multi-step trajectories into static embeddings before computing pairwise relations, TARG leverages pairwise attention [ 15 ] to model agent interactions at each time step and aggregates them via learnable weights, providing a temporally-aware relation modeling where informative time steps (e.g., moments of close proximity or directional change) receive higher weights, while less relevant moments contribute minimally to the final pairwise relations (see Figure 1 ). Second, to reduce redundancy in dense interaction graphs, we propose Adaptive Interaction Pruning (AIP) via top- p filtering, which adaptively selects informative neighbors based on interaction strength [ 24 ] , rather than enforcing a fixed top- k neighborhood. For each agent, only neighbors whose attention weights jointly exceed a threshold p are retained, resulting in an adaptively sparsified interaction graph that retains only informative neighbors. The proposed ART achieves state-of-the-art (SOTA) results on both the ETH/UCY [ 27 , 18 ] and NBA SportVU [ 26 ] benchmarks while maintaining superior computational efficiency over prior approaches. The main contributions are: • We propose an Adaptive Relational Transformer (ART), a lightweight Transformer-based framework for pedestrian trajectory prediction that explicitly models social interactions while maintaining computational efficiency. • We introduce a Temporal-Aware Relation Graph (TARG) that explicitly models pairwise pedestrian interactions by accounting for the relative importance of different time steps. • We propose Adaptive Interaction Pruning (AIP) via top- p filtering to adaptively sparsify dense interaction graphs based on cumulative interaction strength, reducing redundancy and improving adaptivity. II METHOD II-A Problem Formulation Pedestrian trajectory prediction aims to infer pedestrians’ future movements from their observed trajectories. Consider a scene containing M pedestrians observed over T_{h} time steps. The historical trajectory of pedestrian i is represented as \mathbf{p}_{j}^{h}=\{(x^{t}_{i},y^{t}_{i})\mid(x^{t}_{i},y^{t}_{i})\in\mathbb{R}^{2},t=1,\dots,T_{h}\} , where (x^{t}_{i},y^{t}_{i}) denotes the 2D spatial coordinates at time step t . The corresponding ground-truth (GT) future trajectory over a prediction horizon of T_{f} time steps is denoted as \mathbf{p}_{j}^{f}=\{(x^{t}_{i},y^{t}_{i})\mid t=T_{h}+1,\dots,T_{h}+T_{f}\} . By stacking trajectories of all pedestrians, the observed and GT trajectories can be expressed as \mathbf{P}^{h}\in\mathbb{R}^{M\times T_{h}\times 2} and \mathbf{P}^{f}\in\mathbb{R}^{M\times T_{f}\times 2} . The training objective minimizes the discrepancy between the predicted trajectories \hat{\mathbf{P}}^{f} and the ground-truth \mathbf{P}^{f} . II-B Temporal-Aware Relation Graph Construction Existing graph construction methods compress \mathbf{P}_{h}\in\mathbb{R}^{M\times T_{h}\times 2} into static embeddings before computing pairwise weights, losing temporal information about when interactions occur. As shown in Figure 2 (Left), we propose Temporal-Aware Relation Graph (TARG), which preserves temporal attribution by applying attention over observation sequences before spatial aggregation. We encode the observed trajectories into high-dimensional features. Given \mathbf{P}_{h}\in\mathbb{R}^{M\times T_{h}\times 2} , we apply a linear projection with positional encoding to obtain temporal node features: \mathbf{H}=\text{PosEnc}(\mathbf{W}_{\text{in}}\mathbf{P}_{h})\in\mathbb{R}^{M\times T_{h}\times d}, (1) where \mathbf{W}_{\text{in}}\in\mathbb{R}^{d\times 2} is a learnable projection matrix, d is the hidden dimension, and \text{PosEnc}(\cdot) denotes sinusoidal positional encoding [ 31 ] that injects temporal information. To capture time-varying interactions between pedestrian pairs, we reshape \mathbf{H} from \mathbb{R}^{M\times T_{h}\times d} to \mathbb{R}^{T_{h}\times M\times d} and apply multi-head attention [ 31 ] over the temporal dimension. For each attention head h\in\{1,\dots,H\} , we have: \mathbf{Q}^{h}=\mathbf{H}\mathbf{W}_{Q}^{h},\quad\mathbf{K}^{h}=\mathbf{H}\mathbf{W}_{K}^{h},\quad\mathbf{V}^{h}=\mathbf{H}\mathbf{W}_{V}^{h}, (2) where \mathbf{W}_{Q}^{h},\mathbf{W}_{K}^{h},\mathbf{W}_{V}^{h}\in\mathbb{R}^{d\times d_{h}} are learnable weight matrices and d_{h}=d/H is the dimension per head. For each pair of pedestrians (i,j) , we compute time-resolved attention scores by applying attention across all time steps: \alpha_{ij}^{h}(t)=\frac{\exp\left(\frac{\mathbf{Q}^{h}_{i}(t)^{\top}\mathbf{K}^{h}_{j}(t)}{\sqrt{d_{h}}}\right)}{\sum_{t^{\prime}=1}^{T_{h}}\exp\left(\frac{\mathbf{Q}^{h}_{i}(t^{\prime})^{\top}\mathbf{K}^{h}_{j}(t^{\prime})}{\sqrt{d_{h}}}\right)},\quad t=1,\dots,T_{h}, (3) where \alpha_{ij}^{h}(t)\in[0,1] represents the attention weight for pedestrian pair (i,j) at time step t under head h , and \sum_{t=1}^{T_{h}}\alpha_{ij}^{h}(t)=1 . These temporally-aware scores explicitly capture when each interaction becomes significant, preserving temporal attribution throughout the graph construction. The attention scores are used to aggregate value representations across time: \mathbf{R}_{ij}^{h}=\sum_{t=1}^{T_{h}}\alpha_{ij}^{h}(t)\cdot\mathbf{V}^{h}_{j}(t)\in\mathbb{R}^{d_{h}}, (4) where \mathbf{R}_{ij}^{h} is the aggregated relation representation for

Read the complete text