Relation-Aware LNN–Transformer for Intersection-Centric Next-Step Prediction
Zhehong Ren, Tianluo Zhang, Yiheng Lu, Yushen Liang, Promethee Spathis · 2025
Read Relation-Aware LNN–Transformer for Intersection-Centric Next-Step Prediction by Zhehong Ren, Tianluo Zhang, Yiheng Lu, Yushen Liang, Promethee Spathis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01368.txt.
Introduction
Zhehong Ren, Tianluo Zhang, Yiheng Lu, Yushen Liang, and Promethee Spathis predict a road user's next intersection rather than next destination, in a two thousand twenty-five machine learning paper. Most mobility models assume people choose among a fixed set of points of interest.
Their framework represents trajectories on the city's road-intersection graph, dropping the closed-world assumption. Sector-wise directional aggregation of nearby points of interest supplies compact context features, combined with structural graph embeddings. A hybrid model joins a liquid neural network with bearing-biased transformer attention. On city-scale trajectories it beat six leading baselines by wide accuracy margins and stayed resilient under GPS and point-of-interest noise.
Catalog record
| CREATOR | Zhehong Ren, Tianluo Zhang, Yiheng Lu, Yushen Liang, Promethee Spathis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01368
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Relation-Aware LNN–Transformer for Intersection-Centric Next-Step Prediction Zhehong Ren, Tianluo Zhang, Yiheng Lu, Yushen Liang, Promethee Spathis Abstract Next-step location prediction plays a pivotal role in modeling human mobility, underpinning applications from personalized navigation to strategic urban planning. However, approaches that assume a closed world—restricting choices to a predefined set of points of interest (POIs)—often fail to capture exploratory or target-agnostic behavior and the topological constraints of urban road networks. Hence, we introduce a road-node-centric framework that represents road-user trajectories on the city’s road-intersection graph, thereby relaxing the closed-world constraint and supporting next-step forecasting beyond fixed POI sets. To encode environmental context, we introduce a sector-wise directional POI aggregation that produces compact features capturing distance, bearing, density, and presence cues. By combining these cues with structural graph embeddings, we obtain semantically grounded node representations. For sequence modeling, we integrate a Relation-Aware LNN–Transformer—a hybrid of a Continuous-time Forgetting Cell (CfC-LNN) and a bearing-biased self-attention module—to capture both fine-grained temporal dynamics and long-range spatial dependencies. Evaluated on city-scale road-user trajectories, our model outperforms six state-of-the-art baselines by up to 17 percentage points in Acc@1 and 10 percentage points in MRR, and maintains high resilience under noise—losing only 2.4 percentage points Acc@1 at 50 m GPS perturbation and 8.9 percentage points Acc@1 at 25% POI noise. Introduction Predicting the next intersection for a road user—across walking, cycling, and driving—is a cornerstone of human mobility modeling, driving applications in personalized navigation, traffic forecasting, and urban planning. A large body of next‑POI work frames prediction as sequence modeling over a limited POI vocabulary, often overlooking (i) the topological constraints of the road graph and (ii) the directional distribution of nearby semantics. For instance, Transformer‑based approaches typically treat trajectories as flat token streams without encoding intersections as decision points or road connectivity, which can induce candidate bias and a coarse treatment of street structure (e.g. ( Xu, Huang, and Zou 2023 ) ). Even graph‑enhanced POI recommenders such as GETNext ( Yang, Liu, and Zhao 2023 ) still operate on POI tokens rather than directly on the road‑intersection state space. Nevertheless, empirical road‑user movement exhibits substantial exploration: using a high‑precision, longitudinal dataset of 400+ users, Cuttone et al. report that 20–25% of transitions lead to previously unseen places and that roughly 70% of locations are visited only once ( Cuttone, Lehmann, and González 2016 ) . Such exploratory behavior directly challenges fixed‑vocabulary next‑POI predictors and motivates modeling unbounded by a predefined POI token set. We address such gaps with a Road-Node–Centric Next-Step Prediction framework that represents trajectories on the city’s intersection graph and combines static semantic context, topology, and geography information. The three main contributions of our paper are the following: • Sector‑wise, direction‑aware node representation. We partition each intersection’s neighborhood into equal‑angle sectors and, within each sector, summarize multi‑category POI cues (distance, bearing, density, presence) to form a compact, inductive node descriptor. This sectorization integrates directionality at the feature level—capturing the anisotropy of urban space and aligning with outgoing road bearings—so that one‑hop candidates that are topologically similar but differ by direction become distinguishable. • Relation-Aware LNN–Transformer for spatiotemporal encoding. We build a unified architecture by first encoding trajectory dynamics with a Continuous-time Forgetting Cell (CfC-LNN) and then applying a Relation-Aware Transformer that adds a bearing-based bias to its self-attention logits. The Rel-Aware LNN–Transformer captures fine-grained temporal changes, and the bearing-biased attention captures long-range directional dependencies on the road graph—all within a compact 2.34M-parameter model. • Empirical and robustness gains. On city-scale road-user trajectories (1–10 km), our model surpasses six competitive baselines in Accuracy@1 and MRR, and degrades gracefully under realistic GPS perturbations and POI-feature noise. We attribute the robustness partly to the stable continuous-time recurrence in CfC-LNN (closed-form state updates) coupled with topology-aware attention; training remains gradient-based, but the forward dynamics improve conditioning and sample efficiency. Related Work Liquid Neural Networks (LNNs) Liquid neural networks model sequence dynamics in continuous time with input‑modulated state equations, offering compact models and strong stability/robustness properties relative to ODE/RNN baselines. Early “Liquid Time‑constant” (LTC) models define linear first‑order dynamics with gates and are trained by backpropagation through time (BPTT) ( Hasani et al. 2020 ) . Closed‑form continuous‑time (CfC) variants further avoid heavy numerical solvers by deriving analytic updates, improving efficiency while preserving expressivity ( Hasani et al. 2022 ) . LNNs have demonstrated resilience to distribution shifts in control and navigation, e.g., robust out‑of‑distribution flight with vision inputs ( Hasani, Lechner, and Grosu 2023 ) . These properties motivate our use of a CfC‑LNN encoder for irregular dwell times and non‑uniform sampling in trajectories. Next POI/Location Prediction Sequence‑only models. RNN variants inject temporal and spatial intervals into gates or transitions but treat POIs as vocabulary tokens without explicit road topology: ST‑LSTM with time/distance gates ( Zhao et al. 2018 ) , STRNN with time/distance‑specific transition matrices ( liu2016strnn ) , DeepMove with historical attention ( Feng and et al. 2018 ) , and ARNN with attentional sequence modeling ( Guo and et al. 2020 ) . Transformer‑based models. Recent Transformers capture long‑range dependencies yet typically operate on flat token sequences; examples include GETNext, which augments a Transformer with a global trajectory flow map for next‑POI recommendation ( Yang, Liu, and Zhao 2023 ) , and STTF‑Recommender for mining non‑contiguous visit relations ( Xu, Huang, and Zou 2023 ) . Although GETNext and STTFRecommender both advance the next POI prediction, neither incorporates explicit road-graph connectivity or direction-aware POI semantics, which our intersection-level sector-wise aggregation is designed to remedy. Input Embeddings and Information Aggregation Structural priors. Node2Vec learns transductive structural embeddings via biased random walks that mix homophily and structural equivalence ( Grover and Leskovec 2016 ) . GraphSAGE learns inductive neighborhood aggregators that generalize to unseen nodes ( Hamilton, Ying, and Leskovec 2017 ) . Implication for road networks. Human movement on streets is directional and constrained by intersection geometry; we therefore combine strong structural priors (Node2Vec) with an orientation‑aware, sectorized pooling that injects distance/bearing/density semantics into descriptors, complementing GraphSAGE‑style local aggregation. Methodology As Figure 1 illustrates our pipeline, every trajectory is first snapped to intersections and enriched with POI and geometric features. Then, these features are differenced and processed by our Relation-Aware LNN–Transformer architecture. Finally, a softmax head over one‑hop neighbors produces the next‑node prediction. Figure 1: Overview of our Relation‑Aware LNN–Transformer. A road‑user trajectory is projected onto the intersection graph and encoded via sector‑wise POI and geometric features fused with Node2Vec embeddings. The differenced features go through a CfC‑LNN and L\!-\!1 standard Transformer layers, followed by a relation‑aware attention block that biases self‑attention with inter‑node bearings, and finally a softmax over one‑hop candidates. Problem Definition Let G=(V,E) be an undirected road graph whose nodes V are intersections and edges E are road segments. Each node v has a structural embedding \mathbf{h}_{v}^{\mathrm{struct}}\in\mathbb{R}^{d_{s}} and a POI descriptor \mathbf{h}_{v}^{\mathrm{POI}}\in\mathbb{R}^{d_{p}} . Let \mathbf{p}_{v}\in\mathbb{R}^{2} denote its planar coordinates (fixed coordinate reference system). Given a trajectory \tau=(v_{1},\dots,v_{T}) , we predict the next node v_{T+1} from the one‑hop candidate set \mathcal{C}(v_{T})=\mathcal{N}(v_{T}) . We pass \tau through our encoder (Secs. Sector-Wise Directional POI Aggregation - Relation‑Aware Transformer ) to obtain a representation \mathbf{z}_{u} for each u\in\mathcal{C}(v_{T}) , and score \begin{split}s_{\theta}(u\mid\tau)&=\mathbf{w}^{\top}\mathbf{z}_{u}+b,\\ p_{\theta}(u\mid\tau)&=\operatorname{softmax}\bigl(s_{\theta}(u\mid\tau)\bigr)\quad\text{for }u\in\mathcal{N}(v_{T})\,.\end{split} (1) We define \mathcal{L}_{\mathrm{CE}} , the classification loss as follows: \mathcal{L}_{\mathrm{CE}}=-\tfrac{1}{N}\!\sum_{i=1}^{N}\log p_{\theta}\!\bigl(v_{T+1}^{(i)}\mid\tau^{(i)}\bigr). (2)