Efficient Temporal Tokenization for Mobility Prediction with Large Language Models
Haoyu He, Haozheng Luo, Yan Chen, Qi R. Wang · 2025
Read Efficient Temporal Tokenization for Mobility Prediction with Large Language Models by Haoyu He, Haozheng Luo, Yan Chen, Qi R. Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14017.txt.
Introduction
Haoyu He, Haozheng Luo, Yan Chen, and Qi Wang teach language models to predict where people will move, in a 2025 paper. Forecasting a person's next stops helps traffic planning, app design, and emergency work. Raw movement trails are long and messy for a language model to read directly.
Their framework, RHYTHM, chops each trail into daily pieces and encodes them as compact tokens. Attention works at two levels, within days and across weeks, catching both routine and weekly rhythm. The language model stays frozen, which cuts training cost. On three real data sets the method beat prior tools on accuracy, did especially well on weekends, and trained about a quarter faster.
Catalog record
| CREATOR | Haoyu He, Haozheng Luo, Yan Chen, Qi R. Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-14017
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Temporal Tokenization for Mobility Prediction with Large Language Models Haoyu He, Haozheng Luo, Yan Chen, Qi R. Wang Abstract We introduce RHYTHM ( R easoning with H ierarchical T emporal T okenization for H uman M obility), a framework that leverages large language models (LLMs) as spatio-temporal predictors and trajectory reasoners. RHYTHM partitions trajectories into daily segments encoded as discrete tokens with hierarchical attention, capturing both daily and weekly dependencies while substantially reducing the sequence length. Token representations are enriched with pre-computed prompt embeddings via a frozen LLM, enhancing the model’s ability to capture interdependencies without extensive computational overhead. By freezing the LLM backbone, RHYTHM achieves significant computational efficiency. Evaluation on three real-world datasets demonstrates a 2.4% improvement in accuracy, 5.0% increase on weekends, and 24.6% reduction in training time compared to state-of-the-art methods. † † affiliationnotice: Equal contribution 1 Introduction We propose RHYTHM ( R easoning with H ierarchical Temporal T okenization for H uman M obility), a novel foundation model for human mobility prediction. Our approach reconceptualizes mobility modeling through structured temporal abstraction, combining efficient multi-scale temporal tokenization with the complex reasoning capabilities of pretrained LLMs. This integration yields a computationally efficient yet powerful framework for trajectory prediction, offering superior accuracy while maintaining efficiency and scalability across diverse mobility contexts. Our framework is grounded in the inherent patterns of human mobility. Research shows that human movement follows predictable daily and weekly rhythms ( Cho et al. 2011 ; Gonzalez et al. 2008 ; Eagle & Pentland 2006 ) , with Song et al. 2010 demonstrating that 93% of daily trajectories are predictable. Yet modeling the complex interdependence between locations and temporal cycles remains challenging. Existing approaches either neglect long-term periodicity ( Yang et al. 2020 ; Feng et al. 2018 ; Gambs et al. 2012 ) or fail to capture multi-scale temporal patterns ( Hong et al. 2023 ; Wu et al. 2019 ) . We address these limitations by decomposing the trajectories into meaningful segments and tokenizing them into discrete representations. Our approach employs intra-segment attention for local patterns and inter-segment attention for long-range dependencies (see Figure 1 ), significantly reducing computational complexity. Each token is further enriched with pre-computed prompt embeddings derived from a frozen LLM, integrating trajectory context and task descriptions to enhance semantic understanding. Figure 1 : Motivation for RHYTHM. By partitioning trajectories into discrete tokens instead of a continuous stream, RHYTHM more effectively captures recurring mobility patterns. Recent research highlights LLMs’ dual capabilities as both representation extractors for spatio-temporal patterns and sophisticated reasoning engines ( Chowdhery et al. 2023 ) . These models excel through few-shot prompting ( Brown et al. 2020 ) , chain-of-thought reasoning ( Pan et al. 2025 ; Pan et al. 2024 ; Wei et al. 2022 ) , and in-context learning ( Dong et al. 2024 ) . While mobility-specific models like PMT ( Wu et al. 2024b ) and ST-MoE-BERT ( He et al. 2024 ) lack LLM integration to model complex human movement correlations, RHYTHM addresses this limitation through an integrated LLM-based reasoning module. Our parameter-efficient strategy freezes the pre-trained LLM backbone, avoiding extensive fine-tuning while preserving reasoning capabilities. This approach effectively balances fine-grained spatio-temporal modeling with minimal computational overhead, making RHYTHM particularly suitable for resource-constrained settings. In summary, our key contributions are: • We introduce a novel temporal tokenization scheme that represents daily mobility routines as discrete tokens, capturing multi-scale cyclical dependencies while markedly reducing sequence length. • We propose an efficient prompt-guided integration of semantic trajectory cues and task descriptions via segment embeddings, enhancing interpretability of complex mobility behaviors. • We design parameter-efficient LLM adaptation strategy leveraging frozen backbones, requiring only 12.37% of model parameters and cutting computational cost by 24.6% . • Evaluation on three real-world datasets, yielding a 2.4% overall accuracy gain and a 5.0% improvement on weekend predictions over state-of-the-art baselines. Figure 2 : The proposed architecture of RHYTHM. Historical trajectories are first converted into spatio–temporal embeddings and discretized via temporal tokenization ( b ), enabling hierarchical attention to capture both local and global dynamics. Each segment token is enriched with semantic trajectory embeddings, while future time‐step tokens integrate task‐context descriptors ( a ). The resulting token sequence is fed into a frozen LLM backbone, and an output projection layer produces the final location predictions. 2 Method In this section, we present RHYTHM , an LLM-driven architecture for prompt-guided representation learning of periodic spatio-temporal patterns (see Figure 2 ). 2.1 Problem definition Let \mathcal{X}=\{x_{1},x_{2},\ldots,x_{T}\} be a user’s trajectory, where each observation x_{i}=(t_{i},l_{i}) pairs a timestamp t_{i} with a location l_{i}\in\mathcal{L} drawn from a finite set \mathcal{L} . For a prediction horizon H , define the future time points \mathcal{T}=\{t_{T+1},\ldots,t_{T+H}\} . Our goal is to predict the corresponding locations \mathcal{Y}=\{l_{T+1},\ldots,l_{T+H}\} . Formally, we learn a mapping f:\;(\mathcal{X},\,\mathcal{T})\;\mapsto\;\mathcal{Y} that takes the historical trajectory and future timestamps to the future location sequence. 2.2 Model structure Spatio-Temporal Feature Encoding. For each observation x_{i} , we design temporal representations to encode the periodic nature of human mobility: \displaystyle\mathbf{E}^{\text{temporal}}_{i}=\mathbf{E}^{\text{ToD}}(t_{i})\|\mathbf{E}^{\text{DoW}}(t_{i}), where \cdot\|\cdot denotes the concatenation operation, \mathbf{E}^{\text{ToD}} encodes time-of-day information, and \mathbf{E}^{\text{DoW}} captures day-of-week patterns. These trainable embeddings transform discrete time indices into dense vector representations, yielding \mathbf{E}^{\text{temporal}}_{i}\in\mathbb{R}^{D} where D corresponds to the input dimensionality of the underlying LLM. The spatial representation \mathbf{E}^{\text{spatial}}_{i}\in\mathbb{R}^{D} for location l_{i} is formulated as: \mathbf{E}^{\text{spatial}}_{i}=\mathbf{E}^{\text{Loc}}(l_{i})\|(W_{\text{coord}}[\text{lat}_{i},\text{lon}_{i}]^{T}+b_{\text{coord}}), where \mathbf{E}^{\text{Loc}} represents the location-specific categorical embedding, while the latter term transforms the geographic coordinates ( \text{lat}_{i},\text{lon}_{i} ) to the embedding domain through the transformation matrix W_{\text{coord}}\in\mathbb{R}^{d_{\text{coord}}\times 2} and d_{\text{coord}} indicates the output dimension of the projection. The unified spatio-temporal representation \mathbf{E}_{i}\in\mathbb{R}^{D} is computed through element-wise summation: \mathbf{E}_{i}=\mathbf{E}^{\text{temporal}}_{i}+\mathbf{E}^{\text{spatial}}_{i}. When handling incomplete historical observations, we assign zero values to the spatial components, enabling the model to function solely with temporal features while maintaining consistent dimensionality. Temporal Tokenization. Human movement behaviors demonstrate intrinsic hierarchical temporal patterns that encompasses both local behavioral routines and cyclical regularities ( Song et al. 2010 ; Gonzalez et al. 2008 ) . To address these multi-scale dynamics, RHYTHM introduces a temporal segmentation strategy that separates fine-grained local behaviors from overarching temporal relationships, drawing inspiration from Liu et al. 2024c . Specifically, we divide the embedded representation \mathcal{X} into N distinct, consecutive segments \{\mathbf{s}_{1},\ldots,\mathbf{s}_{N}\} , where each segment encodes semantically coherent time periods (such as daily sequences): \displaystyle\mathbf{s}_{i}=\{E_{(i-1)L+1},\dots,E_{iL}\}\quad\text{for }i=1,\dots,N, with L denoting the temporal span within each segment \mathbf{s_{i}} . For capturing local sequential patterns within segments, we utilize intra-segment attention: \displaystyle\tilde{\mathbf{E}}^{(i)}=\mathrm{Attention}(\mathbf{s}_{i}), We implement a pre-normalized transformer design incorporating gated feed-forward modules following Dubey et al. 2024 , facilitating stable gradient propagation and enhancing representational capacity. Detailed architectural specifications can be found in Appendix C . To efficiently encode relationships across segments, we introduce a trainable aggregation mechanism that compresses each segment into a compact token: \displaystyle\mathbf{SE}_{i}=\mathrm{Pool}\bigl(\tilde{\mathbf{E}}^{(i)}\bigr). These compressed segment representations \{\tilde{\mathbf{SE}}_{1},\ldots,\tilde{\mathbf{SE}}_{N}\} are then processed through inter-segment attention to encode extended temporal relationships and distant dependencies: \displaystyle\tilde{\mathbf{SE}}_{1:N}=\operatorname{Attention}(\mathbf{SE}_{1:N}), producing enhanced segment embeddings \mathbf{\tilde{SE}_{i}}\in\mathbb{R}^{D} that incorporate contextual signals from various temporal granularities. This design compresses the operational sequence length from T to N while maintaining both detailed temporal features and long-range correlations, thereby mitigating the computational burden associated with processing lengthy mobility sequences. Semantic Context Integration. The integration of LLMs into mobility prediction presents a fundamental tension: