A Pretrained Probabilistic Transformer for City-Scale Traffic Volume Prediction
Shiyu Shen, Bin Pan, Guirong Xue · 2025
Read A Pretrained Probabilistic Transformer for City-Scale Traffic Volume Prediction by Shiyu Shen, Bin Pan, Guirong Xue (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02654.txt.
Introduction
Counting cars across a whole city is hard because sensors see only part of the traffic, and the unseen part is biased. Shiyu Shen, Bin Pan, and Guirong Xue present a pretrained model for that task in this 2025 paper. Their model, TrafficPPT, gives not a single number but a spread of likely values, so its doubt is part of the answer.
Traffic is treated as the sum of many individual trips. The model blends live sensor readings, past trip records, and the road map to fill the gaps. It is first trained on large simulated cities, then tuned to each real city it serves. On real-world data sets it beat leading rivals, most clearly when observations were sparse. The authors plan to release the code in public.
Catalog record
| CREATOR | Shiyu Shen, Bin Pan, Guirong Xue |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02654
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Pretrained Probabilistic Transformer for City-Scale Traffic Volume Prediction Shiyu Shen, Bin Pan, Guirong Xue Abstract City-scale traffic volume prediction plays a pivotal role in intelligent transportation systems, yet remains a challenge due to the inherent incompleteness and bias in observational data. Although deep learning-based methods have shown considerable promise, most existing approaches produce deterministic point estimates, thereby neglecting the uncertainty arising from unobserved traffic flows. Furthermore, current models are typically trained in a city-specific manner, which hinders their generalizability and limits scalability across diverse urban contexts. To overcome these limitations, we introduce TrafficPPT, a Pretrained Probabilistic Transformer designed to model traffic volume as a distributional aggregation of trajectories. Our framework fuses heterogeneous data sources—including real-time observations, historical trajectory data, and road network topology—enabling robust and uncertainty-aware traffic inference. TrafficPPT is initially pretrained on large-scale simulated data spanning multiple urban scenarios, and later fine-tuned on target cities to ensure effective domain adaptation. Experiments on real-world datasets show that TrafficPPT consistently surpasses state-of-the-art baselines, particularly under conditions of extreme data sparsity. Code will be open. 1 Introduction Traffic volume prediction serves as a fundamental task in urban traffic management, providing essential insights for congestion control, safety improvement, and infrastructure development. Existing approaches encompass time series models vlahogianni2014short , deep learning architectures lv2014traffic , and graph neural networks yu2017spatio ; li2017diffusion . Despite recent advances, many existing methods assume access to complete traffic datasets fang2020meta ; chen2024traffic , which significantly limits their applicability in real-world scenarios. In practice, traffic monitoring relies primarily on two complementary yet imperfect data sources: (1) GPS data, which provides relatively detailed trajectory-level information but only for a subset of vehicles, and (2) fixed road sensors, which capture all passing vehicles but are sparsely deployed, often limited to key intersections or strategic checkpoints. This results in spatially fragmented observations, where urban centers typically exhibit high sensor density while suburban and rural areas remain under-monitored. Even in advanced cities with extensive downtown coverage, the intrinsic sparsity and uneven distribution of monitoring infrastructure still pose fundamental challenges. To overcome these limitations, some researchers have shifted toward utilizing incomplete checkpoint data as a more practical solution for city-scale prediction chen2023vehicle . Current approaches can be broadly categorized into: (1) traditional methods that employ prior probabilities for traffic volume estimation yu2023city ; bao2023pket , and (2) deep learning techniques that reconstruct missing trajectories from partial observations zhang2019short ; guo2024m . While demonstrating promising results, these methods exhibit two shortcomings: First, their end-to-end architectures lack interpretability while failing to account for potential alternative scenarios. Second, the mainstream city-specific training paradigm limits model generalizability and practical deployment potential. This represents a fundamental constraint, as most cities lack sufficient training data despite sharing common traffic pattern characteristics across urban areas. To address these challenges, we propose a pretrained probabilistic framework that provides a comprehensive and universal solution for traffic volume prediction. Our approach fundamentally reformulates the problem by treating traffic volume as an aggregation of probabilistic trajectory distributions. Rather than determining a deterministic trajectory for each vehicle, we estimate the probability of its presence on any road at any time. These probabilistic distributions are estimated through a neural network, which follows a two-stage training paradigm: (1) a pretraining stage on simulated data on diverse cities to learn universal traffic patterns, followed by (2) a fine-tuning stage on city-specific data to adapt to local road network characteristics. Building upon the probabilistic pretraining framework, we develop TrafficPPT, a transformer-based architecture that achieves comprehensive traffic volume inference with three innovations. First, the model integrates multimodal urban data - including real-time observations, historical trajectories, and road network topology - by projecting all heterogeneous inputs into a unified latent representation space. Second, we design a multi-view attention mechanism that dynamically captures spatial-temporal dependencies to estimate vehicle trajectory probabilities across the road network. Third, the architecture enables direct parallel prediction of complete traffic volumes through non-autoregressive inference, satisfying the strict latency requirements of real-world deployment scenarios. Experiments on city-scale road networks demonstrates that TrafficPPT outperforms existing methods in both accuracy and efficiency. Specifically, the model exhibits particular robustness to data sparsity, maintaining accurate predictions even with highly incomplete observations. The primary contributions of this paper are as follows: • Probabilistic Transformer Architecture: We propose TrafficPPT, which integrates multi-source traffic data to probabilistically infer city-scale traffic volumes. This framework captures the uncertainty of missing data and provides a more comprehensive inference. • Generalizable Pretraining Framework: We introduce a two-stage training paradigm combining large-scale simulation pretraining with targeted fine-tuning. The pretraining learns universal traffic patterns from diverse synthetic environments, enabling adaption to specific cities with limited data. • Data-Efficient Performance: TrafficPPT demonstrates high efficiency and high tolerance to missing data. With only 20% observations, TrafficPPT outperforms other models that require 50% observation ratio. These properties enable practical applications in resource-constrained urban settings. 2 Related Work 2.1 Traffic Volume Prediction Traffic volume prediction has traditionally relied on historical trajectory data collected through infrastructure-based sensors or GPS services, using models like LSTMs and GNNs yu2017spatio ; li2017diffusion ; zhang2017deep ; diao2019dynamic . However, the requirement for dense and citywide data collection renders these methods impractical in many real-world scenarios. To address this challenge, some studies focus on checkpoint-based data from key intersections, offering a more accessible alternative liu2020dynamic ; kalander2020spatio ; liu2018urban . A common line of work relies on prior distributions, assuming that missing trajectories follow specific patterns such as the shortest route patterson2020learning ; hunter2013path ; iio2023distribution . However, this approach often fail to reflect real-world driving behaviors. Other approaches adopt deep learning-based reconstruction. Although effective in certain settings, these models often neglect the stochastic and context-dependent vehicle behaviors. Moreover, both statistical and neural models frequently presume a single-path assumption, overlooking the multimodal nature of vehicle routing decisions tang2023explainable . As a result, their predictive capabilities degrade in highly sparse observation scenarios, constraining their applicability to city-scale deployment. Finally, while time-series models have demonstrated strong performance in capturing temporal dependencies, they often suffer from cumulative prediction errors and high inference latency, posing scalability challenges in large urban environments ma2015long ; zhao2019t ; wong2022view ; huang2023reconstructing . 2.2 Pretrained Transformer Pretrained transformers have gained significant attention following the success in natural language processing achiam2023gpt ; liu2024deepseek . Two predominant pretraining paradigms have emerged: (1) Masked Language Modeling, which randomly masks a subset of input tokens and trains the model to reconstruct the masked content devlin2019bert ; (2) Next Token Prediction, which conditions on a sequence of preceding tokens to predict the subsequent one radford2018improving . These strategies have been effectively applied to various tasks, including computer vision he2022masked and graph representation learning hu2019strategies ; hu2020gpt . In the context of traffic prediction, some studies have explored the use of pretrained transformers hou2024masked ; zhou2024trafficformer ; jin2021trafficbert . However, these methods often focus on specific datasets with sufficient data and lack a comprehensive framework that integrates heterogeneous information. 3 Method In this section, we present our model TrafficPPT. We begin by introducing the foundational notations in section 3.1 . The probabilistic modeling framework is detailed in section 3.2 , including the formulation of the probabilistic objective, estimation procedures, and inference strategies. Then, we describe the model architecture of TrafficPPT, with additional implementation details provided in