FlowDistill: Scalable Traffic Flow Prediction via Distillation from LLMs [Scalable Data Science]

Chenyang Yu, Xinpeng Xie, Yan Huang, and Chenxi Qiu · 2025

Read FlowDistill: Scalable Traffic Flow Prediction via Distillation from LLMs [Scalable Data Science] by Chenyang Yu, Xinpeng Xie, Yan Huang, and Chenxi Qiu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02094.txt.

Introduction

Predicting city traffic flow helps manage congestion, but the best models are heavy and hungry for data. Chenyang Yu and co-authors propose FlowDistill, a light framework that learns from a large language model instead. A fine-tuned language model acts as teacher, and a small neural network learns as its student.

The training mix uses an information bottleneck rule and a teacher-bounded loss, so the student keeps only the knowledge that transfers. Spatial and temporal patterns are coded in explicitly to help the small model generalize across cities. In tests, FlowDistill beat leading models on accuracy while using less training data, memory, and time. The authors present it as a practical route for cities with limited data or computing power.

Catalog record

CREATORChenyang Yu, Xinpeng Xie, Yan Huang, and Chenxi Qiu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02094

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FlowDistill: Scalable Traffic Flow Prediction via Distillation from LLMs [Scalable Data Science] Chenyang Yu, Xinpeng Xie, Yan Huang, and Chenxi Qiu Abstract . Accurate traffic flow prediction is vital for optimizing urban mobility, yet it remains difficult in many cities due to complex spatio-temporal dependencies and limited high-quality data. While deep graph-based models demonstrate strong predictive power, their performance often comes at the cost of high computational overhead and substantial training data requirements, making them impractical for deployment in resource-constrained or data-scarce environments. We propose the FlowDistill , a lightweight and scalable traffic prediction framework based on knowledge distillation from large language models (LLMs). In this teacher-student setup, a fine-tuned LLM guides a compact multi-layer perceptron (MLP) student model using a novel combination of the information bottleneck principle and teacher-bounded regression loss, ensuring the distilled model retains only essential and transferable knowledge. Spatial and temporal correlations are explicitly encoded to enhance the model’s generalization across diverse urban settings. Despite its simplicity, FlowDistill consistently outperforms state-of-the-art models in prediction accuracy while requiring significantly less training data, and achieving lower memory usage and inference latency, highlighting its efficiency and suitability for real-world, scalable deployment. † † authors: . PVLDB Reference Format: PVLDB, 14(1): XXX-XXX, 2020. doi:XX.XX/XXX.XX † † This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing [email protected] . Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment. Proceedings of the VLDB Endowment, Vol. 14, No. 1 ISSN 2150-8097. doi:XX.XX/XXX.XX PVLDB Artifact Availability: The source code, data, and/or other artifacts have been made available at https://github.com/zxc2012/FlowDistill . 1. Introduction Traffic flow prediction serves as a foundation for urban transportation systems. It enables smarter urban planning, optimizes resource allocation, and improves traffic management and mobility services ( Zheng et al., 2020 ) . In the past few years, graph-based methods have become dominant in traffic flow prediction tasks. Models such as STSGCN ( Song et al., 2020 ) , ASTGCN ( Zhu et al., 2021 ) ,STWA ( Cirstea et al., 2022 ) and BigST ( Han et al., 2024a ) have demonstrated strong capabilities in capturing complex spatio-temporal dependencies. These methods leverage graph structures to model spatial relationships while effectively capturing temporal dynamics. However, as illustrated in Fig. 1 , which compares memory cost and training data requirements of different methods, graph-based methods (categorized as “Graph (w/o KD)”) depend heavily on sufficient training datasets to maintain high prediction accuracy. This limits their effectiveness ( Tang et al., 2024a ) , especially in limited training data scenarios ( Zhang et al., 2024 ) . Figure 1. The comparison of memory cost and training data required to achieve the same performance (i.e., \text{MAE}=7.22 on the NYC dataset). Note that STSGCN, ASTGCN, ST-MLP, and EasyST fail to reach \text{MAE}=7.22 even with the maximum training data proportion. The detailed explanation of the experimental results related to this figure can be found in Table 3 . Here, memory cost is represented by the bubble size. Meanwhile, the recent fast advances in large language models (LLMs) ( Achiam et al., 2023 ) have opened new opportunities for traffic flow prediction. Unlike graph-based methods, which rely on predefined spatial structures and abundant labeled data to learn relationships, LLMs benefit from a broad base of pre-trained knowledge gathered from large and diverse corpora ( Guo et al., 2024 ) . This foundational knowledge enables them to identify high-level patterns and contextual cues with minimal additional data, making LLMs particularly well-suited to traffic flow prediction tasks in the cities where training data is limited. Notably, existing models such as UrbanGPT ( Li et al., 2024b ) and LLM-COD ( Yu et al., 2024 ) have leveraged these strengths, demonstrating promising prediction accuracy even in scenarios with sparse or incomplete training datasets. Despite their strengths, LLMs, with their billions of parameters, require significant computational resources. This makes their deployment on mobile or edge devices particularly challenging due to the inherent resource constraints of such environments. Overcoming these limitations requires innovative strategies to make LLMs more efficient for spatio-temporal modeling ( Tang et al., 2024a ) . One potential approach to mitigate these challenges is knowledge distillation (KD) ( Hinton, 2015 ) , which transfers knowledge from a high-capacity teacher model to a lightweight student model to reduce computational demands while maintaining predictive accuracy. Recent studies, such as EasyST ( Tang et al., 2024a ) and ST-MLP ( Zhang et al., 2024 ) , have explored the application of KD in spatio-temporal tasks, achieving improvements in inference speed and scalability. However, these approaches primarily rely on graph-based teacher models and have yet to explore the use of KD for traffic flow prediction with LLMs, leaving significant potential untapped. As shown in Fig. 1 , both EasyST and ST-MLP (categorized as “Graph + MLP (KD)”) still need large amount of training datasets to ensure high prediction accuracy and STL-MLP requires substantially large memory size. 1.1. Our Contributions To bridge this gap, in this paper, we propose a new student-teacher framework, called FlowDistill , to enhance traffic flow prediction in scenarios with limited training data and computational resources. In this framework, a fine-tuned LLM, UrbanGPT ( Li et al., 2024b ) , acts as the teacher, and a lightweight multi-layer perceptron (MLP) serves as the student. By distilling rich contextual knowledge of LLM into a scalable model, this framework is optimized for resource-constrained environments, such as real-time traffic management systems in data-scarce cities and implementations on mobile or edge devices. Through KD, the MLP not only replicates the LLM’s reasoning capabilities but also achieves significant computational efficiency. By combining the strengths of LLMs and MLPs, this approach significantly reduces the reliance on extensive training data compared to traditional graph-based methods. For example, as shown by Fig. 1 , our FlowDistill model (categorized as “LLM + MLP (KD)”) achieves comparable performance (e.g., MAE = 7.22 on the NYC dataset) using only 10% of the training data, whereas the best-performing graph-based baseline requires 4 times more data (40%) to reach the same accuracy. This highlights a 75% reduction in data requirements. As a result, it sets a new benchmark for resource-efficient spatio-temporal modeling. Our key contributions are summarized as follows: • A Novel FlowDistill Framework: We propose an innovative teacher-student architecture that leverages a fine-tuned spatial temporal LLM as the teacher and a lightweight MLP as the student. This framework bridges the gap between the high generalization power of LLMs and the computational efficiency of MLPs, enabling efficient traffic flow prediction in resource-constrained and data-scarce environments. • Incorporation of Spatio-Temporal Autocorrelation: Our framework effectively captures spatial and temporal autocorrelations in traffic flow data. By aligning predictions with underlying spatio-temporal dependencies, the model enhances its generalization across diverse urban settings, addressing challenges posed by dynamic traffic patterns and regional variability. Our ablation study underscores the importance of modeling spatio-temporal autocorrelation, demonstrating that removing spatial correlation increases MAE from 6.54 to 7.01 and RMSE from 15.47 to 16.89, while removing temporal correlation leads to a more substantial rise in MAE to 8.01 and RMSE to 17.79. • Scaling with Accuracy:

Read the complete text