AirQualityBench: A Realistic Evaluation Benchmark for Global Air Quality Forecasting

Xing Xu 1,2 Xu Wang 1,2,* Yudong Zhang 1,2 Huilin Zhao 3 Zhengyang Zhou 1,2 Yang Wang 1,2,* 1 University of Science and Technology of China (USTC), Hefei, Anhui, China 2 Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, China 3 The Hong Kong Polytechnic University, Hong Kong, China [email protected] · 2026

Read AirQualityBench: A Realistic Evaluation Benchmark for Global Air Quality Forecasting by Xing Xu 1,2 Xu Wang 1,2,* Yudong Zhang 1,2 Huilin Zhao 3 Zhengyang Zhou 1,2 Yang Wang 1,2,* 1 University of Science and Technology of China (USTC), Hefei, Anhui, China 2 Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, China 3 The Hong Kong Polytechnic University, Hong Kong, China [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05854.txt.

Introduction

Air quality models are usually graded on cleaned local datasets that hide how messy real monitoring is. Xing Xu and five co-authors build AirQualityBench to grade them as they are. The benchmark holds hourly readings from 3,720 monitoring stations worldwide over 2021 to 2025, covering six major pollutants. The data come as the networks recorded them, gaps and all.

Missing data stays missing: the stations' own observation masks are kept rather than filled in. Coverage is uneven, scales differ by pollutant, and running cost counts. Forecasters must cope with the network as it is, making the benchmark a real-world rehearsal rather than a lab exercise.

Catalog record

CREATORXing Xu 1,2 Xu Wang 1,2,* Yudong Zhang 1,2 Huilin Zhao 3 Zhengyang Zhou 1,2 Yang Wang 1,2,* 1 University of Science and Technology of China (USTC), Hefei, Anhui, China 2 Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, China 3 The Hong Kong Polytechnic University, Hong Kong, China [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05854

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AirQualityBench: A Realistic Evaluation Benchmark for Global Air Quality Forecasting Xing Xu 1,2 Xu Wang 1,2,* Yudong Zhang 1,2 Huilin Zhao 3 Zhengyang Zhou 1,2 Yang Wang 1,2,* 1 University of Science and Technology of China (USTC), Hefei, Anhui, China 2 Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, China 3 The Hong Kong Polytechnic University, Hong Kong, China [email protected] Abstract Air-quality forecasting models are commonly evaluated on regional, preprocessed, and normalized datasets, where missing observations are removed or artificially completed. Such protocols simplify comparison but hide the conditions that dominate real monitoring networks: uneven global coverage, structured missingness, heterogeneous pollutant scales, and deployment cost. We introduce AirQualityBench , a global multi-pollutant benchmark designed to evaluate forecasting models under these realistic conditions. The benchmark contains hourly observations from 3,720 monitoring stations over 2021–2025, covers six major pollutants, and preserves provider-native observation masks. Rather than imputing a dense data tensor, AirQualityBench exposes missingness as part of the forecasting problem and reports errors on valid future observations after inverse transformation to physical concentration scales. Evaluating representative spatio-temporal models under this unified protocol shows that strong performance on sanitized datasets does not reliably transfer to global, fragmented monitoring streams. AirQualityBench therefore serves as a realistic testbed for scalable, mask-aware, and physically interpretable air-quality forecasting. All benchmark data, code, evaluation scripts, and baseline implementations are available at GitHub . 1 Introduction Air-quality forecasting is a challenging spatio-temporal prediction problem because pollutant dynamics are shaped by regional transport, meteorology, emissions, and heterogeneous monitoring coverage. [ 3 , 4 , 17 , 2 ] Recent deep spatio-temporal models have achieved strong results on curated regional datasets, but their progress depends critically on whether the underlying benchmarks reflect real monitoring conditions. The rapid evolution of these predictive architectures relies heavily on rigorous benchmarks, which serve as the "yardstick" for measuring algorithmic progress and ensuring reproducibility [ 13 ] . High-quality air quality benchmarks are particularly vital as they provide the standardized environment necessary to validate a model’s ability to generalize across diverse urban topologies and varying climatic conditions. In the field of spatio-temporal learning, established datasets have historically catalyzed breakthroughs by providing a common ground for comparing disparate neural architectures, such as Graph Convolutional Networks (GCNs) and Transformers, thereby bridging the gap between theoretical research and real-world deployment. Despite their value, existing public air-quality forecasting benchmarks often simplify several aspects that are central to deployment-oriented evaluation. First , existing benchmarks [ 21 , 14 , 20 ] are geographically confined to single cities or specific administrative regions with limited node scales, which effectively severs the planetary-scale transport patterns of pollutants and prevents models from validating their scalability in large-scale networks. Second , these benchmarks frequently employ manual interventions to construct "sanitized" data tensors, such as linear interpolation or mean imputation. This deviates from the physical reality of Missing Not At Random (MNAR) patterns [ 19 ] , leading to a pervasive synthetic bias that fails to reflect model robustness against fragmented, real-world observation streams. Third , the prevailing reliance on unitless, normalized evaluation metrics obscures the physical interpretability and toxicological significance of prediction errors [ 11 , 8 ] , creating a profound gap between laboratory experiments and the requirements of environmental regulatory deployment. We argue that the central obstacle is not only the lack of a larger air-quality dataset, but the lack of a realistic evaluation protocol. Existing benchmarks often simplify the monitoring process into dense or partially completed tensors and report errors on normalized scales. While useful for controlled experiments, this setting removes exactly the factors that determine whether a forecasting model can be deployed in real sensing networks: missing observations, non-uniform spatial coverage, pollutant-specific scales, and computational cost. AirQualityBench is designed around these factors. It evaluates whether models can forecast from fragmented multi-pollutant histories, remain robust across uneven global monitoring networks, and produce errors that are interpretable in physical concentration units. To support realistic and reproducible evaluation of air-quality forecasting models, we introduce AirQualityBench , a global multi-pollutant benchmark with authentic missingness and physical-scale reporting (Fig. 1 ). Our contributions are threefold: • Global multi-pollutant benchmark. AirQualityBench contains hourly observations from 3,720 resident monitoring stations over 2021–2025, covering PM 2.5 , PM 10 , O 3 , NO 2 , SO 2 , and CO. It provides a geographically diverse testbed for evaluating scalability and cross-region heterogeneity beyond regional benchmarks. • Mask-aware physical-scale evaluation. We preserve provider-native observation masks instead of constructing fully imputed tensors, and evaluate predictions only on valid future measurements. Errors are reported after inverse transformation to pollutant-specific concentration scales, enabling physically interpretable metric. • Unified baselines and diagnostics. We evaluate representative spatio-temporal forecasting models under the same split, masking protocol, graph construction, and scripts, revealing their behavior under pollutant-specific sparsity, non-uniform station coverage, and accuracy–efficiency trade-offs. Figure 1: Overview of AirQualityBench. The benchmark combines a global network of 3,720 monitoring stations, synchronized observations of six pollutants, authentic missingness patterns, and physical-scale evaluation, providing a realistic testbed for large-scale spatio-temporal air quality forecasting. 2 Problem Definition and Related Work In this section, we formally define the masked spatio-temporal forecasting task and review the relevant literature in deep spatio-temporal learning and air quality modeling. 2.1 Problem Definition Task Formalization. We represent the global monitoring network as a spatio-temporal graph \mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{A}) , where \mathcal{V} denotes the set of N=3,720 stations and \mathbf{A}\in\mathbb{R}^{N\times N} is the adjacency matrix. At each time step t , the concentrations of C=6 pollutants and their corresponding availability are represented by an observation tensor \mathbf{X}_{t}\in\mathbb{R}^{N\times C} and a binary mask tensor \mathbf{M}_{t}\in\{0,1\}^{N\times C} . Given a historical window T , the goal is to learn a mapping f_{\theta} with the following input-output configuration: f_{\theta}:\underbrace{\mathbb{R}^{T\times N\times C}\times\{0,1\}^{T\times N\times C}}_{\text{Historical Observations \& Masks}}\longrightarrow\underbrace{\mathbb{R}^{T^{\prime}\times N\times C}}_{\text{Future Predictions}} (1) where T^{\prime} is the forecasting horizon. This formulation forces the model to reason directly from fragmented historical streams to generate complete future trajectories. Physical-Scale Evaluation. AirQualityBench evaluates predictions after inverse transformation to the provider-native concentration scale. We report masked MAE, MSE, RMSE, and MAPE on valid future observations. Since the benchmark contains heterogeneous pollutants and provider-specific unit conventions, pollutant-wise metrics are the primary physically interpretable results. Cross-pollutant global aggregates are provided only as benchmark-level ranking summaries and should not be interpreted as a single physically homogeneous concentration error. 2.2 Related Work Deep spatio-temporal forecasting. Deep spatio-temporal forecasting has been widely studied in networked sensing systems, including traffic, mobility, and environmental monitoring. Representative models range from graph-recurrent and graph-convolutional methods, such as DCRNN [ 10 ] , STGCN [ 24 ] , and GWN [ 22 ] , to attention-based and Transformer-style architectures, such as ASTGCN [ 6 ] , STTN [ 23 ] , and PDFormer [ 9 ] . Recent methods further explore adaptive graph learning, decoupled spatial–temporal dynamics, scalable temporal modeling, and event-aware forecasting, including AGCRN [ 1 ] , D 2 STGNN [ 18 ] , MAGE [ 15 ] , BiST [ 16 ] , and IGSTGNN [ 5 ] . For air-quality forecasting, models such as AirFormer [ 12 ] and PCDCNet [ 20 ] further incorporate meteorological information, geographical priors, or physical–chemical constraints. Despite these advances, most evaluations remain tied to regional, partially completed, or normalized datasets, leaving

Read the complete text