Design and Evaluation of a Scalable Data Pipeline for AI-Driven Air Quality Monitoring in Low-Resource Settings

Richard Sserunjogi * 1, Daniel Ogenrwot * 1,2, Nicholas Niwamanya 1, Noah Nsimbe 1, Martin Bbaale 1, Benjamin Ssempala 1, Noble Mutabazi 1, Raja Fidel Wabinyai 1, Deo Okure 1, Engineer Bainomugisha 1 · 2025

Read Design and Evaluation of a Scalable Data Pipeline for AI-Driven Air Quality Monitoring in Low-Resource Settings by Richard Sserunjogi * 1, Daniel Ogenrwot * 1,2, Nicholas Niwamanya 1, Noah Nsimbe 1, Martin Bbaale 1, Benjamin Ssempala 1, Noble Mutabazi 1, Raja Fidel Wabinyai 1, Deo Okure 1, Engineer Bainomugisha 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14451.txt.

Introduction

Air pollution data is only useful if it flows, and a 2025 paper by Richard Sserujongi and co-authors describes the pipeline behind an African monitoring network. Cheap sensors are spreading fast, but the regions that need them most often have weak power and patchy connections.

The AirQo pipeline is a modular, cloud-based system that gathers, cleans, and ships air quality data. It mixes streams from cheap sensors, weather services, and reference-grade monitors, and supports both live and batch work. Open tools handle scheduling, streaming, and storage. The system moves millions of readings each month from over four hundred devices, with low delay and strong uptime. It also automates calibration and forecasting for its users.

Catalog record

CREATORRichard Sserunjogi * 1, Daniel Ogenrwot * 1,2, Nicholas Niwamanya 1, Noah Nsimbe 1, Martin Bbaale 1, Benjamin Ssempala 1, Noble Mutabazi 1, Raja Fidel Wabinyai 1, Deo Okure 1, Engineer Bainomugisha 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14451

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Design and Evaluation of a Scalable Data Pipeline for AI-Driven Air Quality Monitoring in Low-Resource Settings Richard Sserunjogi * 1, Daniel Ogenrwot * 1,2, Nicholas Niwamanya 1, Noah Nsimbe 1, Martin Bbaale 1, Benjamin Ssempala 1, Noble Mutabazi 1, Raja Fidel Wabinyai 1, Deo Okure 1, Engineer Bainomugisha 1 Abstract The increasing adoption of low-cost environmental sensors and AI-enabled applications has accelerated the demand for scalable and resilient data infrastructures, particularly in data-scarce and resource-constrained regions. This paper presents the design, implementation, and evaluation of the AirQo data pipeline – a modular, cloud-native Extract-Transform-Load (ETL) system engineered to support both real-time and batch processing of heterogeneous air quality data across urban deployments in Africa. It is Built using open-source technologies such as Apache Airflow, Apache Kafka, and Google BigQuery. The pipeline integrates diverse data streams from low-cost sensors, third-party weather APIs, and reference-grade monitors to enable automated calibration, forecasting, and accessible analytics. We demonstrate the pipeline’s ability to ingest, transform, and distribute millions of air quality measurements monthly from over 400 monitoring devices while achieving low latency, high throughput, and robust data availability, even under constrained power and connectivity conditions. The paper details key architectural features, including workflow orchestration, decoupled ingestion layers, machine learning-driven sensor calibration, and observability frameworks. Performance is evaluated across operational metrics such as resource utilization, ingestion throughput, calibration accuracy, and data availability, offering practical insights into building sustainable environmental data platforms. By open-sourcing the platform and documenting deployment experiences, this work contributes a reusable blueprint for similar initiatives seeking to advance environmental intelligence through data engineering in low-resource settings. * * footnotetext: These authors contributed equally to this work and share first authorship. † † tocauthor: Richard Sserujongi, Daniel Ogenrwot, Nicholas Niwamanya, Noah Nsimbe, Martin Bbaale, Benjamin Ssempala, Noble Mutabazi, Raja Fidel Wabinyai, Deo Okure, and Engineer Bainomugisha † † institute: Makerere University, Plot 56 University Pool Road, Kampala, Uganda https://airqo.net and University of Nevada Las Vegas, Las Vegas, NV 89154, USA 1 Introduction The design and operation of scalable data pipelines are critical challenges in modern data engineering, particularly as organizations increasingly rely on heterogeneous data sources for analytics, decision-making, and the deployment of machine learning systems. Data pipelines are responsible for ingesting, transforming, and delivering structured, semi-structured, and unstructured data from diverse origins while ensuring reliability, scalability, and fault tolerance in production environment munappy2020data ; raj2020modelling ; lipovac2024developing . However, the complexity of maintaining consistent data quality and availability across varying data sources, network conditions, and operational environments presents significant engineering challenges cai2015challenges . Standard approaches to data ingestion and transformation often fall short when applied to resource-constrained environments, where data may arrive in varying formats and frequencies from distributed sources, including IoT devices, third-party APIs, and legacy systems operating over low bandwidth 2G GSM connectivity bainomugisha2023design ; munappy2020data . To address these challenges, there is a growing need for modular, extensible, and resilient ETL pipelines that leverage modern workflow orchestration tools, scalable data warehousing, and decoupled architectures to support real-time and batch processing workloads. Recent advances in Artificial Intelligence (AI), cloud-native infrastructure, and edge-based Internet of Things (IoT) systems have unlocked new capabilities for real-time monitoring and prediction in complex domains such as public health, transportation, and environmental management. In particular, air quality monitoring systems increasingly rely on AI for predictive modeling, automated sensor calibration and anomaly detection. However, real-world implementation of such intelligence demands robust and adaptive data infrastructure that can cope with the scale, velocity, and variability of environmental data, especially in low-resource settings characterized by unreliable power and intermittent internet connectivity. This paper presents the design and evaluation of a scalable, modular, and production-grade data pipeline that underpins the AirQo Platform , an AI-enabled air quality monitoring initiative with over 400 air quality monitors deployed across major African cities bainomugisha2024ai ; adong2022applying ; bainomugisha2023design ; okure2022characterization . AirQo data pipeline aggregates measurements from a range of heterogeneous sources, including proprietary low-cost sensors kumar2015rise , Beta Attenuation Mass (BAM) grade reference monitors, and third-party weather APIs, to provide high-resolution environmental intelligence sserunjogi2022seeing . The platform serves both scientific research and public engagement objectives, supporting dashboards, APIs, and machine learning-driven forecasts okure2025case . The AirQo data pipeline is built on open-source, cloud-native technologies: Apache Airflow orchestrates end-to-end workflows; Google BigQuery provides a scalable analytical warehouse; and Apache Kafka ensures real-time, decoupled streaming to downstream microservices. Through this architecture, the system seamlessly integrates both real-time and batch workflows, performs automated calibration, and feeds analytics and forecasts into public-facing and decision-support tools. The data pipeline is evaluated using both infrastructure-level and data-centric metrics: resource utilization, ingestion latency, throughput, data availability, and calibration success rate. These metrics provide insights into the trade-offs required to sustain performance under production workloads, while also meeting the reliability expectations of scientific and civic stakeholders. Our findings demonstrate how principled data engineering, grounded in modular design and open technologies, can drive sustainable innovation even under stringent constraints. This paper makes the following contributions • We present the design and implementation of a modular, scalable ETL pipeline capable of integrating heterogeneous data sources and supporting AI-driven workflows. • We present a detailed performance evaluation of the pipeline in production based on resource metrics, data quality, and operational reliability. • We articulate practical engineering lessons on observability, resilience, and architectural trade-offs for practitioners building real-world data infrastructure in constrained environments. • We provide the complete source code and documentation of the AirQo data pipeline and platform as an open-source resource to enable collaboration and extension of the pipeline for similar environmental monitoring applications. 2 Related Work In this section, we review prior research on scalable data pipelines, intelligent environmental data systems, and the unique challenges of deploying data infrastructure in low-resource settings. These related works provide critical context for our study, which lies at the intersection of modern data engineering and environmental sensing. Scalable Data Pipeline Architectures Modern data pipelines are designed to handle increasing volumes of diverse data in real-time and batch modes. The increasing demand for timely and actionable insights from large-scale data has led to the development of numerous frameworks and architectures for scalable data pipelines. Workflow orchestration platforms such as Apache Airflow apacheairflow ; harenslak2021data , Luigi luigi , and Prefect prefect have been employed to automate ETL workflows, enabling organizations to manage complex data dependencies while maintaining observability and fault tolerance. These tools provide the backbone for building modular and maintainable pipelines, supporting both batch and streaming data processing use cases. Cloud-based data warehousing solutions like Google BigQuery bigquery , Amazon Redshift redshift , and Snowflake snowflake have transformed the landscape of data storage and analytics, providing scalable and cost-efficient infrastructures capable of handling massive datasets with low latency. These platforms are essential components in modern data engineering architectures, allowing engineers to perform interactive analytics and support machine learning workflows at scale. Prior studies have explored scalable pipelines for IoT data iotpipelines , focusing on data ingestion and processing using cloud services for high-throughput scenarios. However, many existing approaches do not document the practical integration of heterogeneous sensor types, operational considerations in resource-constrained environments, and the end-to-end architecture required for consistent, reliable data delivery to downstream analytics and decision-making systems. Intelligent Environmental Data Integration and Systems Recent

Read the complete text