Where did we fail? — Reproducing build failures in embedded open source software Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 10–13 June, 2026; Glasgow, Scotland, United Kingdom CCS: Software and its engineering Software maintenance tools CCS: Software and its engineering Empirical software validation CCS: Software and its engineering Software testing and debugging Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 10–13 June, 2026; Glasgow, Scotland, United Kingdom The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE ’26), 10–13 June, 2026, Glasgow, Scotland, United Kingdom Conference: Evaluation and Assessment in Software Engineering (EASE) 2026; June 9-12; Glasgow, United Kingdom

Han Fu, Andreas Ermedahl, Sigrid Eldh, Kristian Wiklund, Philipp Haller, Cyrille Artho · 2026

Read Where did we fail? — Reproducing build failures in embedded open source software Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 10–13 June, 2026; Glasgow, Scotland, United Kingdom CCS: Software and its engineering Software maintenance tools CCS: Software and its engineering Empirical software validation CCS: Software and its engineering Software testing and debugging Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 10–13 June, 2026; Glasgow, Scotland, United Kingdom The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE ’26), 10–13 June, 2026, Glasgow, Scotland, United Kingdom Conference: Evaluation and Assessment in Software Engineering (EASE) 2026; June 9-12; Glasgow, United Kingdom by Han Fu, Andreas Ermedahl, Sigrid Eldh, Kristian Wiklund, Philipp Haller, Cyrille Artho (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27075.txt.

Introduction

Fu and five colleagues present PhantomRun, a unified layer and dataset for retrieval, storage, and reproduction of CI build logs and metadata. Embedded builds often fail from cross-compilation, board setups, and toolchain limits, while logs vanish quickly and resist reuse across varied runners and formats. Across 4628 failing runs, PhantomRun rebuilt 91.8 percent of builds and kept outcomes in 98 percent of cases. Rebuilt builds match originals closely, differing only in timestamps or minor reordering.

Catalog record

CREATORHan Fu, Andreas Ermedahl, Sigrid Eldh, Kristian Wiklund, Philipp Haller, Cyrille Artho
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40783
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27075

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Where did we fail? — Reproducing build failures in embedded open source software Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 10–13 June, 2026; Glasgow, Scotland, United Kingdom CCS: Software and its engineering Software maintenance tools CCS: Software and its engineering Empirical software validation CCS: Software and its engineering Software testing and debugging Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 10–13 June, 2026; Glasgow, Scotland, United Kingdom The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE ’26), 10–13 June, 2026, Glasgow, Scotland, United Kingdom Conference: Evaluation and Assessment in Software Engineering (EASE) 2026; June 9-12; Glasgow, United Kingdom Han Fu, Andreas Ermedahl, Sigrid Eldh, Kristian Wiklund, Philipp Haller, Cyrille Artho Abstract . Due to hardware–software co-development in embedded systems, continuous integration (CI) builds frequently fail because of complex cross-compilation, board configurations, and toolchain constraints. Although CI build logs contain valuable diagnostic information, they are short-lived and difficult to reuse due to heterogeneous runners, toolchains, and log formats. To address these challenges, we present PhantomRun , a unified abstraction layer and publicly reusable dataset that standardizes the retrieval, storage, and reproduction of CI build logs and metadata. Across 4628 4628 failing CI runs, we reconstructed 91.8 % of builds and preserved execution outcomes in 98 % of evaluated cases. PhantomRun provides two core capabilities: retrieving the build log of any commit and faithfully re-executing the corresponding build in a controlled environment. By exposing all build artifacts and metadata in a uniform, machine-readable format, PhantomRun enables reproducible and longitudinal studies of CI failures. An empirical evaluation shows that reproduced builds closely match their originals, typically differing only in timestamps or minor nondeterministic reordering, demonstrating the feasibility of large-scale historical CI reconstruction. † † cc-license: by 1. Introduction Continuous integration (CI) is a critical mechanism for maintaining build correctness in embedded software development, where hardware–software co-evolution, cross-compilation toolchains, and platform-specific configurations interact in complex and fragile ways ( 6 ) . CI pipelines orchestrate heterogeneous build environments, toolchains, and configuration layers, serving as the primary automated safeguard against integration regressions. Compilation failures represent a common and consequential category of CI breakage in embedded systems. They frequently arise from mismatched dependencies, evolving toolchains, configuration drift, and implicit environment assumptions that are difficult to reproduce locally. Such failures are costly ( 24 ; 20 ) : they block integration pipelines, delay feedback cycles, and prevent downstream testing and analysis. Because compilation failures halt the pipeline at its earliest stage, they act as an immediate indicator of environmental and configuration inconsistencies. In open-source embedded projects, CI build logs are vital for diagnosing compilation failures, yet they are often short-lived due to retention limits and heterogeneous build environments. Failing builds are especially valuable because they expose mismatches between evolving software components and target hardware configurations, and in our prior work, we show that such failures are highly informative for automated analysis and repair ( 7 ) . Consequently, this work builds on our prior work and focuses specifically on reconstructing failed CI builds to enable a durable, systematic study of compilation-related breakages across both software and hardware dimensions. Therefore, we present PhantomRun , a unified abstraction layer and publicly reusable dataset that standardizes the retrieval, storage, and reproduction of CI build logs and metadata. We mined over 10\,000 pull and merge requests from four widely used embedded projects—OpenIPC, STM32, RTEMS, and Zephyr 1 1 1 OpenIPC , STM32 , RTEMS , Zephyr —and reconstructed 4248 failing CI builds containing compilation errors in controlled containerized environments. Our evaluation shows that 91.8 % of failing builds are successfully reconstructed, and reproduced executions preserve termination outcomes and diagnostic structure with high fidelity, typically differing only in benign nondeterminism such as timestamps or minor log reordering. By transforming short-lived CI failures into durable and reproducible artifacts, PhantomRun enables systematic cross-project analysis and supports longitudinal, reproducible research on embedded CI behavior. Accordingly, we investigate the following research questions: RQ1: How embedded software CI builds be reconstructed outside their original CI environments? RQ2: How faithfully do reconstructed CI builds reproduce the compilation failures observed in the original CI executions? Our main contributions are: (1) Unified CI Reconstruction Framework. We introduce PhantomRun , a reproducible abstraction layer that enables historical CI builds to be retrieved, standardized, and re-executed outside their native CI infrastructures, achieving 91.8 % large-scale reconstruction success across heterogeneous embedded CI environments. (2) Large-Scale Reconstruction and Fidelity Evaluation. We reconstruct 4248 failing CI builds across four embedded projects and empirically quantify reconstructability and behavioral fidelity, preserving execution outcomes in 98 % of cases and structural log characteristics in 93.6 %. (3) Publicly Reusable Dataset for Reproducible CI Research. We release a systematically constructed dataset of reproduced CI build logs, metadata, and normalized diagnostics, enabling longitudinal and reproducible studies of compilation failures in embedded systems. The remainder of the paper is organized as follows. Section 2 introduces the context and motivation of our research and reviews related work on open-source software (OSS) CI reconstruction and log parsing. Section 3 outlines the PhantomRun design. Section 4 describes the methodologies for mining embedded system projects. Section 5 outlines research methodologies, while Section 6 presents our experimental results addressing RQ1 to RQ2. Section 7 discusses threats to validity, Section 8 discusses the results, Section 9 concludes the main findings, and Section 10 concludes with directions for future work. 2. Background and Related Work Continuous integration (CI) accelerates feedback on integration and configuration problems, but embedded CI is harder to stabilize because builds depend on cross-compilation toolchains, hardware-driven configuration layers, and rapidly evolving dependencies. As a result, compilation failures are common and particularly costly because they block later CI stages. Prior work shows that dependency-related issues account for a substantial share of CI failures across settings ( 6 ) , motivating approaches that can reproduce historical executions and analyze failures under controlled conditions. 2.1. CI reconstruction GitHub Actions (GHA) and GitLab CI are widely used CI platforms in both OSS and industrial infrastructure. 2 2 2 https://github.com/features/actions , https://docs.gitlab.com/ci/ Workflows are defined in YAML and specify runner environments, setup commands, and build steps. GHA workflows live under .github/workflows/ and integrate tightly with GitHub’s event model ( 10 ) , whereas GitLab CI uses .gitlab-ci.yml executed by GitLab runners with flexible deployment models ( 11 ) . These differences affect what environment details are explicit and therefore what can be reconstructed reliably. Recent research has explored replaying CI executions for reproducible studies, including workflow reconstruction and replay ( 12 ; 35 ) , mining workflow histories for recurring issues ( 31 ) , and replaying bug-fix scenarios to study CI reliability ( 27 ) . However, cross-platform reconstruction that targets embedded compilation behavior and supports structured, log-based comparisons remains limited, motivating our focus on unified reconstruction across GHA and GitLab CI. 2.2. Hardware-in-the-loop with dependency errors Embedded CI often interacts with hardware constraints (e.g., proprietary SDKs, device availability, or hardware-in-the-loop setups), which can make CI adoption and automation fragile ( 24 ; 20 ) . Dependency inconsistencies further amplify this fragility: mismatched toolchains, BSPs, middleware versions, and configuration assumptions frequently manifest as compilation failures ( 17 ; 6 ) . Related work highlights broader consequences such

Read the complete text