Is this Build Failure Related to my Patch? An Empirical Study of Unrelated Build Failures in Continuous Integration
Yonghui (Andie) Huang · Daniel Alencar da Costa · Grant Dick · Mariam El Mezouar · 2026
Read Is this Build Failure Related to my Patch? An Empirical Study of Unrelated Build Failures in Continuous Integration by Yonghui (Andie) Huang · Daniel Alencar da Costa · Grant Dick · Mariam El Mezouar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05564.txt.
Introduction
A failed build in continuous integration is not always the fault of the latest code push. Andie Huang, Daniel Alencar da Costa, Grant Dick, and Mariam El Mezouar study such unrelated failures across seven Apache projects. Developers reported spending a median of four hours just deciding whether a failure was theirs to fix.
The team analyzed more than seventy-seven thousand build failures and hundreds of confirmed unrelated cases. Unrelated test failures made up a fifth of the confirmed set. Their prediction models, trained with semi-supervised learning, used features like CI latency and repeated error messages to flag likely unrelated failures.
Catalog record
| CREATOR | Yonghui (Andie) Huang · Daniel Alencar da Costa · Grant Dick · Mariam El Mezouar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 106 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05564
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Is this Build Failure Related to my Patch? An Empirical Study of Unrelated Build Failures in Continuous Integration Yonghui (Andie) Huang · Daniel Alencar da Costa · Grant Dick · Mariam El Mezouar Abstract In a hectic Continuous Integration (CI) environment, where several builds are triggered concurrently, legitimate build failures (e.g., not caused by flaky tests) may not always be related to the current push. These unrelated build failures can burden developers as they devote hours to attest whether errors are truly associated with their present changes. In this paper, we extract 77,354 CI build failures from 7 open source projects to understand and identify unrelated build failures. We attempt to provide an indication for developers about whether a build failure is likely to be related to the current push or not. Our results reveal that developers likely invest a median of 4 hours to determine whether a build failure is (un)related to their pushes. We perform a document analysis on a sample of 371 unrelated build failures (based on the 95% confidence level and 5% confidence interval from 10,316 potentially unrelated failures) to understand why build failures are deemed as unrelated by developers. The themes generated from our document analysis reveal that unrelated tests failures represent 20% of the cases of why build failures are deemed unrelated by developers. To predict whether a build failure is unrelated to the current push, we extract 33 features from issue reports, issue comments, and from the commits pertaining to the triggering push. We build semi-supervised PU-learning models over seven Apache projects and achieve precision ranging from 0.70 ± 0.01 0.70\pm 0.01 to 0.88 ± 0.02 0.88\pm 0.02 , recall ranging from 0.30 ± 0.03 0.30\pm 0.03 to 1.00 ± 0.00 1.00\pm 0.00 , and F1-scores ranging from 0.44 ± 0.03 0.44\pm 0.03 to 0.91 ± 0.00 0.91\pm 0.00 , while the area under the ROC curve (AUC) spans 0.63 ± 0.02 0.63\pm 0.02 to 0.97 ± 0.03 0.97\pm 0.03 . Our analysis of feature importance reveals that (i) the time taken from a submitted patch to the build-triggering push (CI latency), (ii) build failures sharing similar error messages with recent failures, and (iii) the number of comments preceding the build failure, are all efficient indicators for identifying potential unrelated build failures. The semi-supervised approach proposed in this work can help developers identify build failures that are unrelated to their current push, helping their decision-making process. ∎ 1 Introduction Continuous Integration (CI), a software development practice that involves the frequent integration of code changes, has gained traction in both industry and academia 22 ; 11 ; 72 ; 10 . Researchers have thoroughly investigated the potential benefits and costs of CI in the last years 75 . For instance, CI has empirically shown associations with increased productivity 5 ; 82 ; 70 despite potential challenges such as extra efforts required to address CI configuration issues 89 . According to Duvall et al. 11 , CI is a combination of several interrelated practices, including frequent check-ins (e.g., frequent Git Pushes ), short build cycles, a comprehensive test suite, and proper build feedback (or bells and whistles ). Regarding proper build feedback , it is important that development teams have visibility into when builds fail, who caused the failure, and the reason for failure. Indeed, Duvall et al. 11 emphasize the utmost importance of fixing a build failure before moving on to other tasks. This is because developers may relax their adherence to CI practices if they perceive that builds are constantly failing. For example, developers may choose to forgo thorough checks of their changes before pushing to the mainline (i.e., the build is always failing anyway). Notwithstanding the importance of fixing build failures, given the prevalence of distributed software development settings, where multiple pushes are made concurrently, it becomes increasingly challenging to fix build failures immediately. In fact, Ghaleb et al. 26 revealed a phenomenon called cascading build breakages , where a build fails not due to the changes of the current push, but due to changes inherited from a previous push that initially caused the build to fail. As such, an adverse effect of cascading build breakages is that developers often wonder whether their code changes are directly responsible for the build failure or if the failure is unrelated to their changes, possibly caused by another developer’s push. Much research has studied the challenges involved in adopting CI, including complexities around build notifications 16 ; 72 ; 90 . Elazhary et al. 16 studied three companies using CI and reported that they face “notification fatigue” due to frequent builds, while also requiring discipline to monitor build status. Zampetti et al. 90 studied CI practitioners to uncover bad patterns related to CI usage. Regarding the status of builds, the authors found that build notifications can often be verbose, containing unrelated information, while also containing the output of different build tasks, which can confuse developers when it comes to fixing build failures. While previous research has highlighted issues related to sub-optimal build notifications, our work is the first to address the problem of automatically identifying unrelated build failures. We define an unrelated build failure as a failure that occurs during the continuous integration (CI) process but is not caused by the current code push. We do not define “unrelated failures” as failures that are entirely unrelated to the codebase or project as a whole. Rather, we refer to failures that are unrelated to the most recent code push. More specifically, an “ unrelated build failure ” as one where (i) no error messages, stack traces, or test failures can be traced back to the source files modified in the associated code push, (ii) the root cause lies outside the submitted code changes before the merging stage, or (iii) follow-up developer comments explicitly confirm that the issue is unrelated to the submitted code. In practice, unrelated failures can lead to significant developer frustration and wasted effort, as teams may spend time investigating issues that are not actionable for their current change. If developers can be certain that a build failure is not related to their push, they can use this information to make decisions, such as proceeding with the integration or better debugging the build breakage. This is particularly crucial in distributed development settings, where different collaborators simultaneously contribute to the project, such as open-source projects or remotely distributed teams 45 ; 33 . It is important to distinguish unrelated build failures from flaky tests. Flaky tests 55 refer to automated tests that exhibit non-deterministic behavior, producing inconsistent results despite unchanged code. Unrelated build failures, on the other hand, pertain to legitimate failures in the build process caused by changes unrelated to the current code push. For example, configuration issues or unforeseen problems due to differences in environments, can contribute to build failures related to code pushes. As such, our goal in this study is to understand and potentially predict when unrelated build failures occur. We conduct seven project-level experiments of Apache projects, where developers often comment on the validity of build failures, exerting efforts to verify whether these failures are unrelated to the current push. To scale our study, we employ a keyword-detection approach to