Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem

Tao Xiao, Dong Wang ✉, Shane McIntosh, Hideaki Hata, Yasutaka Kamei · 2026

Read Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem by Tao Xiao, Dong Wang ✉, Shane McIntosh, Hideaki Hata, Yasutaka Kamei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09311.txt.

Introduction

An empirical study from 2026, led by Tao Xiao with four co-authors, looks at flaky software tests across the OpenStack ecosystem rather than inside single projects. A flaky test is one whose result changes from run to run for no clear reason in the code itself, which weakens trust in test results and wastes computing effort. Earlier work had mostly studied flakiness project by project, leaving its wider effects unclear.

Drawing on hundreds of OpenStack projects, the study separates two patterns: tests that prove flaky in several projects at once, and tests that are flaky in some projects but steady in others. Cross-project flakiness turns out to be common, touching more than half of the projects examined, and even unit tests show it often. Race conditions, uneven build settings, and mismatched dependencies emerge as leading causes. The authors argue for better coordination, standard build setups, and stronger test isolation.

Catalog record

CREATORTao Xiao, Dong Wang ✉, Shane McIntosh, Hideaki Hata, Yasutaka Kamei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09311

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cross-Project Flakiness: A Case Study of the OpenStack Ecosystem Tao Xiao, Dong Wang ✉, Shane McIntosh, Hideaki Hata, Yasutaka Kamei Abstract Automated regression testing is a cornerstone of modern software development, often contributing directly to code review and Continuous Integration (CI). Yet some tests suffer from flakiness, where their outcomes vary non-deterministically. Flakiness erodes developer trust in test results, wastes computational resources, and undermines CI reliability. While prior research has examined test flakiness within individual projects, its broader ecosystem-wide impact remains largely unexplored. In this paper, we present an empirical study of test flakiness in the OpenStack ecosystem, which focuses on (1) cross-project flakiness, where flaky tests impact multiple projects, and (2) inconsistent flakiness, where a test exhibits flakiness in some projects but remains stable in others. By analyzing 649 OpenStack projects, we identify 1,535 cross-project flaky tests and 1,105 inconsistently flaky tests. We find that cross-project flakiness affects 55% of OpenStack projects and significantly increases both review time and computational costs. Surprisingly, 70% of unit tests exhibit cross-project flakiness, challenging the assumption that unit tests are inherently insulated from issues that span modules like integration and system-level tests. Through qualitative analysis, we observe that race conditions in CI, inconsistent build configurations, and dependency mismatches are the primary causes of inconsistent flakiness. These findings underline the need for better coordination across complex ecosystems, standardized CI configurations, and improved test isolation strategies. I Introduction Developers rely on automated regression testing to assess whether key functionality is perturbed by their changes. Ideally, each test failure should serve as a clear indication of problems in a change set; however, failures may occur due to non-deterministic factors. This behavior, known as test flakiness [ 33 ] , provides a misleading signal to developers regarding their changes. Flakiness arises when a test produces both passing and failing results when re-executed under identical conditions. Prior research has highlighted issues caused by test flakiness, such as wasted developer effort in investigating false positives [ 69 , 79 ] , reduced efficiency of Continuous Integration (CI) systems [ 14 , 29 ] , and the obfuscation of software defects [ 78 , 70 ] . Addressing test flakiness is a persistent challenge in software testing. For example, Google has documented that between 2% and 16% of its computational resources are allocated to re-execute flaky tests [ 40 ] . A survey of 58 Microsoft developers found that they ranked test flakiness as the second most important perceived barrier to software deployment [ 28 ] . The economic impact of test flakiness is also substantial. Herzig et al. [ 21 ] estimated that the cost of investigating test failures caused by flakiness at Microsoft amounts to approximately $7.2 million for a single product. Additionally, an analysis of GitHub commits revealed that “1 in 11 commits had at least one [failure] caused by a flaky test” [ 61 ] . Similar issues have been reported by Mozilla [ 7 ] and Huawei [ 24 ] , further indicating the widespread impact of test flakiness. Prior research has predominantly focused on flakiness at the programming level [ 59 ] . To the best of our knowledge, the cascading impact of flakiness at the ecosystem level, which encompasses a set of socio-technically interdependent software projects [ 35 , 38 , 23 ] , has not yet been explored. While individual projects within the ecosystem are managed by their respective project teams or organizations, the strength of the ecosystem stems from the social interactions among these teams (both within and across projects [ 39 , 12 ] ) as well as their ability to foster reuse functionalities through technical dependencies [ 1 , 6 , 13 , 26 ] . Flakiness in such an ecosystem presents unique challenges since it does not remain insulated within a single project. Instead, it propagates through shared dependencies and interfaces that multiple projects rely on. In this paper, we perform a case study of OpenStack, a renowned infrastructure-based ecosystem [ 19 ] developed by 50 participating industrial members, including large-scale and well-known companies, such as AMD, Microsoft, and Cisco. Through our preliminary exploration, we identify cross-project flakiness and inconsistent flakiness phenomena in the OpenStack ecosystem (detailed in Section III ). Cross-project flakiness occurs when a flaky test affects multiple projects. Inconsistent flakiness refers to instances where a test exhibits non-deterministic behavior in a subset of projects while remaining stable in others. While prior research [ 33 , 34 ] has explored test flakiness within individual projects, its characteristics and impact on multiple projects, particularly at the ecosystem level, remain largely unexplored. More specifically, we address the following three Research Questions (RQs): RQ1. To what extent does cross-project flakiness exist? Motivation. The prevalence of cross-project flakiness in the OpenStack ecosystem remains unclear, highlighting a critical knowledge gap given the limited resources available to address the issue. Therefore, we first set out to quantify the prevalence of cross-project flakiness. Providing a foundational understanding of its behavior for assisting team leads of software ecosystems, responsible for overseeing all technical aspects and representing contributors, in making informed decisions for effective intervention. Results. We find that cross-project flakiness impacts a large proportion (55%) of OpenStack projects, occurring with greater frequency than flakiness confined to individual projects. Moreover, cross-project flakiness has rapidly accumulated over time in OpenStack. These findings are a call to action for addressing cross-project flakiness to enhance CI reliability and reduce resource consumption. RQ2. How do test scopes influence the likelihood of cross-project flakiness? Motivations. Software testing can be categorized into various types to comprehensively validate the software, such as unit tests, functional tests, and integration tests. Each test type may interact with system dependencies and infrastructure in unique ways, which could potentially lead to different degrees of flakiness (e.g., number of projects being affected by the flaky test) in interconnected ecosystems like OpenStack. By exploring how these various test types contribute to cross-project flakiness, researchers and developers can pinpoint which test types are most vulnerable to inducing such instability. Consequently, team leads of software ecosystems could leverage these insights to refine existing test design strategies and proactively address identified flakiness phenomena. Results. We find that the two flakiness phenomena are prevalent in the OpenStack ecosystem, with 1,535 and 1,105 flaky tests presenting with cross-project flakiness and inconsistent flakiness, respectively. Unit tests are the most likely to exhibit flakiness, accounting for 7,202 flaky tests observed across 30,550 test executions. Moreover, we observe a strong correlation between test scopes (i.e., unit, integration, system, scenario, or API tests) and flakiness levels (i.e., the extent to which flakiness propagates across the network of projects). We find that 64% of API tests and 41% of scenario tests are prone to causing flakiness across multiple OpenStack projects. Surprisingly, cross-project flakiness affects 70% of unit tests, suggesting that even tests that are traditionally well-isolated can contribute to ecosystem-wide flakiness. These findings suggest that test strategies may benefit from refinement, especially in terms of test isolation and resource management, to mitigate flakiness at its root cause. RQ3. What are the causes of inconsistent flakiness? Motivations. In this RQ, we conduct an in-depth analysis to uncover the underlying causes of inconsistent flakiness, that is, why certain projects experience

Read the complete text