Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript
Negar Hashemi 1 , Amjed Tahir 1 , Shawn Rasheed 2 , August Shi 3 , Rachel Blagojevic 1 · 2025
Read Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript by Negar Hashemi 1 , Amjed Tahir 1 , Shawn Rasheed 2 , August Shi 3 , Rachel Blagojevic 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12680.txt.
Introduction
Negar Hashemi and co-authors hunt order-dependent flaky tests in JavaScript projects. Their twenty twenty-five paper studies tests that pass or fail by luck of run order. Such flaky tests erode trust in the whole test suite.
The team built a tool that shuffles tests at three levels: single tests, suites, and describe blocks. Each project got ten orders, each run ten times over. Any flip in outcome was logged, then read by hand to find the cause. The focus is Jest, the most used JavaScript test frame. Results show how often shared state between tests is to blame and how teams can expose it before release.
Catalog record
| CREATOR | Negar Hashemi 1 , Amjed Tahir 1 , Shawn Rasheed 2 , August Shi 3 , Rachel Blagojevic 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12680
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript Negar Hashemi 1 , Amjed Tahir 1 , Shawn Rasheed 2 , August Shi 3 , Rachel Blagojevic 1 Abstract Flaky tests pose a significant issue for software testing. A test with a non-deterministic outcome may undermine the reliability of the testing process, making tests untrustworthy. Previous research has identified test order dependency as one of the most prevalent causes of flakiness, particularly in Java and Python. However, little is known about test order dependency in JavaScript tests. This paper aims to investigate test order dependency in JavaScript projects that use Jest, a widely used JavaScript testing framework. We implemented a systematic approach to randomise tests, test suites and describe blocks and produced 10 unique test reorders for each level. We reran each order 10 times (100 reruns for each test suite/project) and recorded any changes in test outcomes. We then manually analysed each case that showed flaky outcomes to determine the cause of flakiness. We examined our detection approach on a dataset of 81 projects obtained from GitHub. Our results revealed 55 order-dependent tests across 10 projects. Most order-dependent tests (52) occurred between tests, while the remaining three occurred between describe blocks. Those order-dependent tests are caused by either shared files (13) or shared mocking state (42) between tests. While sharing files is a known cause of order-dependent tests in other languages, our results underline a new cause (shared mocking state) that was not reported previously. I Introduction Flaky tests (i.e., tests with non-deterministic outcomes) may pass on some runs and fail on others on the same version of the code. Flaky tests are known to negatively impact product quality and delivery [ 1 , 2 , 3 ] , as well as testing-related activities [ 4 , 5 ] . A flaky test that fails may not indicate any bug in the code, and developers waste time trying to identify non-existent bugs. Flaky tests can then lead developers to lose trust in their builds and ignore test failures [ 6 ] , resulting in additional costs for software projects. Indeed, flaky tests present a significant challenge in the software industry [ 7 , 1 , 3 , 8 ] . A report from Microsoft noted that flaky tests could cost up to $7.2 million per year for Microsoft Dynamics product line [ 9 ] . GitHub also reported that 9% of their commits had at least one failed build caused by flaky tests [ 10 ] . Previous studies have identified and categorized potential causes of flaky tests across different programming languages, including Java [ 11 ] , Python [ 12 ] , and JavaScript [ 13 ] . Luo et al. [ 11 ] reported the first extensive empirical study on flaky tests in open-source software, and they categorized the causes and fixing strategies for these flaky tests. They found that one of the major causes of flaky tests is test order dependency—that is, the tests’ outcomes depend on the order in which they are executed. Flaky tests that pass in one order but fail in another are known as order-dependent tests . While test order dependency is acknowledged as one of the top causes of test flakiness in several languages, including Java and Python [ 11 , 6 , 14 , 12 ] , the number of order-dependent tests reported in JavaScript projects is low [ 13 ] . However, these findings were based on the reported flaky tests in open-source projects (i.e., extracted from issue trackers). The projects may contain order-dependent tests that the developers have yet to discover, potentially failing and misleading developers later during development. Indeed, the most popular JavaScript testing framework, Jest [ 15 , 16 ] , introduced a feature that allows developers to randomize the order in which they run Jest tests due to concerns from developers about the potential of having order-dependent tests [ 17 ] . This feature is still relatively new (made available in late 2023) and not widely adopted, meaning many projects may still contain order-dependent tests that developers are unaware of. In this study, we conduct an in-depth investigation into order-dependent tests in JavaScript, focusing on the Jest testing framework. Jest offers many features, such as parallelization and ordered test execution, but tests may still share state between each other, leading to order-dependent tests. Our goal is to understand the prevalence of order-dependent tests within popular open-source JavaScript projects after actively searching for them. We also want to understand the reasons for their failures, i.e., what state these tests share between them. The results of our study can provide better insights towards how developers can manage order-dependent tests in their test suites, debugging their root causes to eventually fix them so they no longer have different test outcomes in different orders. The main contributions of this paper are the answers to the following two research questions: • RQ1 How can order-dependent tests manifest their failures in JavaScript projects? • RQ2 What are the main causes of order-dependent tests? RQ1 aims to investigate ways that order-dependent tests’ failures can manifest in Jest. We employed a multi-layer reorder and rerun approach to uncover order-dependent tests in Jest. By altering the sequence in which the tests are executed within programs/test suites and rerunning them multiple times, we observe whether the test outcomes change. RQ2 aims to understand the main causes of order-dependent tests in Jest. We manually inspected each detected order-dependent test and classified the cause of flakiness into different categories. To answer our research questions, we developed a tool called JS-TOD , which extracts, reorders, and reruns tests and test suites using user-defined numbers of reorders and reruns. We evaluated on a dataset of 81 projects. Using JS-TOD , we reordered tests and test suites 10 times each and rerun them 10 times each. Our results show that order-dependent tests are possible despite Jest executing tests sequentially (which is supposed to guarantee the default test order). Of the 55 detected order-dependent tests, 52 were found between individual tests, and three were detected between describe blocks within the same test suite rather than between different test suites. We found that the main causes of those flaky tests are shared files (e.g., access to a shared file created in one of the tests and accessed by other tests) and shared mocking state (e.g., some mocked state was changed by tests). II Background II-A Test Order-Dependency Ideally, tests should function independently of each other and produce the same results regardless of their order of execution [ 18 ] . In reality, tests often have different outcomes when executed in different orders [ 5 , 18 ] . These tests are called order-dependent tests , and previous studies found them to form a significant proportion of flaky tests [ 11 , 12 ] . Prior work classified order-dependent tests into one of two categories based on their result when run in isolation: victim or brittle [ 19 ] . A victim passes when run in isolation but fails when run after another test(s), known as a polluter . Essentially, a polluter “pollutes” the state shared between the two tests, leading the victim to