On the Flakiness of LLM-Generated Tests for Industrial and Open-Source Database Management Systems Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-SEIP

Alexander Berndt, Thomas Bach, Rainer Gemulla, Marcus Kessel, Sebastian Baltes · 2026

Read On the Flakiness of LLM-Generated Tests for Industrial and Open-Source Database Management Systems Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-SEIP by Alexander Berndt, Thomas Bach, Rainer Gemulla, Marcus Kessel, Sebastian Baltes (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08998.txt.

Introduction

Alexander Berndt and four co-authors study flaky tests made by chat models for large database tools. A flaky test passes at one run and fails at the next, with no code change. Such tests waste staff time, since a fail may mean no bug at all. Labs now use chat models to write tests fast, but the flake rate of those tests was not known.

The team runs model made tests on both firm tools and open tools. They track how oft flakes show, and what makes them show. Causes span time waits, test order, and shared state. A reader who uses AI test help gets hard rates, not fears. The work shows where a human must check model tests with most care.

Catalog record

CREATORAlexander Berndt, Thomas Bach, Rainer Gemulla, Marcus Kessel, Sebastian Baltes
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08998

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On the Flakiness of LLM-Generated Tests for Industrial and Open-Source Database Management Systems Conference: 2026 IEEE/ACM 48th International Conference on Software Engineering; April 12–18, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE-SEIP ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3786583.3786919 ISBN: 979-8-4007-2426-8/2026/04 CCS: Software and its engineering Empirical software validation CCS: Software and its engineering Software evolution CCS: Software and its engineering Development frameworks and environments CCS: Software and its engineering Software reliability Alexander Berndt, Thomas Bach, Rainer Gemulla, Marcus Kessel, Sebastian Baltes Abstract . Flaky tests are a common problem in software testing. They produce inconsistent results when executed multiple times on the same code, invalidating the assumption that a test failure indicates a software defect. Recent work on LLM-based test generation has identified flakiness as a potential problem with generated tests. However, its prevalence and underlying causes are unclear. We examined the flakiness of LLM-generated tests in the context of four relational database management systems: SAP HANA, DuckDB, MySQL, and SQLite. We amplified test suites with two LLMs, GPT-4o and Mistral-Large-Instruct-2407, to assess the flakiness of the generated test cases. Our results suggest that generated tests have a slightly higher proportion of flaky tests compared to existing tests. Based on a manual inspection, we found that the most common root cause of flakiness was the reliance of a test on a certain order that is not guaranteed (“unordered collection”), which was present in 72 of 115 flaky tests (63%). Furthermore, both LLMs transferred the flakiness from the existing tests to the newly generated tests via the provided prompt context. Our experiments suggest that flakiness transfer is more prevalent in closed-source systems such as SAP HANA than in open-source systems. Our study informs developers on what types of flakiness to expect from LLM-generated tests. It also highlights the importance of providing LLMs with tailored context when employing LLMs for test generation. † † cc-license: by 1. Introduction Large language models (LLMs) have emerged as a promising tool to automate test generation ( 70 ) . Although traditional test generation approaches, such as search-based techniques, have been shown to produce high-coverage test suites ( 58 ; 34 ) , the resulting test code suffers from poor readability and does not integrate well with existing tests ( 59 ; 24 ; 16 ) . In contrast, LLMs have the ability to generate test code that appears natural to humans ( 24 ; 60 ; 39 ) . Hence, various LLM-based test generation approaches have been evaluated ( 29 ; 70 ) , leading to industrial adoption of such approaches ( 37 ; 5 ) . To evaluate the resulting tests, existing studies on LLM-based test generation typically measure the effectiveness of LLM-generated tests based on metrics such as code coverage and the mutation score, that is, quantifiable signals regarding the quality of the system under test ( 70 ; 22 ) . However, to our knowledge, there exists no dedicated research that focuses on the flakiness of LLM-generated tests. Test flakiness is a major problem in industrial software testing, which has been studied by large software companies such as Google, Meta, Microsoft, and SAP ( 36 ; 40 ; 46 ; 14 ) . Flaky tests produce inconsistent results when executed multiple times under the same conditions ( 50 ) . Such flaky behavior is undesirable as it reduces the efficiency and effectiveness of testing, thus leading to increased testing costs ( 53 ; 36 ; 12 ) . Flaky tests invalidate the assumption that a test failure indicates the presence of a defect in the system under test. Thus, developers may lose trust in the test suite and waste time debugging spurious failures ( 28 ) . In continuous integration (CI) pipelines, flaky tests impede the automated assessment of code changes. For example, Google reported that the CI pipeline in their testing infrastructure fails in 3.5% of runs due to flaky failures ( 40 ) . Previous research on LLM-based test generation indicates that LLMs might produce flaky tests ( 64 ; 5 ) . However, the prevalence of flaky tests and the root causes of flakiness remain unclear. As a practical solution, previous work suggested repeatedly executing LLM-generated tests to filter flaky tests before handing them to developers for review ( 6 ) . Although this strategy can be helpful to filter out flaky tests, we consider it important to obtain additional information and a more holistic view of how and why LLM-generated tests are flaky ( 14 ) . Gaining insight into the flaky tests that LLMs produce can help practitioners set realistic expectations about the types of flakiness they may encounter when reviewing and using LLM-generated test code ( 28 ; 33 ) . Furthermore, comparing the prevalence of flakiness in LLM-generated tests with existing tests can help project the long-term flakiness rate of a test suite when LLMs are increasingly used to automatically generate tests. In this work, we have evaluated the flakiness of LLM-generated tests for relational database management systems. Previous research in the context of the large industrial database system SAP HANA has identified test flakiness as a major issue for automated testing ( 14 ; 15 ; 12 ; 8 ) . Motivated by this previous work, we focused on test generation for SAP HANA. In addition, we added three popular open-source databases to foster reproducibility and enable comparisons between open-source and closed-source software ( 11 ) . We consider this comparison valuable, as the source code of SAP HANA was not included in the training data, which might impact the results of our experiments. We adapted Alshawan et al.’s approach for LLM-based test generation, which was previously applied in an industrial study at Meta ( 5 ) , to amplify tests in the four target systems and generate new test cases. Our study focused on two types of tests: i) native unit tests written in C ++ , and ii) SQL system tests. We generated new tests for more than 5000 test files using two LLMs, GPT-4o and Mistral-Large-Instruct-2407 ( 68 ; 56 ) . In accordance with previous empirical studies on flakiness ( 61 ; 14 ; 33 ; 63 ) , we compiled and repeatedly executed the tests to obtain information on their flakiness. We compared the prevalence and root causes of the resulting flaky tests based on a manual inspection of more than 100 samples. Our results suggested that LLMs struggle to produce compilable C ++ code, especially in the closed-source context of SAP HANA. We attribute this finding to the model’s limited understanding of the provided code bases. In LLM-generated tests, the prevalence of flaky tests was often higher than in existing tests. Based on our manual inspection, we identified the reliance of tests on ordered test results from a collection that does not guarantee an order (“unordered collection”) as the most common root cause of flakiness in LLM-generated tests for database systems. To test the robustness of LLMs against flakiness transfer from existing tests, we injected a flaky assertion into the original test code before providing it as input for generating additional tests. In the generated tests, we found

Read the complete text