Unit Testing Past vs. Present: Examining LLMs’ Impact on Defect Detection and Efficiency
Rudolf Ramler, Philipp Straubinger, Reinhold Plösch, Dietmar Winkler · 2025
Read Unit Testing Past vs. Present: Examining LLMs’ Impact on Defect Detection and Efficiency by Rudolf Ramler, Philipp Straubinger, Reinhold Plösch, Dietmar Winkler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09801.txt.
Introduction
Rudolf Ramler and co-authors rerun a classic testing experiment with AI help in the room, in a 2025 paper. Unit testing asks coders to write small checks that catch planted faults in a program. Past rounds compared hand work with tool support. This round adds language models like ChatGPT and Copilot: participants wrote tests for a Java system seeded with defects, against the clock, with the models at hand.
The AI-backed testers wrote more tests than the hand group. They also caught more of the seeded faults, and finished the work more swiftly overall. The authors read real, measured gains, not just speed: the models lift both the net's width and its hit rate. The study offers rare hard evidence in a field full of vendor claims, drawn from a controlled rerun of a known design. It also hints where human judgment still steers the hunt.
Catalog record
| CREATOR | Rudolf Ramler, Philipp Straubinger, Reinhold Plösch, Dietmar Winkler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-09801
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unit Testing Past vs. Present: Examining LLMs’ Impact on Defect Detection and Efficiency Rudolf Ramler, Philipp Straubinger, Reinhold Plösch, Dietmar Winkler Abstract The integration of Large Language Models (LLMs), such as ChatGPT and GitHub Copilot, into software engineering workflows has shown potential to enhance productivity, particularly in software testing. This paper investigates whether LLM support improves defect detection effectiveness during unit testing. Building on prior studies comparing manual and tool-supported testing, we replicated and extended an experiment where participants wrote unit tests for a Java-based system with seeded defects within a time-boxed session, supported by LLMs. Comparing LLM supported and manual testing, results show that LLM support significantly increases the number of unit tests generated, defect detection rates, and overall testing efficiency. These findings highlight the potential of LLMs to improve testing and defect detection outcomes, providing empirical insights into their practical application in software testing. I Introduction Large Language Models (LLMs) in form of ChatGPT, Copilot, Gemini, etc. are now commonly used for everyday tasks. They are also increasingly integrated into software engineering workflows, offering new ways to enhance productivity and accuracy [ 1 ] . While their applications span diverse areas like requirements analysis, code generation, documentation, and debugging, their role in software testing—particularly unit testing—has garnered widespread interest. Recent studies suggest that LLMs can assist in generating test cases, improving coverage, and enhancing defect detection [ 2 ] . Most of these studies, however, focus on applying LLMs for automating the test case generation process, while in practice testers are interactively using LLMs to support all kind of tasks during unit testing. Thus, we aim to investigate the broad question: Do LLMs support human testers in finding defects in unit testing? In previous research, we compared tool-supported and manual unit testing approaches in controlled experiments [ 3 ] . Manual unit testing was evaluated through a time-constrained competition among students, who aimed to identify as many defects as possible in a Java-based system with seeded defects. The findings revealed that the participants were able to detect an average of 3.71 defects in 60 minutes testing, showing varying levels of efficiency. Subsequent replication of the experiment with professional developers demonstrated no significant differences in the results, validating the use of student participants as a proxy for practitioners [ 4 ] . Building on this foundation, we investigate the impact of LLMs on defect detection during unit testing by repeating and extending the original study. Our experiment evaluates whether integrating LLMs enhances participants’ ability to detect defects under similar time and resource constraints. By analyzing the created test cases and their defect detection performance, we aim to provide empirical evidence on the efficacy of LLM supported unit testing, where LLMs are used interactively in the loop of human testing activities. Our study with 30 participants shows that LLM-supported unit testing significantly outperforms manually writing unit tests in both productivity and defect detection. Participants using LLMs created more tests, achieved higher coverage, and identified more defects. However, the increase in test quantity also led to a higher rate of false positives, highlighting a trade-off between productivity and precision. Given that the original study was conducted over a decade ago, our comparison offers an interesting insight into how unit testing practices have evolved. After many years of relatively small and stable advancements, the advent of LLMs appears to have had the most significant impact on unit testing. II Experiment Design With the research objective outlined in the introduction, the experiment was conducted following the experimental design established in the original studies [ 3 ] and [ 4 ] . II-A Study Material The materials for the experiment include the system under test (SUT) with seeded defects from the original study [ 3 ] . Additionally, we created detailed instructions for the participants and a post-experiment questionnaire to complete after the testing task. The SUT consists of a Java collection class library similar to the standard Java Collections Framework. It comprises about 2,800 lines of code in 34 classes and interfaces with 164 methods, includes algorithmically complex implementations, and a variety of object-oriented features. Its familiarity to participants and prior successful uses in experimentation make it well-suited for our study. The SUT was provided as a folder containing a JAR archive with the compiled binary files of the collection classes to be tested, the corresponding Javadoc documentation for these classes, and a Maven test project for participants to create and execute their unit tests. The source code of the collection classes was not part of the SUT. The post-experiment questionnaire collected information on participants’ prior experience with unit testing and LLM-based tools. It also inquired whether LLM support was utilized during the experiment (optional) and, if so, which tools were used and how participants assessed their usefulness. II-B Participants The participants for the experiment were recruited from a master’s-level software testing course. As part of the course, students had gained both theoretical knowledge and practical experience in testing, including assignments that focused on writing unit tests with the JUnit framework. Additionally, many students had prior knowledge of unit testing and practical experience in software development. Participation in the experiment was voluntary, with all results anonymized for evaluation. Participants were given the option to exclude their data from the study. Of the 34 students who participated, 3 chose to opt out, leaving data from 31 participants available for analysis. II-C Execution The experiment was offered as an optional exercise within the software testing course, with all participants receiving extra points as an incentive. Each participant was required to complete the task individually within a 60-minute time limit. Participants received the following instructions: ”Write JUnit tests for the collection classes that represent the system under test, which contains defects. You may use LLM support or LLM-based tools of your choice for this task; however, their use is optional. Your goal is to identify as many defects as possible. For each defect found, provide a failing unit test that exposes the defect. At the end of the experiment, submit all tests you wrote—both failing and passing. The time allotted for writing tests is strictly limited to 60 minutes.” Upon completing the task, participants submitted a folder containing the source code of the unit tests they created, along with the completed questionnaire. II-D Analysis and Evaluation All submissions were anonymized, verified for completeness and consistency, and prepared for automated processing. Data from participants who opted out was excluded from the evaluation. Tests that failed to compile were commented out and excluded from further analysis. To identify false positives—tests failing despite no seeded defects being present—all tests were executed against the clean, defect-free version of the SUT. False positive tests were excluded from subsequent runs using the JUnit annotation @Disable . Prior to exclusion, each failing test was manually inspected by a researcher to confirm its classification as a false positive. For instance, tests were marked as false positives when their assertions contradicted the documented behavior in the associated Javadoc. A total of 158 failing tests were inspected, during which three instances revealed a new, previously unknown defect in the SUT. This newly discovered defect was included in the defect counts for subsequent analysis. To evaluate defect