EvoGPT: Enhancing Test Suite Robustness via LLM-Based Generation and Genetic Optimization
Lior Broide, Roni Stern · 2025
Read EvoGPT: Enhancing Test Suite Robustness via LLM-Based Generation and Genetic Optimization by Lior Broide, Roni Stern (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12424.txt.
Introduction
Lior Broide and co-authors mix large language models with evolutionary search to write stronger unit tests for software. Their 2025 paper tackles a known flaw of search-based test writing: it stalls early and its test suites lack variety. Language models write tests in a fresh style, but they do not optimize a suite as a whole.
Their system, EvoGPT, seeds the search with suites drafted by a model. It forces variety by varying the model's heat and prompts, and it repairs each test in a loop while adding checks aimed at code coverage. When the search stalls, fresh model tests target the branches still missed. On a standard Java bug benchmark, the blend lifted both coverage and fault-finding scores by about ten percent.
Catalog record
| CREATOR | Lior Broide, Roni Stern |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-12424
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EvoGPT: Enhancing Test Suite Robustness via LLM-Based Generation and Genetic Optimization Lior Broide, Roni Stern Abstract Large Language Models (LLMs) have recently emerged as promising tools for automated unit test generation. We introduce a hybrid framework called EvoGPT that combines LLM-based test generation with more traditional evolutionary search techniques to produce diverse, bug-revealing unit tests. Unit tests are initially generated using diverse temperature sampling to enhance behavioral and test suites diversity, followed by a generation-repair loop and coverage-guided assertion generation. The improved test suites are subsequently evolved using genetic algorithms, with a fitness function that prioritizes mutation score over traditional coverage metrics. This design reflects the primary goal of unit testing — detecting faults. Evaluated across multiple open-source Java projects, our framework achieves an average improvement of 10% in both code coverage and mutation score, compared to LLM and traditional search-based software testing baselines. These results demonstrate that combining LLM-driven diversity, targeted repair, and search-based optimization yields more effective and resilient test suites. † † paperid: 123 1 Introduction Software testing is a fundamental practice in software engineering, critical for ensuring the reliability, correctness, and maintainability of software systems [ Beck(2022) ] . Among its various forms, unit testing serves as the first line of defense against bugs by verifying the behavior of individual program units in isolation. However, despite its importance, the process of manually designing high-quality unit tests remains labor-intensive and error-prone. Furthermore, the oracle problem i.e., the difficulty of defining the expected outcomes for arbitrary software behaviors, poses a persistent challenge, particularly when aiming to automate the generation of meaningful and verifiable test cases [ Weyuker(1982) ] . Search-Based Software Testing (SBST) is an extensively studies paradigm for automated test generation [ Harman(2007) ] [ Almulla and Gay(2022) ] . Techniques such as evolutionary algorithms, implemented in tools like EvoSuite [ Fraser and Arcuri(2013) ] , systematically optimize test suites toward structural coverage objectives (e.g., branch and line coverage). While SBST methods often achieve high coverage, they typically generate test cases that lack readability, semantic relevance, insufficient fault detection, and their optimization processes are computationally expensive [ Setiani et al.(2020)Setiani, Ferdiana, and Hartanto ] . In recent years, Large Language Models (LLMs) have emerged as promising tools for automating unit test generation [ Chen et al.(2021)Chen, Tworek, Jun, et al. ] . Approaches such as TestART [ Gu et al.(2024)Gu, Zhang, Fang, Tian, Zhu, Zhou, and Chen ] , ChatUniTest [ Chen et al.(2024)Chen, Hu, Zhi, Han, Deng, and Yin ] , and A3Test [ Alagarsamy et al.(2024)Alagarsamy, Tantithamthavorn, and Aleti ] demonstrate the ability of LLMs to produce syntactically correct, semantically coherent, and human-readable unit tests. These models leverage their natural language understanding and code generation capabilities to approximate human-level testing strategies. Nevertheless, LLM-based methods often suffer from limited exploration of complex input spaces, instability caused by hallucinations, and an inability to iteratively improve upon initial generations without external feedback [ Madaan et al.(2023)Madaan, Tandon, Gupta, Hallinan, Gao, Wiegreffe, Alon, Dziri, Prabhumoye, Yang, et al. ] . Motivated by the complementary strengths and weaknesses of LLM-based and SBST-based approaches, we propose a hybrid framework that we call EvoGPT, which integrates diverse LLM-based generation with evolutionary search-based refinement. Our framework first generates a large, diverse pool of initial test suites by querying LLMs under varied temperature settings, encouraging exploration across diverse execution paths that include repair loops, and coverage guidance. Subsequently, an evolutionary algorithm refines the generated test suite through selection , crossover and mutation operations, optimizing for both coverage mutation score. Empirical results show that our framework already significantly outperforms state-of-the-art LLM-only and SBST-only baselines in both code coverage and mutation score of successfully generated tests. Contributions The main contributions of this paper are: • Hybrid Test Generation Framework: a novel method for combining LLM-based test generation with evolutionary search to produce high-quality, high-coverage unit test suites. • Prompt- and Temperature-Driven Diversity: We employ temperature-based sampling and different system prompt instructions during LLM querying to encourage diversity among initial test suites, improving input space exploration. • Effective evolutionary Optimization: Our evolutionary loop achieves strong performance improvements, demonstrating the intrinsic effectiveness of hybridization. • Extensive Empirical Evaluation: We rigorously evaluate our approach on standard benchmarks, demonstrating substantial improvements in code coverage and mutation testing over strong baselines, namely EvoSuite [ Fraser and Arcuri(2013) ] and TestART [ Gu et al.(2024)Gu, Zhang, Fang, Tian, Zhu, Zhou, and Chen ] . 2 Background and Problem Definition A unit test is a self-contained function designed to validate the behavior of a specific method or unit of source code. As introduced in Test-Driven Development (TDD): ”Tests are used to specify expected behavior and detect regressions when code changes over time” [ Beck(2022) ] . Executing a test typically involves invoking the target method under specific inputs and comparing the observed outcome against expected outputs through assertions. The result of a test is often binary: a test either passes or fails , depending on whether all assertions hold. In the context of automated unit test generation, it is common to focus on testing a subset of methods within a class. Typically those that are externally visible and designed to be invoked directly. Following prior work [ Gu et al.(2024)Gu, Zhang, Fang, Tian, Zhu, Zhou, and Chen ] , we designate the public methods of each class under test as focal methods . These focal methods serve as the primary targets for test generation and evaluation, allowing the testing effort to concentrate on actionable and externally observable behavior. Evaluating the quality of a unit test involves several well-established metrics [ Fraser and Arcuri(2013) , Harman(2007) , Jia and Harman(2010) ] : • Line Coverage of Correct Tests (LCCT) : the proportion of source code lines in the focal methods that are executed at least once by a passing test. While high line coverage indicates broader code exploration, it does not guarantee semantic correctness. • Branch Coverage of Correct Tests (BCCT) : the proportion of conditional branches (e.g., if-else, switch cases) in the focal methods that are exercised in passed tests. This metric captures control-flow depth and is more sensitive to logical path diversity. • Mutation Score of Correct Tests (MSCT) : the percentage of artificially injected faults (mutants) in the focal methods that are detected, i.e., cause at least one test to fail. This is widely regarded as the strongest proxy for fault detection capability [ Fraser and Arcuri(2013) , Jia and Harman(2010) ] . The objective of an automated test generation algorithm is therefore to synthesize a suite of correct, diverse, and assertion rich tests that maximize test quality metrics, particularly LCCT, BCCT, and MSCT, while remaining readable and executable. Traditional test generation methods. Several techniques have been proposed to automatically generate test suites. Random test generation methods such as Randoop [ Pacheco and Ernst(2007) ] sample inputs and assertions without guided heuristics. While computationally inexpensive, they frequently produce shallow, brittle, or irrelevant tests that lack meaningful and relevant assertions, or practical utility. Search-Based Software Testing methods, such as EvoSuite [ Fraser and Arcuri(2013) ] applies evolutionary algorithms to optimize the aforementioned test quality metrics over multiple generations. It begins by generating an initial population of random test suites, which are then