Evaluating LLM-Based Test Generation Under Software Evolution

Sabaat Haroon, Mohammad Taha Khan, Muhammad Ali Gulzar · 2026

Read Evaluating LLM-Based Test Generation Under Software Evolution by Sabaat Haroon, Mohammad Taha Khan, Muhammad Ali Gulzar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23443.txt.

Introduction

Haroon, Khan, and Gulzar test whether language models truly reason when they write unit tests. The team tracks generated tests across 22,374 changed program versions and eight models. On clean code, models hit 79% line coverage with fully passing suites. After meaning-changing edits, new tests pass only 66% of the time. Over 99% of those failures pass on the old code while running the changed region. Meaning-preserving edits also hurt, which hints at surface cue reliance. The authors say today's test generators lack regression awareness.

Catalog record

CREATORSabaat Haroon, Mohammad Taha Khan, Muhammad Ali Gulzar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55494
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.23443

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating LLM-Based Test Generation Under Software Evolution Sabaat Haroon, Mohammad Taha Khan, Muhammad Ali Gulzar Abstract . Large Language Models (LLMs) are increasingly used for automated unit test generation. However, it remains unclear whether these tests reflect genuine reasoning about program behavior or simply reproduce superficial patterns learned during training. If the latter dominates, LLM-generated tests may exhibit important weaknesses, including reduced coverage, missed regressions, and undetected faults. Understanding how LLMs generate tests for a program and how those tests respond to code evolution is therefore essential. In this work, we present a large-scale empirical study of LLM-based test generation under program evolution. Using an automated mutation-driven framework, we analyze how generated tests react to semantic-altering changes (SAC) and semantic-preserving changes (SPC). Our evaluation spans eight LLMs and 22,374 program variants derived from widely used benchmarks. LLMs achieve strong baseline results, reaching 79% line coverage and 76% branch coverage with fully passing test suites on the original programs. However, performance degrades as programs evolve. Under SACs, the pass rate of newly generated tests drops to 66%, and branch coverage declines to 60%. More than 99% of failing SAC tests pass on the original program while executing the modified region, indicating residual alignment with the original program behavior rather than adaptation to the updated semantics. Performance also drops under SPCs, despite unchanged functionality: test pass rates fall to 79% and branch coverage to 69%. SPC edits typically introduce larger syntactic changes than SAC edits while preserving semantics, yet they trigger greater instability in generated test suites. On average, models produce 1.2× more new tests while discarding many baseline tests, suggesting sensitivity to lexical changes rather than true semantic impact. Overall, our findings show that current LLM-based test generation relies heavily on surface-level cues and struggles to maintain regression awareness as programs evolve. 1. Introduction Large Language Models (LLMs) are increasingly integrated into software engineering workflows, with automated test generation emerging as a prominent use case ( 12 ) . Early studies and benchmarks suggest that LLMs can generate syntactically valid and often semantically correct test cases for well-known public programming tasks ( 8 ; 45 ; 25 ; 18 ) . However, producing complete and reliable automated test suites requires deep reasoning about control flow, execution paths, and the specific functional state of the provided implementation. Real-world software is inherently dynamic, code is frequently reused, refactored, and slightly tweaked to serve different functional purposes. When developers supply this modified code to an LLM to generate a high-coverage test suite, they expect the resulting tests to accurately reflect the current code. Problem. Current studies on LLM-based test generation largely overlook how code evolution impacts model robustness and behavior ( 41 ; 18 ; 40 ) . This gap leaves a critical question: do LLMs genuinely comprehend the semantics of the provided code, or are they performing shallow pattern replication of programs seen in pre-training? Consider a scenario in which a developer takes an existing open-source function and makes a minor semantic-altering modification to adapt it to a new use case. Even when explicitly prompted to generate high-coverage tests for this new code, an ideal test generator (e.g., random test generators ( 34 ) , Fuzzers ( 27 ; 45 ) or Symbolic execution ( 5 ) ) should adapt to the provided logic. However, if the LLM relies heavily on memorized structures, it may implicitly assume the modified code is a "buggy" version of the original program. It may also completely overlook the semantic implication of the code change. Consequently, it might generate test cases that fail on the provided code but paradoxically pass on the original, unmodified version seen during training. Conversely, if a developer applies a semantic-preserving mutation, such as renaming variables or refactoring logic without changing the output, the generated test coverage and passing rates should remain stable. If an LLM cannot maintain resilience under these harmless lexical changes, its utility in real-world test generation scenarios is compromised. Current State of the Art. Traditional test generation systems ( 14 ; 26 ; 20 ; 38 ) and fuzzers ( 45 ; 27 ) , are designed to maximize code coverage, while modern LLM code generation benchmarks (e.g., HumanEval, LiveCodeBench) focus almost exclusively on functional correctness in code generation tasks ( 11 ) . Neither approach characterizes how an LLM’s test generation behavior shifts as software evolves. Although mutation testing is a standard technique for assessing test suite adequacy, it has not been systematically adapted to evaluate LLM-based test generation under program changes. It is currently unknown whether LLMs can successfully adapt tests to functional changes while remaining resilient against changes that do not affect behavior. Contributions. A challenge in evaluating LLM-based test generation is distinguishing genuine semantic reasoning from alignment with patterns observed during training. High coverage on static benchmarks alone cannot resolve this question. A model may appear effective simply because the program under test resembles implementations encountered during training, allowing it to reproduce familiar test structures without reasoning about the specific code instance. To address this limitation, we evaluate LLM behavior under controlled program changes. We introduce a mutation-driven evaluation framework that applies two classes of code changes. Semantic-altering changes (SACs) produce program variants with modified behavior, requiring tests to adapt to new semantics. Semantic-preserving changes (SPCs) modify the code surface while preserving functionality. Because the program behavior remains unchanged, degradation under SPCs reveals sensitivity to structural or lexical variation rather than semantic reasoning. By analyzing LLM behavior under both change types, we isolate whether performance differences arise from semantic misunderstanding or reliance on superficial code patterns. This dual perspective allows us to characterize three properties of LLM-based test generation: sensitivity to semantic change, resilience to non-functional structural change, and stability of generated test suites across evolving programs. We design an automated end-to-end evaluation framework that generates baseline tests, measures coverage, injects SACs and SPCs, and evaluates test quality across program variants. Beyond coverage metrics, we analyze failing tests to determine whether their assertions align with the mutated program behavior or with the original program semantics. This analysis enables us to directly assess whether LLM-generated tests reflect true reasoning about program behavior or shallow pattern replication—insights that static public benchmarks cannot provide. Experimental Results. We evaluated 8 state-of-the-art LLMs using 22,374 program variants sourced from the Project CodeNet dataset ( 35 ) , specifically focusing on Java and Python implementations. Our baseline evaluation reveals that LLMs perform well on the original, unmodified programs, achieving an average of 79.2% line coverage and 76.1% branch coverage with fully passing test suites, containing 13.1 tests per program, on average. However, this performance degrades sharply under code changes. When subjected to SACs, the pass rate of newly generated tests plummets to 66.5%, and branch coverage falls to 60.6%. Crucially, further analysis of failing test cases finds that, among 119,163 tests generated under SACs, over 99% of the 23,977 tests that failed on the mutated code pass on the original program when executing the modified region. These results support our hypothesis that LLMs lack the precision to reason about semantic code changes and instead rely heavily on patterns observed during training . As a result, they often fail to capture behavioral changes

Read the complete text