LLM Test Generation via Iterative Hybrid Program Analysis
Sijia Gu, Noor Nashid, Ali Mesbah · 2025
Read LLM Test Generation via Iterative Hybrid Program Analysis by Sijia Gu, Noor Nashid, Ali Mesbah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13580.txt.
Introduction
Sijia Gu, Noor Nashid, and Ali Mesbah present Panta, a 2025 technique that helps language models write better unit tests for tangled code. Models can draft tests quickly but miss deep branches because they cannot follow complex control flow on their own.
Panta mimics a human tester: it reads the code's static flow map, runs drafts to see which paths stay untested, and feeds that back to the model for another round. On the most complex classes from open-source projects, the loop gained twenty-six percent more line coverage and twenty-three percent more branch coverage than the best prior method. Iteration, not bigger prompts, did the work.
Catalog record
| CREATOR | Sijia Gu, Noor Nashid, Ali Mesbah |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13580
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM Test Generation via Iterative Hybrid Program Analysis Sijia Gu, Noor Nashid, Ali Mesbah Abstract . Automating unit test generation remains a significant challenge, particularly for complex methods in real-world projects. While Large Language Models (LLMs) have made strides in code generation, they struggle to achieve high branch coverage due to their limited ability to reason about intricate control flow structures. To address this limitation, we introduce Panta , a technique that emulates the iterative process human developers follow when analyzing code and constructing test cases. Panta integrates static control flow analysis and dynamic code coverage analysis to systematically guide LLMs in identifying uncovered execution paths and generating better test cases. By incorporating an iterative feedback-driven mechanism, our technique continuously refines test generation based on static and dynamic path coverage insights, ensuring more comprehensive and effective testing. Our empirical evaluation, conducted on classes with high cyclomatic complexity from open-source projects, demonstrates that Panta achieves 26% higher line coverage and 23% higher branch coverage compared to the state-of-the-art. 1. Introduction Automated test generation plays a crucial role in software engineering by enhancing code reliability and maintainability. Large Language Models (LLMs) can replicate human coding styles and generate test cases with meaningful variable names, leveraging patterns learned from extensive training data. This capability has led to the adoption of LLMs for test code generation ( Nashid et al. 2023 ; Lemieux et al. 2023 ; Alshahwan et al. 2024 ; Chen et al. 2024 ; Pan et al. 2025 ; Ryan et al. 2024 ; Wang et al. 2024a ; Deljouyi et al. 2024 ) . Despite their effectiveness in general coding tasks, LLMs—trained on diverse datasets—often struggle with the unique code structures and logic of specific projects. This limitation is particularly evident in cases involving complex control flow and conditional statements. Recent studies ( Siddiq et al. 2024 ; Schäfer et al. 2023 ; El Haji 2023 ; Wang et al. 2024b ) have shown that LLM-generated test cases frequently fail to achieve sufficient branch coverage, as they inadequately capture intricate conditional logic. To address this limitation, researchers have explored techniques that supplement LLMs with additional program-related information, such as focal method slices ( Wang et al. 2024a ) and path constraints ( Ryan et al. 2024 ) . These approaches rely on static prompts containing method fragments to generate tests for each independent segment. However, they lack mechanisms to guide the generation process toward areas requiring additional coverage. Furthermore, since LLMs do not always produce correct code, only a subset of the generated tests compile and pass, leaving significant portions of the code untested. Additionally, these methods primarily focus on generating new test classes rather than augmenting existing ones. CoverUp ( Pizzorno and Berger 2024 ) takes a different approach by leveraging code coverage information to highlight uncovered portions of the code. However, our empirical evidence suggests that this approach is insufficient for more complex code, where additional contextual guidance is needed to drive meaningful improvements in branch coverage. (a) Execute existing tests to locate code that lacks coverage ⬇ 1 public static Options parsePattern ( String pattern ){ 2 char opt = ’ ’ ; 3 boolean required = false ; 4 Class <?> type = null ; 5 final Options options = new Options (); 6 for ( int i = 0; i < pattern . length (); i ++){ 7 final char ch = pattern . charAt ( i ); 8 if (! isValueCode ( ch )) { 9 // omit some details 10 opt = ch ; 11 } else if ( ch == ’!’ ) { 12 required = true ; 13 } else { 14 type = ( Class <?>) getValueClass ( ch ); 15 } 16 } 17 if ( opt != ’ ’ ) { 18 // omit some details 19 } 20 return options ; 21 } (b) Identify paths and reason about execution scenarios ⬇ 2 char opt = ’ ’ ; 3 boolean required = false ; 4 Class <?> type = null ; 5 final Options options = new Options (); 6 for ( int i =0; i < pattern . length (); i ++) is False 17 if ( opt != ’ ’ ) is False 20 return options ; When pattern is empty ⬇ 2 char opt = ’ ’ ; 3 boolean required = false ; 4 Class <?> type = null ; 5 final Options options = new Options (); 6 for ( int i =0; i < pattern . length (); i ++) is True 7 final char ch = pattern . charAt ( i ); 8 if (! isValueCode ( ch )) is False 11 else if ( ch == ’!’ ) is True 12 required = true ; 6 for ( int i =0; i < pattern . length (); i ++) is False 17 if ( opt != ’ ’ ) is False 20 return options ; When pattern contains only character "!" (c) Add new tests for the execution scenarios ⬇ @Test public void testEmptyPattern () { Options options = PatternOptionBuilder . parsePattern ( "" ); assertTrue ( options . getOptions (). isEmpty ()); } @Test public void testRequiredOption () { Options options = PatternOptionBuilder . parsePattern ( "!" ); // proper assertions } Figure 1 . Workflow of a developer to add new test cases In this paper, we present Panta ( P rogram AN alysis guided T est A utomation), a novel approach that combines dynamic code coverage analysis and static control flow analysis to generate tests for under-tested execution paths. Our approach follows an iterative workflow, implementing a multi-step, feedback-driven fully automated process. Panta first gathers dynamic code coverage information and static control flow insights. Based on the extracted program analysis information, it identifies and selects execution paths that require additional test coverage for the class, and prompts an LLM to generate test cases that cover those paths. After test generation, Panta validates the generated tests, adding the passing ones back to the test class while repairing failed tests. The feedback from the test repair process is incorporated into subsequent test generation iterations. Our work makes the following contributions: • An iterative feedback-driven technique that integrates dynamic and static program analysis to assist LLMs in generating tests that cover complex control flow paths. • A novel path selection strategy that first ranks linearly independent execution paths—identified via control flow analysis—using a coverage deficiency score computed from code coverage data. It then selects prioritized paths while incorporating historical selection data to effectively balance exploitation and exploration. • A fully automated test generation tool, Panta , that supports both generating new test classes and augmenting existing tests. • An empirical evaluation of our approach using real-world open-source programs from the Defects4J ( Just et al. 2014 ) benchmark. Our study demonstrates that Panta achieves 26% higher line coverage and 23% higher branch coverage compared to the state of the art. 2. Motivation Recent advances in LLMs have dramatically enhanced their code generation capabilities, including the ability to create tests for diverse execution scenarios. However, when dealing with complex methods that contain multiple conditional branches, LLMs often struggle to