LLM Based Input Space Partitioning Testing for Library APIs
Jiageng Li, Zhen Dong 1, Chong Wang, Haozhen You, Cen Zhang, Yang Liu, Xin Peng · 2025
Read LLM Based Input Space Partitioning Testing for Library APIs by Jiageng Li, Zhen Dong 1, Chong Wang, Haozhen You, Cen Zhang, Yang Liu, Xin Peng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05456.txt.
Introduction
Jiageng Li and co-authors use large language models to test library interfaces by dividing the input space into sensible regions. The paper is listed as 2025 in the catalog, but the text itself says it was published in 2024. Testing an interface means exploring vast input combinations, and blind search wastes effort on invalid inputs.
Their approach hands the model the interface's signature and code, asking it to describe each input region in words. Test inputs are then sampled from every described region, so the suite probes behavior systematically. The team evaluated the method on thousands of methods from ten popular open-source Java libraries. Language knowledge of data types and relations replaces hand-built rules, aiming at coverage that random generation rarely reaches.
Catalog record
| CREATOR | Jiageng Li, Zhen Dong 1, Chong Wang, Haozhen You, Cen Zhang, Yang Liu, Xin Peng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05456
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM Based Input Space Partitioning Testing for Library APIs Jiageng Li, Zhen Dong 1, Chong Wang, Haozhen You, Cen Zhang, Yang Liu, Xin Peng Abstract Automated library APIs testing is difficult as it requires exploring a vast space of parameter inputs that may involve objects with complex data types. Existing search based approaches, with limited knowledge of relations between object states and program branches, often suffer from the low efficiency issue, i.e., tending to generate invalid inputs. Symbolic execution based approaches can effectively identify such relations, but fail to scale to large programs. In this work, we present an LLM-based input space partitioning testing approach, LISP , for library APIs. The approach leverages LLMs to understand the code of a library API under test and perform input space partitioning based on its understanding and rich common knowledge. Specifically, we provide the signature and code of the API under test to LLMs, with the expectation of obtaining a text description of each input space partition of the API under test. Then, we generate inputs through employing the generated text description to sample inputs from each partition, ultimately resulting in test suites that systematically explore the program behavior of the API. We evaluate LISP on more than 2,205 library API methods taken from 10 popular open-source Java libraries ( e.g., apache/commons-lang with 2.6k stars, guava with 48.8k stars on GitHub). Our experiment results show that LISP is effective in library API testing. It significantly outperforms state-of-the-art tool EvoSuite in terms of edge coverage. On average, LISP achieves 67.82% branch coverage, surpassing EvoSuite by 1.21 times. In total, LISP triggers 404 exceptions or errors in the experiments, and discovers 13 previously unknown vulnerabilities during evaluation, which have been assigned CVE IDs. I Introduction The third party libraries, as an essential part in software ecosystems, have become one of the most significant contributors to fast development of today’s software system. According to a recent study [ 1 ] , a Java project directly relies on different 14 third party libraries. Vulnerabilities within these libraries can pose significant risks to numerous software systems. Consequently, testing libraries is imperative to ensure system security. However, testing library APIs is notoriously challenging as it entails exploring a vast input space of multiple parameters, particularly when these parameters involve objects with complex data types. The behavior of the libraries can be constrained by a specific state of one or more input objects. Triggering such a state involves generating input values satisfying relevant conditions as well as generating statements to instantiate these objects. This poses numerous challenges for existing automated test generation techniques: (1) Search based testing: Most existing techniques [ 2 , 3 , 4 , 5 , 6 , 7 ] frame automated test generation as an optimization problem over the input space with the goal of generating inputs to achieve maximal code coverage, for instance, EvoSuite [ 3 ] , a widely used automated test generation tool adopts a genetic algorithm to generate tests. The problem with this type of techniques is the low efficiency issue when tackling the expansive space of inputs involving multiple objects with complex data types within library APIs; (2) Symbolic execution: Symbolic execution is an effective testing technique that can generate inputs that cover desired program paths. Yannic Noller et al. leverages symbolic execution to guide fuzzing to generate inputs that cover deep program behavior [ 8 ] . Despite significant efficiency improvement, these techniques face difficulties in scaling to large programs due to inherent limitations of symbolic execution, e.g., SPF [ 9 ] has limited support for heap input. SUSHI [ 10 ] proposed by Pietro Braione et al. can be only applied to Java classes. In this paper, we view automated test generation as a program input space sampling problem. The ideal way to sample is to compute input space partitions , and then choose inputs from each partition so as to cover all possible program behavior. In this perspective, search-based approaches kind of leverage heuristics to guide search, aiming to sample inputs from as many partitions as possible. Symbolic execution based approaches attempt to compute input space partitions by solving program path conditions and then sample inputs from each partition. Both type of approaches come at a cost. The former requires executing a large amount of inputs that go through redundant program paths, the latter requires heavy computation resources to solve path conditions. In this work, we propose an Large Language Model (LLM) based input space partitioning testing approach for library APIs. Specifically, we leverage LLMs to infer the input space partitions of a library API under test and then sample inputs from each partition so as to generate test suites with high quality. Recently, LLMs have demonstrated promising capabilities in understanding programs and common knowledge reasoning, leading to their widespread adoption in the software engineering domain [ 11 , 12 ] . Motivated by these capabilities, we explore using LLMs to automate input space partitioning, achieving the objectives of symbolic execution without explicitly performing it . To this end, we propose a framework that interacts with LLMs to compute input space partitions for a given library API and generate input values based on textual descriptions of each partition, resulting in high-quality test inputs. Subsequently, the framework takes those inputs to generate test suites for library API testing. We evaluated LISP on 2,205 APIs from 10 widely used libraries, including Apache Commons-lang3 and Google Guava. The results show LISP is highly effective in testing library APIs, achieving exceptionally high code coverage with a minimal number of generated inputs. In the comparison experiments, LISP outperformed the state-of-the-art technique EvoSuite, achieving 1.21 times higher edge coverage. Furthermore, LISP identified 404 exceptions across the 10 libraries, including 13 previously undiscovered vulnerabilities, which have been assigned CVE IDs. To support future research, we make our the experimental data and results publicly available at the following link: https://github.com/FudanSELab/LISP [ 13 ] . II Motivating Examples ⬇ 1 // ApcomplexMath.java 2 public static Apcomplex pow ( Apcomplex z , Apcomplex w ) 3 throws ApfloatRuntimeException { 4 Apcomplex result = ApfloatHelper . checkPow ( 5 z , w , Math . min ( z . precision (), w . precision ())); 6 if ( result != null ) { 7 return result ; 8 } else if ( z . real (). signum () >= 0 && 9 z . imag (). signum () == 0) { 10 Apfloat x = z . real (); 11 Apfloat one = new Apfloat ( 12 1 L , Long . MAX_VALUE , x . radix ()); 13 x = // ignore some code 14 return exp ( w . multiply ( ApfloatMath . log ( x ))); 15 } else { 16 return exp ( w . multiply ( log ( z ))); 17 } 18 } Listing 1: org.apfloat.ApcomplexMath::pow II-A Importance of Code Understanding and Common Knowledge presents an API method named pow within the class ApcomplexMath from the apfloat . This method, which takes two parameters named z and w , exhibits distinct behaviors based on the content of z and w , which means that each input space can be represented by the states