Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models

Jakub Walczak, Piotr Tomalak, Artur Laskowski · 2025

Read Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models by Jakub Walczak, Piotr Tomalak, Artur Laskowski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14256.txt.

Introduction

Jakub Walczak, Piotr Tomalak, and Artur Laskowski studied how added code context and prompt style change the unit tests that language models write, in a paper from 2025. Unit tests make up most test suites and follow routine patterns. The team compared several model families under different context levels.

Adding code comments clearly improved test adequacy, while adding the full code body gave smaller extra gains. Step by step reasoning prompts did best, reaching high branch coverage and strong fault catching scores with near perfect compile rates. One model led on both fault catching and coverage. All code and test suites are released in public. The authors frame context choice as a real design lever for AI testing help.

Catalog record

CREATORJakub Walczak, Piotr Tomalak, Artur Laskowski
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-14256

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Impact of Code Context and Prompting Strategies on Automated Unit Test Generation with Modern General-Purpose Large Language Models Jakub Walczak, Piotr Tomalak, Artur Laskowski Abstract Generative AI is gaining increasing attention in software engineering, where testing remains an indispensable reliability mechanism. According to the widely adopted testing pyramid, unit tests constitute the majority of test cases and are often schematic, requiring minimal domain expertise. Automatically generating such tests under the supervision of software engineers can significantly enhance productivity during the development phase of the software lifecycle. This paper investigates the impact of code context and prompting strategies on the quality and adequacy of unit tests generated by various large language models (LLMs) across several families. The results show that including docstrings notably improves code adequacy, while further extending context to the full implementation yields definitely smaller gains. Notably, the chain-of-thought prompting strategy — applied even to ’reasoning‘ models — achieves the best results, with up to 96.3% branch coverage, a 57% average mutation score, and near-perfect compilation success rate. Among the evaluated models, M5 (Gemini 2.5 Pro) demonstrated superior performance in both mutation score and branch coverage being still in top in terms of compilation success rate. All the code and resulting test suites are publicly available at https://github.com/peetery/LLM-analysis . I Introduction Recent years have brought significant advancements in artificial intelligence (AI), particularly in the areas of performance and productivity enhancement. However, AI — and particularly large language models (LLMs) — still suffer from several weaknesses. Among them, convincing but senseless content generation (’hallucination‘), safety misalignment (’ethicality‘) [ 1 ] , unfairness [ 2 ] , and limited processing context are the most critical. In spite of these restrictions, and bearing in mind the limited and merely apparent creativity of LLMs [ 3 ] , they have become versatile tools already widely used across a variety of domains (creative industries [ 4 ] , entertainment, reporting, and software engineering [ 5 ] are just cases in point) for multiple tasks. Yet, in most pursuits, the output of LLMs should undergo strict human supervision, especially in activities related to privacy, security, health, or society. This is a gap filled by hybrid intelligence [ 6 ] , understood as amplifying human capabilities [ 7 ] and, in particular, productivity [ 8 ] in schematic and tedious activities. In this paper, we present a comprehensive survey in the context of hybrid intelligence in automated unit-test generation using general-purpose models (across a variety of LLM families) verified against different prompting strategies. We juxtapose them with the test cases of the software development practitioner. We established and justified a solid methodology and, based on the results, proposed a set of guidelines for the process of automated efficient and reliable unit-test generation. We investigated several different approaches, elaborated and justified in detail in the Methodology section. II Related works II-A Large Language Models Large language models, such as GPT, Claude, and Gemini, have recently gained significant attention in the field of software engineering due to their ability to understand and generate source code. These models are typically based on transformer architectures, which process input as a sequence of tokens using self-attention mechanisms [ 9 ] . Each output token is generated based on the entire input context, enabling the model to capture complex dependencies within the data. A core concept in LLMs is the token mentioned before. It is the smallest chunk of text the model ingests and emits. Depending on the tokenizer (algorithm for converting text into tokens), a token might be an individual character, a frequent word, or a subword piece (e.g. the word ’tokenization‘ split into ’token‘ + ’ization‘). Subword-level tokenization such as Byte-Pair Encoding (BPE) [ 10 ] or WordPiece [ 11 ] which strikes a balance between fixed vocabularies and open-ended text, allowing models to handle rare or compound words gracefully. During both training and inference, the transformer treats each token as an input embedding and applies self-attention across the entire sequence, generating outputs one token at a time conditioned on all preceding tokens. Besides pure processing meaning, tokens play significant role also in the usage cost of LLMs. The price of a single token varies significantly among LLM models (see Tab. I ) families and their ’advancement‘. TABLE I: Differences in prices of input tokens for chosen LLMs Model Cost per 10^{6} tokens [$] GPT-4.1 mini 0.40 GPT-4.1 2.00 Claude Haiku 3 0.25 Claude Opus 4 15.00 In conversational settings, such as with ChatGPT, the full conversation history is included as a part of the prompt context at each step. This means that LLMs operates statelessly — they do not ’remember‘ previous interactions unless those interactions are explicitly included in the current input. As such, the complete context visible to the model is the concatenation of all prior messages in the conversation window , subject to the model’s token limit [ 1 ] . This design makes prompt engineering — the deliberate crafting and structuring of prompts — critical for eliciting accurate, relevant outputs. Even minor violations in phrasing or the order of context examples can significantly affect LLM performance, especially in structured tasks such as code completion or test-case generation [ 12 , 3 ] . Despite their general capabilities, LLMs can suffer from limitations such as hallucinations (confidently incorrect outputs) [ 13 ] , invalid syntax, or shallow semantic understanding. These issues are especially problematic in automated software testing, where both syntactic correctness and logical coverage are of utmost importance [ 2 ] . II-B Testing in software development process Software testing is one of the most critical phases of the Software Development Life Cycle as it increases reliability of software (and hence, a company) and enable to reduce potential costs wreaked by buggy releases [ 14 ] . Testing can be categorized into three main techniques: • functional, • performance, • security. During the coding phase, developers primarily focus on verifying the functionality of their code, which falls under the category of functional testing. This category includes, among others, unit tests and integration tests . The type and number of tests required in a project depend on the nature of the software. Projects with complex domain logic — often among the most profitable — tend to require a large number of tests to verify the correctness of that logic. Domain logic is independent of technologies or frameworks; rather, it represents the business rules behind the process. For instance, domain logic might define whether an order can transition directly from the PENDING state to the APPROVED state, or if it must first pass through the PAID state. Unit tests are typically used to verify such scenarios. For systems with rich domain logic, the well-known testing pyramid approach introduced by Cohn [ 15 ] is commonly adopted. This framework, further discussed in [ 16 , 17 ] , emphasizes the importance of unit tests, which usually form the majority of a project’s test suite and occupy a significant portion of developers’ time. This is particularly true for tests that do not require deep domain expertise but still need to be written exhaustively. These factors motivate our focus on automated unit test generation. However, unit tests should not be evaluated

Read the complete text