Assertion-Aware Test Code Summarization with Large Language Models
Anamul Haque Mollah, Ahmed Aljohani, Hyunsook Do · 2025
Read Assertion-Aware Test Code Summarization with Large Language Models by Anamul Haque Mollah, Ahmed Aljohani, Hyunsook Do (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06227.txt.
Introduction
Anamul Haque Mollah, Ahmed Aljohani, and Hyunsook Do study how large language models summarize unit tests. Test code often lacks short summaries that state what a test checks, especially in generated or poorly documented projects. Because tests express intent through assertions, summarizing them differs from summarizing ordinary code.
The authors built a benchmark of ninety-one real Java test cases paired with developer-written summaries. Four code models were tried under seven prompt designs that varied which test parts were shown. Prompting with assertion meanings beat showing the full method under test while using fewer tokens. Codex and Qwen-Coder matched human summaries best in their tests.
Catalog record
| CREATOR | Anamul Haque Mollah, Ahmed Aljohani, Hyunsook Do |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06227
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assertion-Aware Test Code Summarization with Large Language Models Anamul Haque Mollah, Ahmed Aljohani, Hyunsook Do Abstract Unit tests often lack concise summaries that convey test intent, especially in auto-generated or poorly documented codebases. Large Language Models (LLMs) offer a promising solution, but their effectiveness depends heavily on how they are prompted. Unlike generic code summarization, test-code summarization poses distinct challenges because test methods validate expected behavior through assertions rather than implementing functionality. This paper presents a new benchmark of 91 real-world Java test cases paired with developer-written summaries and conducts a controlled ablation study to investigate how test code-related components-such as the method under test (MUT), assertion messages, and assertion semantics-affect the performance of LLM-generated test summaries. We evaluate four code LLMs (Codex, Codestral, DeepSeek, and Qwen-Coder) across seven prompt configurations using n-gram metrics (BLEU, ROUGE-L, METEOR), semantic similarity (BERTScore), and LLM-based evaluation. Results show that prompting with assertion semantics improves summary quality by an average of 0.10 points (2.3%) over full MUT context (4.45 vs. 4.35) while requiring fewer input tokens. Codex and Qwen-Coder achieve the highest alignment with human-written summaries, while DeepSeek underperforms despite high lexical overlap. The replication package is publicly available at https://doi.org/10.5281/zenodo.17067550 I Introduction Unit tests are a critical part of the codebase; they help ensure the correctness of the Method Under Test (MUT), prevent regressions, and document expected behavior. Yet, as test suites grow in size and complexity, understanding individual test cases—especially without comments—can become time-consuming. Developers often rely on summaries or comments to quickly understand a test’s intent, but such documentation is inconsistently written or entirely missing in many codebases. Prior work [ 1 ] has shown that developers often mitigate issues like vague tests or obscure behavior by writing descriptive test names or clear test comments [ 2 ] . These high-level summaries help developers understand the rationale behind a test without reading the entire test code. Despite their usefulness, test comments are often missing, outdated, or inconsistent—especially in auto-generated test suites from search-based approaches like EvoSuite [ 3 ] and transformer-based models [ 4 , 5 ] . To address this, several tools have been proposed to automatically generate summaries for test cases. TestDescriber, introduced by Panichella et al. [ 6 ] , uses a template-based strategy grounded in the SWUM (Software Word Usage Model) representation of code statements. Similarly, DeepTC-Enhancer, proposed by Roy et al. [ 7 ] , combines deep learning with fixed templates to produce step-by-step natural language descriptions of tests. While both approaches improve test readability to some extent, they often produce verbose and redundant outputs that fall short of developer expectations for concise, high-quality summaries. Large language models (LLMs) have recently emerged as promising tools for code comprehension [ 8 ] . When carefully prompted, models such as GPT-4 can produce fluent, context-appropriate text. Unlike production methods, test methods encode expected behavior through assertion constructs and often reference external methods under test. Summarizing them therefore requires reasoning over validation intent rather than implementation logic, making test-code summarization a distinct challenge compared to general code summarization. Nevertheless, most existing summarization benchmarks, such as CodeXGLUE [ 9 ] , and many LLM-based techniques [ 10 ] , focus primarily on production code, overlooking the linguistic and structural nuances of test methods. Developer-written test summaries often begin with behavioral cues like “tests whether” or “verifies that,” and test bodies encode intent through assertions that specify expected outcomes. In the context of test code, UTGen by Deljouyi et al. [ 11 ] demonstrated that large language models (LLMs) can enhance the readability of automatically generated unit tests, particularly through improvements in variable naming, test data selection, and embedded comments. However, their work does not provide empirical evidence regarding whether the generated test code summaries are meaningful or aligned with the original developer intent. Following this, Djajadi et al. [ 12 ] explored the capability of LLMs to generate high-quality summaries for test methods. Their study compared several prompting strategies, including baseline prompting, the inclusion of MUT context, and few-shot examples. The results indicated that prompt design plays a critical role in determining summary quality, influencing key dimensions such as content relevance, conciseness, and linguistic naturalness. Despite recent advances, two key gaps remain. First, prior evaluations have been narrow in scale. For instance, Djajadi et al. [ 12 ] assessed LLM-generated summaries using only four test cases, which constrains the generalizability and robustness of their findings. Second, existing studies have largely neglected structural characteristics unique to test code—notably, assertion statements and their accompanying messages. These elements encode the intended behavior of the code and offer high-value semantic features for summarization. When such features are underemphasized, LLMs may fail to capture the behavioral intent of tests, particularly in long or complex cases [ 13 ] . Recent work [ 14 , 15 ] demonstrates that enriching prompts with semantic context can substantially enhance LLM performance in code summarization. While incorporating the full MUT context can provide additional behavioral information, it also increases token overhead and inference latency. In contrast, focusing on test code features offers a lightweight, scalable, and behaviorally rich alternative for improving summarization. To address these limitations, we propose the following research questions: 1. RQ1: To what extent do assertion-level features influence the quality of LLM-generated summaries? 2. RQ2: How do different code LLMs vary in their effectiveness across prompt variants? This paper introduces a benchmark and ablation study for test code summarization. The benchmark comprises 91 developer-written Java tests, each paired with a gold comment written by the original developer. We evaluate seven prompt configurations that vary the inclusion of test code, MUT, assertion statements, and assertion semantics. We measure quality using n-gram metrics (BLEU, ROUGE-L, METEOR), semantic measures (BERTScore), and LLM-as-a-judge (LLM-Eval). This paper makes the following contributions: 1. We introduce a benchmark of 91 developer-written Java test cases paired with developer-written comments. 2. We conduct an ablation study across seven prompt configurations. 3. We perform evaluation across four state-of-the-art code LLMs (Codex, Codestral, DeepSeek, and Qwen-Coder) and five evaluation metrics (BLEU, METEOR, ROUGE-L, BERTScore, and LLM Eval). 4. We release a replication package to support reuse and future benchmarking. 1 1 1 https://doi.org/10.5281/zenodo.17067550 Our results provide actionable guidance on how to prompt code LLMs for test summarization. We show when MUT context helps, when assertion semantics can substitute for or complement MUT to reduce input length and cost, and how these components interact to improve content fidelity and conciseness in the generated summaries. The remainder of this paper is organized as follows: Section II reviews related work on automated test code summarization. Section III outlines the methodology and experimental design. Section IV presents the results and key findings derived from our evaluation. Section V discusses the broader implications of the findings and provides additional insights. Section VI outlines the potential limitations and threats to the validity of our study. Finally, Section VII concludes the paper and highlights directions for future work. II Related Work This section reviews studies on automated test code summarization, focusing on the role of Template-based, Deep learning, and Large Language Models (LLMs). Prior work has shown that comments enable developers to fix bugs faster, understand test intent, and navigate large codebases more efficiently.