HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning
Yujian Liu, Jiabao Ji, Yang Zhang, Wenbo Guo, Tommi Jaakkola, Shiyu Chang · 2025
Read HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning by Yujian Liu, Jiabao Ji, Yang Zhang, Wenbo Guo, Tommi Jaakkola, Shiyu Chang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01104.txt.
Introduction
Yujian Liu and colleagues train language models to write testing harnesses, not just test cases. Ordinary automatic test generation produces input and expected output pairs, which limits the variety of tests and gives little help when a failure needs debugging.
A harness is code that synthesizes inputs and validates observed outputs, so it can express complex cases and check invariants rather than one fixed answer. HarnessLLM is a two stage pipeline: supervised fine tuning first, then reinforcement learning from verifiable rewards with a custom reward design. The resulting models beat input output based testing at finding bugs and at strategy diversity. Their generated tests also help at generation time, improving code produced by models that validate drafts against the harnesses.
Catalog record
| CREATOR | Yujian Liu, Jiabao Ji, Yang Zhang, Wenbo Guo, Tommi Jaakkola, Shiyu Chang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-01104
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning Yujian Liu, Jiabao Ji, Yang Zhang, Wenbo Guo, Tommi Jaakkola, Shiyu Chang Abstract Existing LLM-based automatic test generation methods mainly produce input and expected output pairs to categorize the intended behavior of correct programs. Although straightforward, these methods have limited diversity in generated tests and cannot provide enough debugging information. We propose HarnessLLM, a two-stage training pipeline that enables LLMs to write harness code for testing. Particularly, LLMs generate code that synthesizes inputs and validates the observed outputs, allowing complex test cases and flexible output validation such as invariant checking. To achieve this, we train LLMs with SFT followed by RLVR with a customized reward design. Experiments show that HarnessLLM outperforms input-output-based testing in bug finding and testing strategy diversity. HarnessLLM further benefits the code generation performance through test-time scaling with our generated test cases as inference-phase validation. Our code is available at https://github.com/UCSB-NLP-Chang/HarnessLLM.git . 1 Introduction Large language models (LLMs) have demonstrated remarkable proficiency in code-related tasks, including code generation, completion, and even resolving software engineering issues through tool use Chen et al. 2021 ; Li et al. 2022 ; OpenAI 2024 ; DeepSeek-AI 2025 ; Jimenez et al. 2024 ; Li et al. 2025 . However, compared to these code generation tasks, automatic testing and debugging AI-generated programs have received comparatively little attention, even though comprehensive test suites are critical for ensuring the correctness and robustness of the AI-generated code Chen et al. 2024a ; Prasad et al. 2025 ; Sinha et al. 2025 ; He et al. 2025 ; Zhang et al. 2023a . Existing works in automatic testing mainly prompt the language model to generate input–output pairs that characterize the intended behavior of correct programs Chen et al. 2022 ; Prasad et al. 2025 ; Zeng et al. 2025 ; Lin et al. 2025 . As depicted in Figure 2 , the model produces examples of test inputs alongside their expected test outputs. The target program is then executed on a test input, and the output that the program generates is compared against the corresponding expected test output. A bug is exposed if the two outputs diverge. Although straightforward, such an input-output test case generation paradigm has two potential drawbacks. First , the test inputs generated by the language model tend to be simple and homogeneous, so they may not have sufficient coverage of the sophisticated corner cases that could expose bugs. Second , such a paradigm requires that the model generates the correct output by itself, which becomes extremely challenging for complicated programming tasks or for complicated test inputs. In short, the fundamental paradox is that the ‘tester’, i.e., the language model that generates test cases, is often much weaker than the ‘testee’, i.e., the program, in accomplishing the complex programming tasks (otherwise, the language models would not have to rely on code generation to solve these tasks). Figure 1: Comparison between input-output pairs (top) and test harness (bottom). Figure 2: Percentage of found bugs (average of 8 runs, higher is better) for two strategies with different models. In this paper, we explore a novel debugging paradigm that could resolve this paradox – LLM-based test harness generation . Rather than letting the language model directly generate input-output pairs, we prompt it to write executable code, a matching rival to the testee, to generate test inputs and validate target program outputs. In this way, both aforementioned drawbacks can be addressed simultaneously. On the input side, executable code can easily generate various richly structured, diverse, and complicated inputs. On the output side, executable code opens up many possibilities to validate target program outputs. It can ❶ directly generate hardcoded expected output, as does the input-output paradigm, or ❷ write a reference program to compute the expected output, or, most interestingly ❸ assert output properties and requirements. As shown in Figure 2 , for a program that sorts a list of input integers, the LLM first writes an input generator, generate_input_1 , to generate random lists, which are fed to the target program for execution. The returned outputs are then validated by an LLM-defined function, check_output , which checks that the result is sorted and preserves the original integers. With programmatic input generation and output validation, testing harnesses can support complex invariant checking and stress testing, enabling more comprehensive testing and detection of deep logical bugs. However, the key challenge of the test harness generation paradigm is that even the strong coding LLMs are not inherently capable of test code generation, which requires different skills than generating code for programming tasks: The former requires understanding the given program’s logic, control and data flow, designing proper stress tests, and writing validation logic, while the latter is mainly about writing code to fulfill the required functionality. To demonstrate this, our initial experiment compares the bug-finding rates of the input–output strategy versus test harness generation on the LiveCodeBench and Codeforces datasets Jain et al. 2025 ; Penedo et al. 2025 , using a strong reasoning model Qwen3-32B Yang et al. 2025 . Surprisingly, direct prompting for test harnesses does not yield better bug finding capabilities (Figure 2 ). To close this gap, we propose HarnessLLM, a two-stage training pipeline combining supervised fine-tuning (SFT) with reinforcement learning (RL) with customized reward functions. First, we collect SFT data by prompting Qwen3-32B and filtering for harnesses that successfully expose a bug. We warm up a smaller model ( e.g., Qwen3-4B ) with SFT on collected data. The purpose of this stage is to provide a reasonable starting point for reinforcement learning, which improves RL’s training efficiency. Second, we further train the SFT model using RL with our customized verifiable outcome reward. Here, we assume access to a ground-truth program during training. To encourage the model to generate valid harnesses, we first give a zero reward to generated tests that trigger compilation or runtime errors on the ground-truth program. Then, we design rewards to incentivize the model to generate effective tests that crash the target programs. Specifically, a positive reward is assigned when the ground-truth program can pass the generated tests but the target program fails, indicating that the test harness correctly identifies bugs in the target program. We train the model to maximize the expected reward using the GRPO algorithm Shao et al. 2024 . The RL training can further strengthen the model’s capabilities to generate effective test harnesses and improve the model’s generalizability. We train on two base models ( Qwen3-4B and Llama3.2-3B Llama 2024 ) and evaluate on three benchmarks containing buggy programs. Experiments show that our model outperforms all baselines, including the off-the-shelf Qwen3-32B and another model that is also trained with RL but only generates input-output pairs (Figure 2 presents an overview). Moreover, the learned harness generator generalizes to code produced by unseen models and can be used for improving code generation performance. Specifically, using the execution results of generated test cases to select the best out of 8 responses improves Qwen3-32B ’s performance from 63.5% to 69.5% on LiveCodeBench Jain et al. 2025 . To the best of our knowledge, HarnessLLM is the first LLM-based testing harness generation that enables comprehensive testing and benefits competitive programming tasks . We summarize our