You Don’t Need Public Tests to Generate Correct Code Conference: Preprint; 2026;

Kaushitha Silva, Srinath Perera · 2026

Read You Don’t Need Public Tests to Generate Correct Code Conference: Preprint; 2026; by Kaushitha Silva, Srinath Perera (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21598.txt.

Introduction

Kaushitha Silva and Srinath Perera introduce DryRUN, a framework for self-directed code writing without public tests or test runs. Multi-agent code writing leans on public tests for testing and debugging, but crafting complete pairs is labor-intensive; ground-truth examples are seldom available beforehand. Relying on public tests creates overconfidence: frameworks overfit examples and lag on hidden suites. DryRUN lets language models plan, create test inputs, and run mock runs to fix itself. On LiveCodeBench v6, it matches CodeSIM, a test-based baseline, while cutting tokens.

Catalog record

CREATORKaushitha Silva, Srinath Perera
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41878
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21598

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

You Don’t Need Public Tests to Generate Correct Code Conference: Preprint; 2026; Kaushitha Silva, Srinath Perera Abstract . Multi-agent systems are frequently employed for autonomous code generation, demonstrating strong utility in complex algorithmic problem-solving. Recent studies tackle the difficulty of producing functionally correct programs by leveraging simulation-guided planning and debugging, wherein language models step through execution traces to validate logic. Nevertheless, these methods rely heavily on human-authored public test cases to anchor the simulation and debugging cycles. Hand-crafting exhaustive input-output pairs creates a significant, labor-intensive bottleneck within the software development lifecycle. Since ground-truth examples are seldom accessible before actual implementation in real-world scenarios, this reliance limits existing approaches primarily to curated competitive programming datasets. Additionally, we demonstrate that depending on these public tests creates an "overconfidence gap," leading frameworks to overfit to basic examples and underperform on hidden test suites. Conversely, we note that external input samples are not an absolute requirement for successful code generation. We show that large language models possess the capability to autonomously construct valid inputs and simulate execution flows for self-correction. Building on this, we introduce DryRUN, a framework that removes the necessity for ground-truth data by enabling the LLM to iteratively plan, synthesize its own test inputs, and run simulated executions, thereby mitigating algorithmic overconfidence. Assessments using the LiveCodeBench v6 dataset (post-March 2025) reveal that DryRUN achieves comparable performance to CodeSIM, a state-of-the-art, test-dependent baseline. Notably, it does so entirely without public tests or external execution signals, all while decreasing overall output token usage. Standard Paradigm (e.g., CodeSIM) Problem Spec + Public Tests Plan & Generate Initial Code External Execution Sandbox Final Code Mental Trace on Failed Public Test Higher Overconfidence Gap Passes Public Tests Fails Public Tests DryRUN Paradigm (Ours) Standardized Spec (Zero Examples) Plan & Generate Initial Code Autonomously Synthesize Input Final Code Mental Trace & Refine (No Execution) Lower Overconfidence Gap Start Loop N iterations completed Figure 1. State-of-the-art Code Generation methods (top) rely heavily on sample examples provided in the problem specification, passing code to an external sandbox and exclusively tracing code that fails public tests . As demonstrated later in our Overconfidence Gap analysis, this heavy reliance on trivial public tests often induces algorithmic overconfidence, leading to overfitting and subsequent failure on hidden private suites. In contrast, DryRUN (bottom) operates under strictly zero-example constraints. It shifts the burden of verification entirely inward, utilizing the LLM’s ability to plan, autonomously synthesize inputs and simulate mental traces, eliminating the need for external oracles. State-of-the-art Code Generation methods (top) rely heavily on sample examples provided in the problem specification, passing code to an external sandbox and exclusively tracing code that fails public tests. As demonstrated later in our Overconfidence Gap analysis, this heavy reliance on trivial public tests often induces overconfidence, leading to overfitting and subsequent failure on hidden private suites. In contrast, DryRUN (bottom) operates under strictly zero-example constraints. It shifts the burden of verification entirely inward, utilizing the LLM's ability to plan, autonomously synthesize inputs and execute mental traces, eliminating the need for external oracles. 1. Introduction Program synthesis from natural language specifications represents a core objective in modern software engineering. Large language models (LLMs) have demonstrated remarkable proficiency in this domain, a capability primarily evaluated on standardized programming benchmarks such as HumanEval ( 3 ) and LiveCodeBench ( 9 ) . However, the problem specifications within these benchmarks inherently embed human-authored sample input-output examples. Consequently, most code generation methods rely on these public samples for fundamental problem comprehension Beyond this implicit reliance, recent state-of-the-art methods actively and explicitly exploit these public tests to drive their performance. This exploitation typically occurs across two distinct phases. During pre-generation reasoning, frameworks such as AlphaCodium ( 17 ) force the model to explicitly reason about how the provided public inputs map to their outputs to anchor the logic before writing any code. During post-generation debugging, methods utilize these public tests to obtain execution feedback. Furthermore, frameworks like CodeSIM ( 8 ) rely specifically on failed public test cases, prompting the model to mentally simulate an execution trace on that exact failed input to diagnose logical flaws. This deep dependency on public tests can create a massive disconnect from reality. In real-world software engineering contexts, ground-truth input-output examples are rarely available before the code is actually implemented. Furthermore, heavily relying on these tests introduces critical vulnerabilities. As observed by Liu et al. ( 14 ) and Liu et al. ( 15 ) , public test cases in standard benchmarks are often too trivial to catch complex edge cases. When models depend on signals from these simplistic tests, they frequently become overconfident in flawed solutions; passing the public tests but failing the hidden private suites. We formalize this disparity as the Overconfidence Gap , which is illustrated in Figure 1 and analyzed extensively in Section 5 . Conversely, manually authoring detailed public test cases that comprehensively cover edge cases is a tedious, labor-intensive process that places an unreasonable burden on developers. These challenges force us to ask, are public test cases the only way? Despite this ubiquitous implicit and explicit reliance on public tests, to the best of our knowledge, no prior studies have systematically evaluated how models perform when these sample examples are completely redacted from the prompt. Based on results we encountered while improving the planning for code generation, we hypothesize that this reliance on public tests is a crutch rather than a necessity. In this work, we first argue that removing human-authored examples from standard benchmarks yields negligible performance drops, indicating that modern LLMs do not strictly require them. Building on this insight, we introduce DryRUN ( D ebugging and R efinement U nder N on-execution), a framework that completely replaces the need for public tests. DryRUN prompts the LLM to autonomously synthesize its own valid, hypothetical inputs and mentally simulate execution traces, allowing it to self-correct logical errors without relying on external examples or sandbox execution. To rigorously evaluate our framework and mitigate the risk of data contamination, we conduct our experiments on the LiveCodeBench v6 dataset, strictly restricting our evaluation to problems published after March 2025. Because this benchmark natively includes public test cases, it provides an ideal testbed for a direct empirical comparison between standard public-test-dependent generation and our zero-example approach. We validate DryRUN’s efficacy across two LLMs: gpt-5-mini , gemini-3-flash . Our key contribution is questioning the role of public test cases as a major driver in correctness in code generation problems, exploring alternative approaches, and empirically demonstrating that the role of public test cases is much smaller than the community believes. Specifically, our main findings demonstrate that DryRUN matches the performance of CodeSIM, while operating under strictly zero-example constraints, no code execution, and consuming fewer tokens. Crucially, as depicted in Figure 1, DryRUN substantially mitigates the Overconfidence Gap by forcing the model to validate logic against its own non-trivial inputs rather than overfitting to trivial public oracles. Algorithm 1 The DryRUN Framework 1: \mathcal{P} (Standardized problem specification), N_{plan} , N_{sim} , N_{refine} , LLM 2: Code 3: Plan\leftarrow\text{GeneratePlan}(\text{LLM},\mathcal{P}) 4: for i=1 to N_{plan} do 5: Plan\leftarrow\text{RefinePlan}(\text{LLM},\mathcal{P},Plan) 6: end for 7: Code\leftarrow\text{SynthesizeCode}(\text{LLM},\mathcal{P},Plan) 8: for j=1 to N_{sim} do 9: Trace\leftarrow\text{SimulateExecution}(\text{LLM},\mathcal{P},Code) 10: Plan\leftarrow\text{TraceDrivenRefine}(\text{LLM},\mathcal{P},Plan,Trace) 11: Code\leftarrow\text{SynthesizeCode}(\text{LLM},\mathcal{P},Plan) 12: end for 13:

Read the complete text