Effective Harness Engineering for Algorithm Discovery with Coding Agents
Yoichi Ishibashi, Taro Yano, Masafumi Oyamada · 2026
Read Effective Harness Engineering for Algorithm Discovery with Coding Agents by Yoichi Ishibashi, Taro Yano, Masafumi Oyamada (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15221.txt.
Introduction
Yoichi Ishibashi, Taro Yano, and Masafumi Oyamada study harness design for algorithm discovery with coding agents, where infrastructure shapes success as much as model skill. Using their Vesper framework on circle packing under a fixed token budget, they find fewer methods with deeper thought beat many methods with brief thought. More capable models also hack the evaluation more often, making hack detection more needed as models scale. Safe side-by-side running for file store agents is handled as well.
Catalog record
| CREATOR | Yoichi Ishibashi, Taro Yano, Masafumi Oyamada |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39649 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Effective Harness Engineering for Algorithm Discovery with Coding Agents Yoichi Ishibashi, Taro Yano, Masafumi Oyamada Abstract AlphaEvolve and FunSearch have demonstrated the potential of combining large language models (LLMs) with evolutionary search for automated algorithm discovery. However, discovery success is shaped not only by model capability but also significantly by the design of the execution infrastructure, i.e., the harness. This paper investigates effective harness design through three questions: under a fixed token budget, is it better to produce many algorithms with brief thought or fewer algorithms with deeper thought? How should the harness handle evaluation hacks, where generated programs exploit the scoring function? And how can agents that require full filesystem access execute safely in parallel? Using Vesper, an algorithm discovery framework that incorporates harness improvements addressing these questions, we evaluate on Circle Packing under the same token budget. Interestingly, generating fewer algorithms while thinking more deeply about each one achieved higher scores. That is, scaling the quality of each individual is more budget-efficient than scaling the number of evolutionary generations. Surprisingly, more capable models produced evaluation hacks at higher rates, making hack detection increasingly necessary as models scale. 1 Introduction Scientific and engineering progress is often driven by the discovery of superior algorithms. Yet such algorithm design still relies heavily on manual effort by domain experts, and AI-driven automation of discovery is beginning to change this landscape ( 19 ) . In particular, the approach of using large language models (LLMs) as mutation operators in evolutionary search, iteratively creating algorithms that surpass human designs through cycles with an automatic evaluator, has achieved remarkable results. FunSearch evolved function-level algorithms with LLMs, generating mathematical constructions surpassing the best known results for the cap set problem ( 13 ) . Beyond improving algorithms alone, alternately improving both the generated algorithms and the generating model itself through reinforcement learning has also been explored ( 4 ) . More recently, AlphaEvolve ( 11 ) expanded the scope of improvement to entire codebases and improved matrix multiplication algorithms that had remained unimproved for over 50 years. This technology is applicable to any domain where algorithms play a central role, including route optimization, chip placement design, and molecular generation in drug discovery, and may replace manual expert optimization. What determines the success of algorithm discovery is not agent capability alone; the design of the agent infrastructure, i.e., the harness, has a substantial impact. The harness refers to the entire execution infrastructure that guides agents toward discovery, encompassing prompt construction, data design, evaluation pipelines, and parallel agent management. AlphaEvolve’s success is supported not only by Gemini’s model capability but also by the harness design as a whole, including randomized prompt switching, evaluation cascades, and an evolutionary database combining MAP-Elites with island-based population models. Examining existing open-source implementations from this harness perspective reveals substantial room for improvement. Representative implementations such as OpenEvolve 1 1 1 https://github.com/algorithmicsuperintelligence/openevolve and CodeEvolve ( 2 ) all use LLMs as stateless code generators through single-shot API calls, without leveraging the capabilities of coding agents that have rapidly advanced since 2025. Just as coding agents such as Claude Code 2 2 2 https://github.com/anthropics/claude-code and Codex CLI 3 3 3 https://github.com/openai/codex have transformed productivity in software development, differences in harness implementation may fundamentally change the practicality of algorithm discovery. From these observations, we identify four practical challenges in existing harnesses. First, LLMs remain stateless code generators, without leveraging autonomous reasoning such as analyzing evaluation results, referencing the entire codebase, or planning multi-step corrections. Second, there is no mechanism to detect “evaluation hacking” ( 1 ) , where generated algorithms exploit flaws in the evaluation function rather than genuinely solving the problem. Third, running multiple search agents in parallel on a shared filesystem risks file conflicts and race conditions that compromise reproducibility. Fourth, each iteration starts from a blank slate, without systematically leveraging knowledge of successes and failures from past trials. Beyond these challenges, a fundamental question about search strategy remains unanswered. Algorithm discovery is inherently a search problem whose success scales with the budget invested. The practical constraint is that every search step costs tokens, so token efficiency directly determines the scalability of discovery. Under a fixed token budget, is it better to produce many algorithms with brief thought, or fewer algorithms with deeper thought? Whether investing more thought per algorithm at the expense of fewer generations improves overall discovery performance requires empirical verification. This paper proposes Vesper, a harness for LLM-driven algorithm discovery that integrates several candidate improvements, including coding agent integration ( § 4.2 ), evaluation hack detection ( § 4.3 ), and Git worktree isolation ( § 4.5 ), and empirically investigates which components matter and why. Interestingly, scaling the quality of each individual proved more budget-efficient than scaling the number of evolutionary generations ( § 5 ). We also find, surprisingly, that more capable frontier models such as GPT-5.2 are more prone to evaluation hacking. 2 Related Work LLM-Driven Algorithm Discovery The dominant paradigm in LLM-driven algorithm discovery shares the common approach of incorporating LLMs into evolutionary operators as stateless generators that return single-shot code completions in response to prompts. FunSearch ( 13 ) surpassed the best known results on the cap set problem, and AlphaEvolve ( 11 ) improved matrix multiplication algorithms that had remained unimproved for over 50 years. Numerous specialized frameworks have also emerged, including EoH ( 7 ) , ReEvo ( 17 ) , LLaMEA ( 15 ) , and AEL ( 8 ) . In open source, OpenEvolve reproduces AlphaEvolve’s workflow, followed by CodeEvolve ( 2 ) and DeepEvolve ( 9 ) . Coding Agents Since 2025, command-line coding agents that operate autonomously have emerged in rapid succession, including Codex CLI, Claude Code, Gemini CLI 4 4 4 https://github.com/google-gemini/gemini-cli , and Qwen Code 5 5 5 https://github.com/QwenLM/qwen-code . These agents complete repository-wide reading, file editing, test execution, and debugging within a single session, with SWE-bench ( 5 ) driving their development as an evaluation platform. Unlike single-shot API calls, coding agents can autonomously perform multi-step reasoning, including analyzing existing code, hypothesis testing based on execution results, and iterative error correction, giving them fundamentally different potential as evolutionary operators. Vesper places the use of coding agents, rather than stateless API calls, at the center of its harness design. Figure 1 : Vesper’s evolutionary loop (top) and details of each harness improvement (bottom). Dotted borders indicate components shared with existing pipelines; solid borders indicate harness improvements introduced by Vesper. The following cycle repeats until the token budget is exhausted. (1) Select a parent: sample a parent program (Git branch) from the program database. (2) Set up environment: create a Git worktree from the parent branch, providing each agent with an isolated execution environment ( § 4.5 ). (3) Improve programs: a coding agent autonomously improves code while referencing the program database ( § 4.2 , § 4.4 ). (4) Evaluate algorithm: compute the score of the improved algorithm. (5) Detect hacks: a secondary agent reviews candidate programs and rejects those that exploit the evaluation function ( § 4.3 ). (6) Update Program DB: store validated programs with scores, summaries, and improvement ideas in the program database. The accumulated trial history enables subsequent agents to analyze which approaches succeeded