AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents
Hailin Zhong, Shengxin Zhu · 2026
Read AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents by Hailin Zhong, Shengxin Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13357.txt.
Introduction
Zhong and Zhu argue that software agent skill lives in the runtime harness, not the model alone. The harness shapes what the agent sees, which tools it can use, how feedback returns, and how completion is proven. They name eleven harness duties and grade support on a four level ladder from bare patch output to full audit trails. Applied to a validation task, higher levels yield reproduction logs, failure causes and deterministic checks, reframing agent evaluation around verifiable change.
Catalog record
| CREATOR | Hailin Zhong, Shengxin Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47554 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13357
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents Hailin Zhong, Shengxin Zhu Abstract Foundation models have transformed automated code generation, yet autonomous software-engineering agents remain unreliable in realistic development settings. The dominant explanation locates this gap in model capability. We propose a different locus: software-engineering capability emerges from a model–harness–environment system , in which a runtime substrate—the harness—mediates how a foundation-model agent observes a project, acts on it, receives feedback, and establishes that a change is complete. We formalize this substrate as an AI Harness Engineering and identify eleven component responsibilities: task specification, context selection, tool access, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing, and intervention recording. We operationalize the harness through a four-level ladder (H0–H3) that progressively exposes runtime support to the agent, and we propose a trace-based evaluation protocol that converts each agent run into an auditable episode package. Applied to a controlled validation task, the framework yields episode packages whose evidence structure varies systematically with harness level: lower levels produce only a final patch, higher levels produce reproduction logs, failure attributions, deterministic requirement checks, and structured verification reports. The framework reframes the central question of autonomous software engineering from whether a foundation model can produce a patch to whether the model–harness–environment system can produce a verifiably correct, attributed, and maintainable change. We outline a research program for the runtime systems that foundation-model software agents will require. Keywords: harness engineering; foundation models; autonomous software engineering; runtime systems; agent evaluation; verification; software-engineering agents. Introduction Foundation models have rapidly become capable programming assistants [ 3 , 2 ] . They generate functions, modify files, explain code, write tests, invoke tools, and interact with software repositories. This progress has energized an ambition that predates the current generation of models but has only recently become plausible: autonomous software-engineering agents that take a high-level development task and carry it through implementation, testing, verification, and maintenance with minimal human supervision. Recent benchmarks and agent systems [ 5 , 16 , 12 , 15 , 18 ] show that the gap between local code generation and complete software work is real and persistent. A model that writes a correct local patch may still fail to complete a task. It may inspect the wrong files, apply a surface-level patch to a user interface while the underlying API behavior remains broken, run the wrong tests, misinterpret a failure, forget task state, leave behind obsolete artifacts, or declare success without sufficient verification. Humans remain involved not primarily because they are writing every line of code, but because they provide the missing runtime support: they identify relevant context, explain repository structure, select tools, interpret feedback, enforce architectural boundaries, verify behavior, and clean up residue. The dominant framing of this gap locates it in model capability. On that view, an agent that fails on a software task does so because the model is not yet capable enough at coding, reasoning, planning, or tool use, and the field’s task is to train better models or compose them into more sophisticated agent loops [ 11 , 13 , 17 ] . We do not contest that model capability matters. We argue that this framing is incomplete. Software engineering is a long-horizon, stateful, tool-mediated, feedback-driven activity. It depends on context management, project memory, tool interfaces, execution traces, validation signals, permissions, rollback, and maintenance discipline. When a foundation model is placed inside a development environment designed primarily for human developers, many of these supports remain implicit, inaccessible, or unstable for the agent. The components that human developers acquire through socialization, documentation, and experience are not freely available to a model invocation; they must be exposed, structured, and traced. Where they are not, the agent either improvises or asks the human to fill the gap. We therefore propose a different framing. Autonomous software-engineering capability is an emergent property of a model–harness–environment system , not of the model alone: C_{\text{system}}\;=\;F\!\bigl(C_{\text{model}},\,C_{\text{harness}},\,C_{\text{environment}},\,T\bigr), where C_{\text{model}} is the foundation model’s latent capability, C_{\text{environment}} is what the software environment exposes, C_{\text{harness}} is the runtime substrate that mediates between them, and T is the task distribution. We call this substrate an AI Harness Engineering : a runtime layer that surrounds a foundation-model software agent and manages context, tools, project memory, task state, observability, failure attribution, verification, permissions, and maintenance state. The harness is what determines whether latent model capability becomes auditable software-engineering behavior. Contributions. This paper makes four contributions. (i) We define the AI Harness Engineering as a new research object distinct from agent–computer interfaces [ 16 ] , agent frameworks [ 14 , 1 ] , and agent operating systems [ 8 ] , and we identify its eleven component responsibilities and five design principles. (ii) We propose the H0–H3 harness ladder , a controlled-visibility ablation that exposes progressively more runtime support to the agent and makes the harness’s contribution empirically separable from the model’s. (iii) We define a trace-based evaluation protocol that records eight classes of execution evidence—action, tool, context, verification, failure attribution, intervention, entropy, and outcome—and adjudicates each agent run by verification autonomy rather than task success alone. (iv) We instantiate the framework on a controlled validation task and show that the resulting episode packages differ systematically in evidence structure across harness levels, with the highest level producing reproduction logs, failure attributions, requirement-level verification, and structured verification reports that lower levels do not. Why now. Industrial development practice has independently begun to converge on harness-like structures around coding agents. Reports from OpenAI on Codex and from Microsoft on agent harnesses describe context management, repository knowledge, observability, tool interfaces, feedback loops, and human attention as first-class concerns of agent deployment [ 10 , 9 ] . These practitioner accounts confirm that something like a harness exists and matters. They do not, however, treat the harness as a research object. They do not define its components, expose its support as a controlled ablation, or specify the evidence that an episode should produce. This paper does. A runtime view of autonomous software engineering From coding ability to software-engineering capability. A foundation model that generates correct code fragments, explains an existing function, or proposes a patch for a localized bug exhibits coding ability . Software-engineering capability is more than this. It is a stateful process involving repository navigation, context selection, tool use, test execution, failure interpretation, verification, documentation, and maintenance. The two are commonly conflated: a model that performs well on isolated code-generation benchmarks is taken to be a competent software-engineering agent. The performance gap between local benchmarks and realistic software tasks [ 5 , 7 ] indicates otherwise. The unit of analysis. Treating the model alone as the unit of analysis produces a characteristic attribution error. A failed autonomous episode is read as a model failure; a successful episode is credited to the model. But the model rarely acts alone in a realistic setting. It receives a representation of the task, observes a subset of the repository, invokes tools through an interface, receives feedback from tests or commands, and decides when the task is complete. Every one of these steps is mediated by runtime structure. When that structure is present and well-designed, the system performs as if the model is competent. When it is absent or