Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, Tong Yang · 2026

Read Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows by Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, Tong Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27922.txt.

Introduction

Yao and colleagues show that an AI agent's harness, the layer managing tools, context, state, and recovery, matters as much as the model itself. Their Harness-Bench runs a hundred six realistic sandboxed tasks across many model-harness pairings, logging artifacts, traces, and failures. Capability varied widely by pairing, so results should be reported per configuration, not per model alone.

Catalog record

CREATORYilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, Tong Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43028
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27922

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, Tong Yang Abstract LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench , a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness’s native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model--harness pairings. These results suggest that agent capability should be reported at the model--harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks. 1 1 1 Our code and data are available at https://github.com/Qihoo360/harness-bench . Additional resources and updates can be found on our project website at http://www.harness-bench.ai/ . 1 Introduction Large language models are increasingly deployed as agents that act in external environments, using tools, modifying workspaces, and producing artifacts that satisfy concrete user requirements ( 21 ; 16 ) . In such executable workflows, practical performance depends not only on the underlying model, but also on the system layer that turns model capability into executable action. We refer to this layer as a harness : the mechanism that organizes context, tools, state, permissions, constraints, and recovery to mediate between model outputs and external actions. Harnesses are therefore central to agent system design: they shape how model capability is exposed, constrained, and realized, affecting completion, cost, safety, robustness, and auditability ( 20 ) . Existing benchmarks have advanced LLM and agent evaluation across static reasoning, executable environments, and standardized workflow settings. Static benchmarks such as MMLU ( 3 ) , GSM8K ( 2 ) , BIG-bench ( 17 ) , and HELM ( 7 ) measure text-based model capabilities, while agent benchmarks such as SWE-bench ( 5 ) , WebArena ( 26 ) , OSWorld ( 19 ) , and Terminal-Bench ( 10 ) evaluate complete systems in executable environments. Workflow-oriented and assistant-agent benchmarks such as AgentBench ( 8 ) , GAIA ( 11 ) , and Claw-Eval ( 22 ) further compare model backends under shared execution setups. However, the harness itself remains largely unmeasured: existing benchmarks either abstract away execution, conflate the harness with the full agent system, or fix the harness when comparing models. As a result, we lack a diagnostic protocol for studying how model–harness configurations affect success, token cost, robustness, and traceability in realistic workflows. We introduce Harness-Bench , a diagnostic benchmark for studying configuration-level harness effects in realistic agent workflows. Recent work on agent-computer interfaces and agent-evaluation infrastructure reflects growing interest in execution-layer design ( 5 ; 6 ) . However, existing efforts typically evaluate a particular agent system, standardize evaluation infrastructure, or compare heterogeneous agent stacks, rather than systematically varying harness configurations across shared task environments and model backends. To our knowledge, Harness-Bench is among the first benchmarks to make the harness a primary axis of evaluation under common external task conditions. Rather than forcing all systems into an identical internal implementation, Harness-Bench fixes the task environment, budget, timeout, and evaluator while preserving each harness’s native execution behavior. The resulting measurements should therefore be interpreted as configuration-level diagnostics of model–harness pairings, not as causal decompositions of individual harness mechanisms. Each run records execution evidence, enabling analysis beyond final completion scores. Harness-Bench contains 106 realistic, end-to-end agent tasks constructed from practical agent-use patterns and common user requests, each executed in its own sandboxed offline environment with task-specific configuration and evaluation criteria. The task suite is manually reviewed for realism, difficulty, solvability, and evaluation reliability. These environments emulate practical agent settings while avoiding dependence on live services, reducing benchmark drift and making runs reproducible and independently scorable. The tasks require agents to complete concrete workflows rather than isolated tool calls. They span diverse execution demands, including workspace/tool operation, software engineering, data analysis, evidence-grounded knowledge work, and permission-sensitive, stateful, or long-horizon operational workflows. This design preserves realism while providing enough difficulty and diversity to expose meaningful differences across harnesses. We make three main contributions. (1) Benchmark asset. We introduce Harness-Bench, a suite of 106 sandboxed offline tasks for evaluating realistic end-to-end agent workflows with task manifests, fixtures, evaluators, and execution traces. (2) Evaluation protocol. We define a model–harness evaluation protocol that fixes external task conditions, budgets, timeouts, and evaluators while preserving each harness’s native execution behavior, enabling configuration-level comparison across representative harnesses and model backends. (3) Diagnostic analysis. Across 5,194 execution trajectories, we analyze completion, process quality, efficiency, and recurring failure symptoms. Our results show that performance varies across model–harness pairings and support reporting agent capability at the configuration level rather than attributing it to the base model alone. 2 Related Work LLM and agent benchmarks. LLM evaluation has progressed from static language and reasoning benchmarks to executable agent benchmarks. Static benchmarks such as MMLU ( 3 ) , GSM8K ( 2 ) , BIG-bench ( 17 ) , and HELM ( 7 ) measure model capabilities in text-based settings, while agent benchmarks such as SWE-bench ( 5 ) , Terminal-Bench ( 10 ) , WebArena ( 26 ) , and OSWorld ( 19 ) evaluate agents in software, terminal, web, and operating-system environments. More recent workflow-agent benchmarks, including AgentBench ( 8 ) , GAIA ( 11 ) , and Claw-Eval ( 22 ) , further emphasize multi-step execution, external state, traceability, safety, robustness, and cost-aware evaluation. ClawMark ( 9 ) pushes this direction to long-horizon, multimodal coworker settings, coupling multi-turn and multi-day tasks with persistent tool-backed services and drifting external state. ClawBench ( 24 ) instead stress-tests agents on everyday online workflows over many live production websites, highlighting gaps between sandboxed web benchmarks and real-site complexity. These benchmarks are essential for measuring model or end-to-end agent capability, but they do not directly evaluate the harness as the variable of interest: they either abstract away execution, evaluate a complete submitted agent stack, or hold the execution setup fixed to compare models. Harness-Bench is complementary: it controls external task conditions while varying harness configurations, enabling diagnostic comparison of completion, token cost, execution safety, robustness, and traceability. Harnesses and harness engineering. Recent agent systems increasingly treat the model as one component of a larger execution stack. Work on agent-computer interfaces ( 20 ) , tool-use protocols such as the Model

Read the complete text