It’s Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
Yong-eun Cho · 2026
Read It’s Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers by Yong-eun Cho (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26731.txt.
Introduction
Cho's 2026 experiment tests a common deployment belief: more structured harnesses always help language-model agents, and stronger models need less guidance. Across 432 runs spanning six models, four capability tiers, and three harness styles, the belief fails both ways. Verbose harnesses sharply cut one frontier chat model's success rate, while strict harnesses best suit a frontier reasoning model. Harness choice, he argues, must follow model type rather than capability alone.
Catalog record
| CREATOR | Yong-eun Cho |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34061 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26731
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
It’s Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers Yong-eun Cho Abstract A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability, and that higher-capability models need proportionally less structural guidance—together implying a monotone inverse relationship between model capability tier and optimal harness complexity. We test this hypothesis through a controlled 432-run experiment crossing six models across four capability tiers with three harness conditions ( light , balanced , strict ) on HEAT-24, a 24-task synthetic benchmark with git-based workspace verification. Our results refute the monotone inverse relationship on two fronts. First, for the frontier chat model evaluated (Gemini 2.5 Flash), increased harness verbosity lowers VTSR by 29–38 percentage points—a harness-complexity paradox . Second, for the frontier reasoning model evaluated (Qwen3.5-122B, extended thinking enabled), strict harness achieves the highest VTSR (91.7%) and the lowest latency, the opposite of the prediction. Within the constrained tier, a 2 B model (Gemma4:e2B) matches strong-open-tier stability at 91.7% across all harnesses. Because each tier is represented by a single model in this study, these results should be interpreted as model-specific observations; harness sensitivity appears non-monotone across the models evaluated, and depends critically on model type (chat vs. reasoning). We introduce a six-label failure taxonomy showing that format_violation dominates capable-model failures while wrong_file dominates low-capability failures, and we derive practical tier-aware harness selection guidelines. 1 Introduction Autonomous LLM agents that read, reason about, and modify workspace artifacts are increasingly deployed in software engineering ( 7 ; 4 ) , document processing, and operational workflows. The quality of the harness —the system-level prompt that specifies task scope, allowed operations, output format, and verification procedure—is widely believed to be a primary lever for improving agent reliability ( 14 ; 12 ) . Existing benchmarks evaluate agents on a fixed harness and report aggregate accuracy, obscuring the interaction between harness complexity and model capability. Practitioners routinely apply strict, highly-structured harnesses to all models in a deployment fleet under two implicit assumptions: that more structure always improves reliability, and that higher-capability models need less structural guidance—forming a monotone inverse relationship between capability tier and optimal harness complexity. We ask whether this monotone inverse hypothesis holds empirically across a diverse set of capability tiers, and whether the answer differs between chat-oriented and reasoning-oriented frontier models. We make the following contributions: • HEAT-24 ( H arness E valuation for A gent T asks), a deterministic 24-task synthetic benchmark with workspace-level git-based verification covering six task categories. • The first controlled empirical test of the monotone inverse capability-harness hypothesis, crossing six models across four capability tiers with three harness conditions (432 total runs). • Evidence that the hypothesis fails in opposite directions simultaneously: a harness-complexity paradox (strict harness hurts the frontier chat model) and a non-monotonic pattern (strict harness helps the frontier reasoning model most). • The finding that parameter count is an unreliable proxy for harness sensitivity: a 2 B model (Gemma4:e2B) matches strong-open-tier stability, demonstrating that instruction-tuning quality is the true moderating variable. • A six-label failure taxonomy and practical tier-aware and type-aware harness selection guidelines. 2 Related Work LLM agent benchmarks. 7 evaluate LLMs across eight interactive environments and show strong capability gaps between frontier and open-source models. 13 demonstrate that chain-of-thought reasoning only reliably emerges above a model-size threshold, suggesting that structured prompts may impose undue cognitive overhead on smaller models—a pattern we observe in our constrained tier. Instruction following and format compliance. 8 survey instruction-following capabilities and find that compliance degrades as instruction complexity increases, directly motivating our harness-complexity investigation. 11 show that subtle changes in prompt formatting cause up to 76-point performance differences across open-source models, establishing that prompt structure is a major source of variance—a concern we directly investigate at the harness level. 9 argue that single-prompt evaluations are brittle and call for multi-prompt evaluation, supporting our cross-harness design. Work on structured output generation ( 2 ) shows that JSON and schema compliance varies substantially across model families, motivating our format-sensitive task category. 1 find that separating task-solving from output formatting improves both dimensions, consistent with the format violations we observe when elaborate process instructions are mixed with output format requirements. Agent scaffolding. ReAct ( 14 ) and Reflexion ( 12 ) demonstrate that structured reasoning-action loops improve agent performance. Our study complements this by asking whether different levels of harness structure suit different model tiers, and whether extended thinking modes interact with harness structure in predictable ways. Prompt complexity and performance inversion. 10 provide a systematic survey of prompting techniques and catalog conditions under which prompt engineering improves or degrades performance, providing a broad empirical context for our harness-complexity findings. 3 find that brevity constraints on model outputs reverse performance hierarchies across model scales—larger models become relatively worse when forced to be concise. 5 argue that increasing prompt specificity can invert the expected performance ordering, with simpler prompts outperforming engineered ones for capable models. Both findings align with our harness-complexity paradox: the relationship between instruction richness and task success is non-monotonic and tier-dependent. Self-correction and error recovery. 6 decompose LLM self-correction and find an accuracy–correction paradox where stronger models make “deeper” errors that resist self-correction, while weaker models make more tractable surface errors—a pattern partially consistent with our failure taxonomy. 3 The HEAT-24 Benchmark 3.1 Workspace and Task Design All tasks operate on a shared synthetic workspace containing twelve files: configuration YAML and JSON, Python source and test files, Markdown documentation, a CSV data file, and changelog fragments. The workspace is initialized as a git repository before each run; verification uses git diff to detect and scope file changes. All twelve workspace files are injected into every harness prompt, enabling models to read any file without external tool access. We define 24 tasks across six categories (Table 1 ): Category # Tasks inspect_local 4 structured_edit 4 format_sensitive 4 verification_recovery 4 repair 4 multi_step_ops 4 Table 1: HEAT-24 task categories (4 tasks each; 24 total). inspect_local : read files, return JSON; structured_edit : modify one file; format_sensitive : emit strict JSON schema; verification_recovery : fix bug, run tests; repair : correct malformed content; multi_step_ops : coordinate multiple files. Each task has a deterministic binary verifier. Tasks are designed to have deterministic, binary outcomes. Verifiers check: JSON key presence and values, git-scoped file modifications, YAML/JSON parse validity, and substring presence. File modifications expressed in model output use a structured <<<WRITE:path>>> \cdots <<<END>>> marker that the harness runner parses and applies to the workspace before verification. This marker format is included in the raw task instruction for all file-modification tasks across all three harness conditions; harness conditions differ only in the additional process instructions, scope constraints, and verification specifications layered on top. 3.2 Harness Conditions We define three harness conditions of increasing structural complexity: Light A two-line prompt: role statement plus the raw task instruction. No format specification, no scope constraint, no verification procedure. Balanced Adds a four-step process template (plan, execute, check, respond) and lists the allowed files. No schema or verification spec. Strict Adds six explicit stages (preflight / plan / execute / verify / recover / report), an allowed-file list, explicit success criteria, a verification specification, and instructions to express file changes using the <<<WRITE:path>>> marker.