Consistency Amplifies: How Behavioral Variance Shapes Agent Accuracy

Aman Mehta · 2026

Read Consistency Amplifies: How Behavioral Variance Shapes Agent Accuracy by Aman Mehta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25764.txt.

Introduction

Mehta studies whether coding agents can be trusted, not just whether they finish tasks. He runs four frontier models over 1,750 runs on 50 SWE-bench Verified tasks. GPT-5 submitted a patch every time but solved only 44 percent of tasks. Gemini submitted least often, at 70 percent, yet solved 50 percent. The main failure mode is what Mehta calls silent semantic failure. The agent repeats the same wrong reading of the task with full confidence. This mode covers most failing runs and stays hidden from simple monitoring.

Catalog record

CREATORAman Mehta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30546
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25764

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Consistency Amplifies: How Behavioral Variance Shapes Agent Accuracy Aman Mehta Abstract As LLM-based agents are deployed in production systems, understanding their behavioral consistency (whether they produce similar action sequences when given identical tasks) becomes critical for reliability. We study consistency in the context of SWE-bench, a challenging software engineering benchmark requiring complex, multi-step reasoning. Comparing Claude 4.5 Sonnet, GPT-5, and Llama-3.1-70B across 50 runs each (10 tasks × \times 5 runs), we find that across models, higher consistency aligns with higher accuracy: Claude achieves the lowest variance (CV: 15.2%) and highest accuracy (58%), GPT-5 is intermediate (CV: 32.2%, accuracy: 32%), and Llama shows the highest variance (CV: 47.0%) with lowest accuracy (4%). However, within a model, consistency can amplify both correct and incorrect interpretations. Our analysis reveals a critical nuance: consistency amplifies outcomes rather than guaranteeing correctness . 71% of Claude’s failures stem from “consistent wrong interpretation”: making the same incorrect assumption across all runs. Interestingly, GPT-5 achieves similar early strategic agreement as Claude (diverging at step 3.4 vs 3.2) but exhibits 2.1 × \times higher variance, suggesting that divergence timing alone does not determine consistency. These findings suggest that for production deployment, interpretation accuracy matters more than execution consistency, with implications for agent evaluation and training. 1 Introduction The deployment of LLM-based agents in production systems (from code assistants to autonomous research tools) has accelerated dramatically. Yet a fundamental question remains underexplored: when given the same task multiple times, do these agents behave consistently? Behavioral consistency matters for several reasons. First, inconsistent agents are unpredictable, making them difficult to trust in high-stakes applications. Second, variance in agent behavior complicates debugging and improvement; if an agent sometimes succeeds and sometimes fails on identical inputs, isolating the cause becomes challenging. Third, consistency is a prerequisite for meaningful benchmarking: if results vary substantially across runs, single-run evaluations may be misleading. Prior work has established that LLMs exhibit significant variance in simple reasoning tasks ( 7 ) . However, the relationship between consistency and task complexity remains unclear. Do consistency challenges grow with action space size? Does the variance in simple tasks predict variance in complex ones? We investigate these questions using SWE-bench ( 4 ) , a benchmark requiring agents to resolve real GitHub issues through multi-step code modifications. SWE-bench tasks demand exploration of large codebases, understanding of bug context, implementation of fixes, and verification, representing a substantial increase in complexity from prior consistency studies. Our contributions are: 1. Quantitative characterization across capability tiers: We find that consistency aligns with accuracy across three models: Claude 4.5 Sonnet (CV: 15.2%, accuracy: 58%), GPT-5 (CV: 32.2%, accuracy: 32%), and Llama-3.1-70B (CV: 47.0%, accuracy: 4%). 2. The amplification insight: We show that consistency amplifies outcomes rather than guaranteeing correctness. 71% of Claude’s failures are “consistent wrong interpretation”: the same incorrect approach across all runs. 3. The speed-accuracy-consistency tradeoff: GPT-5 is 4.7 \times faster than Claude (9.9 vs 46.1 steps) but achieves 1.8 \times lower accuracy and 2.1 \times worse consistency, revealing a fundamental tradeoff. 4. The fixation failure mode: Through detailed trajectory analysis, we identify cases where Claude’s thoroughness causes fixation on incorrect interpretations, while less thorough approaches occasionally allow course correction. 5. Divergence timing vs consistency: Despite similar divergence timing (Claude: step 3.2, GPT-5: step 3.4), Claude achieves 2.1 \times better consistency, showing that early agreement alone does not determine behavioral variance. These findings have implications for agent deployment: consistency alone is insufficient for reliability; interpretation quality is the bottleneck. 2 Related Work LLM Agent Benchmarks. SWE-bench ( 4 ) evaluates agents on real GitHub issues, requiring multi-step reasoning and code modification. Other benchmarks include WebArena ( 11 ) for web navigation, OSWorld ( 10 ) for computer use, and various coding benchmarks ( 2 ; 1 ) . Most report single-run accuracy, leaving consistency unexplored. Recent work on SWE-bench Verified ( 5 ) provides human-validated solutions, enabling more reliable evaluation. LLM Consistency and Reliability. Prior work has examined consistency in simpler settings: mathematical reasoning ( 9 ) , factual questions ( 3 ) , and multi-hop QA ( 7 ) . These studies find substantial variance even in straightforward tasks. Self-consistency methods ( 9 ) leverage this variance through majority voting, but assume independent samples, an assumption that may not hold for complex agent trajectories. We extend this line of work to complex, multi-step agent behavior where actions are sequentially dependent. Agent Failure Analysis. Studies of agent failures have focused on error categorization ( 6 ) and recovery mechanisms ( 8 ) . Reflexion ( 8 ) shows that agents can improve through verbal self-reflection, but assumes failures are detectable. Our work contributes a new perspective: systematic analysis of when in the trajectory failures originate, and how consistency interacts with correctness. We find that many failures are “consistent wrong”; the agent confidently repeats the same mistake, making reflection-based recovery unlikely. 3 Experimental Setup 3.1 Benchmark and Tasks We use SWE-bench Verified, a curated subset of SWE-bench with human-validated solutions. We select 10 tasks from the astropy repository, chosen for diversity in bug types and fix complexity. Table 1 summarizes the task characteristics. Table 1: Task characteristics. Tasks vary in bug type, fix complexity (lines changed), and file count. Task ID Bug Type Fix Size Files 12907 Logic error 3 lines 1 13033 Missing check 5 lines 1 13236 Silent conversion 4 lines 1 13398 Edge case 8 lines 2 13453 Type handling 6 lines 1 13579 Format string 2 lines 1 13977 Deprecation 12 lines 2 14096 Boundary check 7 lines 1 14182 Default value 4 lines 1 14309 Import error 3 lines 1 Each task requires the agent to understand a GitHub issue, locate relevant code, implement a fix, and verify correctness. The median fix size is 4.5 lines, but finding the right location and understanding the bug context requires substantial exploration. 3.2 Models We compare three models representing different capability levels: • Claude 4.5 Sonnet ( claude-sonnet-4-5-20250929 ): A frontier model known for strong coding capabilities, with extended context and tool use. • GPT-5 ( openai-gpt-5 , February 2026): OpenAI’s frontier model with strong reasoning capabilities, available through the OpenAI API. • Llama-3.1-70B-Instruct : An open-weights model, smaller but widely deployed in production systems. All models have access to identical tools: bash commands for file system navigation, code editing, and test execution. We use the same system prompt and tool definitions for all. Agent Framework. We use mini-SWE-agent ( 4 ) , a minimal agent scaffold that provides only a bash interface (no tool-calling API). Each action is executed via subprocess.run in an isolated Docker container. During development, we identified and fixed a format-error bug where agents could inadvertently submit during error recovery; patch submission is now blocked during format-error handling. Our framework and fix are available in the supplementary code. 3.3 Experimental Protocol For each model-task pair, we run 5 independent trials with: • Temperature: 0.5 (moderate stochasticity) • Maximum steps: 250 • Isolated Docker containers per run • Identical prompts and tool access • Fresh repository state for each run This yields 50 runs per model (10 tasks \times 5 runs), totaling 150 agent trajectories across three models. We set a per-run cost limit of $10 (estimated from API

Read the complete text