Three Roles, One Model: Role Orchestration at Inference Time to Close the Performance Gap Between Small and Large Agents
S. Aaron McClendon, Jorge Gallego-Feliciano, Stavros Zervoudakis, Antonios Saravanos · 2026
Read Three Roles, One Model: Role Orchestration at Inference Time to Close the Performance Gap Between Small and Large Agents by S. Aaron McClendon, Jorge Gallego-Feliciano, Stavros Zervoudakis, Antonios Saravanos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11465.txt.
Introduction
McClendon and three colleagues close the small-large agent gap with inference-time role orchestration. Raw Qwen3-8B manages just 5.4 percent completion at full precision and 3.0 percent quantized on AppWorld. Their three-role scaffold uses the same frozen model to compress dialogue history, to act as main agent, and to review code output in isolation. That isolation breaks failure loops. Completion rises to 8.9 and 5.9 percent, roughly doubling performance. The scaffolded 8B model at full precision surpasses DeepSeek-Coder 33B Instruct at 7.1 percent.
Catalog record
| CREATOR | S. Aaron McClendon, Jorge Gallego-Feliciano, Stavros Zervoudakis, Antonios Saravanos |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48711 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11465
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Three Roles, One Model: Role Orchestration at Inference Time to Close the Performance Gap Between Small and Large Agents S. Aaron McClendon, Jorge Gallego-Feliciano, Stavros Zervoudakis, Antonios Saravanos Abstract Large language model (LLM) agents show promise on realistic tool-use tasks, but deploying capable agents on modest hardware remains challenging. We study whether inference-time scaffolding alone, without any additional training compute, can improve the performance of a small model in complex multi-step environments. Operating on a single 24GB GPU, we evaluate Qwen3-8B on the AppWorld benchmark under both full-precision (FP16, 12K context) and 4-bit quantized (AWQ, 32K context) configurations. Without any intervention, the raw model achieves just 5.4% (FP16) and 3.0% (AWQ) task goal completion. Guided by a systematic failure mode analysis, we introduce a three-tier inference scaffolding pipeline that deploys the same frozen model in three distinct roles: (1) a summarization model that preserves critical artifacts (tokens, credentials, API responses) while compressing dialogue history; (2) the main agent model that reasons over the compressed context; and (3) an isolated correction model that reviews and revises the agent’s code output without access to conversation history, breaking repetitive failure loops. Applied to the same unmodified model, this scaffolding yields 8.9% (FP16) and 5.9% (AWQ) task goal completion, roughly doubling performance in both settings, with particularly strong gains on difficulty-1 tasks (15.8% → \to 26.3% FP16; 5.3% → \to 14.0% AWQ). On full-precision inference, our scaffolded 8B model surpasses DeepSeek-Coder 33B Instruct (7.1%) from the original AppWorld evaluation, demonstrating that structured inference-time interventions can make small models competitive with systems 4 × \times their size. We formalize the approach as a scaffolded policy over a frozen base model, three invocations of the same weights with different conditioning, drawing connections to test-time compute scaling and action-space shaping in reinforcement learning. Introduction Tool-augmented language models are increasingly used as agents that can reason over intermediate observations, invoke external tools, and act in software environments [ 10 , 27 , 16 ] . Their promise is especially clear for realistic digital tasks, where success depends not only on generating a plausible next action, but also on maintaining state across long trajectories, consulting documentation, handling authentication, recovering from execution errors, and adapting plans as new observations arrive. While many recent benchmarks evaluate tool use or reasoning in controlled settings, they often abstract away the constraints that dominate real deployments. In practice, language model agents must operate under limited hardware budgets, maintain state across long interaction histories, handle authentication and API failures, and recover from execution errors. We refer to this setting as deployment realism, specifically, evaluating agents under the same resource constraints, execution dynamics, and failure modes they encounter in real-world software environments. AppWorld [ 21 ] was designed to capture this regime. It evaluates agents on realistic multi-step tasks spanning nine interconnected applications and 457 API endpoints, requiring persistent state management, correct API usage, and robust recovery from intermediate failures. This makes AppWorld not merely a tool-use benchmark, but a test of whether a language model can function as a persistent software agent under realistic deployment constraints. Performance on AppWorld, however, remains strongly associated with model scale. In the original benchmark evaluation, GPT-4o achieves 48.8% task goal completion on test_normal , whereas smaller open-weight models lag substantially: DeepSeek-Coder 33B Instruct reaches 7.1%, LLaMA-3-70B-Instruct 20.8%, and Mistral 7B 0.0% [ 21 ] . This gap is practically important. Many academic labs and smaller organizations cannot fine-tune or serve frontier-scale systems, and instead must operate within the memory and compute budget of a single consumer-grade GPU. In that regime, methods that improve a frozen model at inference time, rather than through additional training, are especially attractive. Prior work suggests that structured test-time computation can materially improve model behavior without updating parameters. Chain-of-thought prompting, self-consistency, and tree-of-thought reasoning show that additional inference-time structure can improve reasoning performance [ 24 , 23 , 26 ] . More recent work extends this idea to self-correction and memory-aware inference in multi-step settings. These results motivate a concrete question for agentic systems: can carefully designed inference-time scaffolding make a small model materially more effective in a realistic tool-use environment, even when the underlying weights remain fixed? We investigate this question using Qwen3-8B [ 20 ] on AppWorld under a strict single-GPU deployment budget (24 GB VRAM). We evaluate two configurations: full-precision FP16 with a truncated 12K-token context window, and 4-bit AWQ quantization with the full 32K context window [ 11 ] . Baseline performance is weak in both settings: the raw model achieves 5.4% task goal completion in FP16 and 3.0% in AWQ. A trajectory-level failure analysis shows that these failures are not explained by parameter count alone. Instead, the model frequently loses critical state, mishandles credentials, violates API schemas, and becomes trapped in repetitive correction loops after early mistakes. Guided by this analysis, we design a three-tier inference scaffolding pipeline that deploys the same frozen model in three distinct roles. A summarization module compresses dialogue history while preserving critical artifacts such as tokens, credentials, intermediate API outputs, and error resolutions. A main agent reasons over the compressed context and proposes the next action. An isolated correction module then reviews and revises the proposed code using execution feedback and relevant API documentation, but without access to the full conversational history. The method is entirely training-free: it does not modify model weights, introduce additional supervision, or alter the benchmark environment. Instead, it changes the effective policy through structured conditioning and staged inference over the same base model. This simple intervention yields consistent gains. The scaffolded system improves task goal completion from 5.4% to 8.9% in FP16 and from 3.0% to 5.9% in AWQ, with the largest gains on difficulty-1 tasks (15.8% \to 26.3% in FP16; 5.3% \to 14.0% in AWQ). Under full-precision inference, the scaffolded 8B model numerically exceeds the 7.1% result reported for DeepSeek-Coder 33B Instruct on the same benchmark split [ 21 ] , although this difference is not statistically significant at the 95% level given the benchmark size ( N=168 ). The broader implication is not that small models match frontier systems, but that a meaningful share of small-model underperformance on agentic benchmarks arises from mechanical failures that can be mitigated at inference time. Our work makes three contributions. First, we provide an empirical characterization of failure modes for a small AppWorld agent, showing that authentication failures, planning errors, and API-schema mismatches dominate unsuccessful trajectories. Second, we present a diagnostic-first design methodology for inference-time scaffolding, in which each component is motivated by recurrent error patterns observed in baseline behavior. Third, we show that this scaffolded policy can substantially improve the effective performance of a frozen 8B model running on consumer-grade hardware, clarifying the role of inference-time structure as a practical lever for resource-constrained agent deployment. The remainder of this paper is organized as follows. We first review related work on tool-using language model agents, reinforcement learning for agent improvement, and inference-time methods for self-correction and memory management. We then describe the experimental setup and formalize our scaffolded policy for AppWorld as an inference-time composition over a frozen base model. Next, we present our failure mode analysis and the design of the three-tier scaffolding pipeline. We then report the main empirical results,