DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows

Yuxuan Gao, Megan Wang, Yi Ling Yu, Zijian Carl Ma, Ao Qu, [6pt] OpenMesh AI, [2pt] Stanford University · 2026

Read DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows by Yuxuan Gao, Megan Wang, Yi Ling Yu, Zijian Carl Ma, Ao Qu, [6pt] OpenMesh AI, [2pt] Stanford University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19099.txt.

Introduction

Yuxuan Gao and four colleagues introduce DecisionBench, a testbed for studying how AI agents delegate work to peer models over long tasks. It fixes a task suite, an eleven-model peer pool, a delegation interface, and metrics spanning quality, cost, latency, and routing accuracy. A reference sweep over 23,375 task instances finds end-task quality barely differs across awareness conditions, while routing accuracy varies widely and the delivery channel matters more than description wording. Perfect delegation would beat measured performance by 15 to 31 percentage points on every suite, showing headroom for orchestration.

Catalog record

CREATORYuxuan Gao, Megan Wang, Yi Ling Yu, Zijian Carl Ma, Ao Qu, [6pt] OpenMesh AI, [2pt] Stanford University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80811
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19099

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows Yuxuan Gao, Megan Wang, Yi Ling Yu, Zijian Carl Ma, Ao Qu, [6pt] OpenMesh AI, [2pt] Stanford University Abstract We introduce DecisionBench , a benchmark substrate for emergent delegation in long-horizon agentic workflows. The substrate fixes a task suite (GAIA, τ \tau -bench, BFCL multi-turn), a peer-model pool (11 models, 7 vendor families), a delegation interface ( call_model plus an optional read_profile channel), a deterministic skill-annotation layer, and a multi-axis metric suite covering quality, cost, latency, delegation rate, routing fidelity-at- k k , vendor self-preference, and a counterfactual-delegation ceiling. The substrate is agnostic to how peer information is generated or delivered, so learned routers, richer peer memories, adaptive profile construction, and multi-step delegation can all be evaluated against it. We characterize the substrate with a five-condition reference sweep on the full pool ( n = 23,375 n{=}23{,}375 task instances). Three benchmark-level findings emerge: (i) mean end-task quality is statistically indistinguishable across the four awareness conditions ( | β | ≤ 0.010 |\beta|{\leq}0.010 , p ≥ 0.21 p{\geq}0.21 ), so quality-only evaluation would miss the orchestration signal; (ii) routing fidelity-at- 1 1 ranges from 7.5 % 7.5\% to 29.5 % 29.5\% across conditions at near-equal mean quality, with delivery channel (on-demand tool vs. preloaded description) dominating description content; (iii) a counterfactual ceiling places perfect delegation 15 15 – 31 31 percentage points above measured performance on every suite, locating large unrealized headroom for future orchestration methods. We release the substrate, annotation layer, reference intervention suite, analysis pipeline, and 220 per-condition run archives. 1 Introduction Large language model (LLM) agents are increasingly deployed for tasks that span minutes to hours of tool use ( 49 ; 44 ) , with the dominant cost shifting from per-token quality to compute spent on sub-tasks where a smaller or differently-specialized peer model would have sufficed ( 24 ; 6 ; 13 ) . Routing a sub-task to a peer rather than solving it locally is therefore a Pareto-frontier shifter alongside raw model capability. A practitioner deploying an agentic system today must decide whether to delegate, to whom, and what (if anything) to tell the orchestrator about its peers. These decisions are typically made by intuition because no benchmark has measured them in isolation. Existing agentic benchmarks measure single-agent capability on a fixed task — GAIA ( 23 ) , \tau -bench ( 47 ) , BFCL ( 29 ) and others (§ 2 ). Multi-agent systems work ( 43 ; 11 ) typically uses hand-coded role assignment rather than emergent delegation. Cost-aware routing benchmarks ( 24 ; 13 ) treat routing as a learned external policy rather than a behavior the orchestrator must exhibit on its own. SkillsBench ( 17 ) measures skill scaffolds attached to a single agent. None of these directly measure whether an agent delegates well between peer models in the course of solving a long-horizon task, or what process-level metrics would reveal about how those delegation decisions are made. We introduce DecisionBench , a benchmark substrate for emergent delegation in long-horizon agentic workflows. The substrate fixes (1) a task suite (GAIA, \tau -bench, BFCL multi-turn, with deterministic Stage-1/Stage-2 splits), (2) a peer-model pool (11 models from 7 vendor families pinned to a freeze date), (3) a delegation interface (a call_model tool, optionally paired with a read_profile channel for peer-information interventions), (4) an annotation layer (a frozen 7-skill taxonomy and a deterministic step tagger, released as evaluation machinery rather than a method we propose), and (5) a metric suite covering end-task quality, cost, latency, delegation rate, routing fidelity-at- k , vendor self-preference, and a counterfactual-delegation ceiling. The substrate is agnostic to how peer information is generated or delivered; methods that produce learned routers, richer peer memories, adaptive profile construction, or multi-step delegation can all be evaluated against it. To demonstrate how DecisionBench evaluates peer-awareness interventions, we instantiate four reference conditions over the released annotation layer, against a no-information baseline: ( blind ) only call_model , no peer description; ( aware-c1 / c2 / c3 ) preload a peer description constructed three different ways — a curated rubric, deterministic statistics from Stage-1 traces, or dual out-of-pool LLM-judge summaries — and expose read_profile ; and ( aware-tool-only ) the same tools as the aware variants but with the preloaded description suppressed, a one-variable ablation that isolates delivery channel from description content. These reference interventions are not part of the benchmark definition; they are the baselines we use to demonstrate the substrate and to seed comparisons for future methods. Findings. Across the 5-condition reference sweep on the 11-model pool ( 11{\times}3{\times}5{=}165 cells, n{=}23{,}375 tasks), three benchmark-level patterns emerge. (1) End-task quality is mostly flat across awareness conditions ( |\beta|{\leq}0.010 , p{\geq}0.21 in a mixed-effects fit), so quality-only evaluation would miss the orchestration signal entirely. (2) Delegation fidelity moves cleanly : on-demand tool access ( aware-tool-only ) more than doubles routing precision-at- 1 over blind ( 14.2\%{\to}29.5\% ) at quality parity and lower mean cost; preloaded variants capture less than half of the gain (C2 20.8\% , C3 15.5\% ; C1 7.5\% , below blind). The dissociation between fidelity and quality is itself the methodological payoff of measuring process-level metrics, not just outcomes. (3) A counterfactual-delegation ceiling places perfect single-step delegation 15–31 pp above measured performance on every suite, locating substantial unrealized headroom for future orchestration methods. We additionally document cross-vendor self-preference in delegation behavior at 1.5 – 3.7\times chance. Contributions. (1) The DecisionBench benchmark for emergent delegation in long-horizon agentic workflows: a fixed task suite, peer-model pool, delegation interface, annotation layer (frozen 7-skill taxonomy plus deterministic step tagger), and a multi-axis metric suite (quality, cost, latency, delegation rate, fidelity-at- k , vendor self-preference, counterfactual ceiling). (2) A reference intervention suite for peer-awareness on the benchmark: three profile-card content variants (curated rubric, deterministic statistics, dual out-of-pool LLM-judge summary) and a delivery-channel ablation ( aware-tool-only ) that isolates information content from delivery mechanism. These are baselines on DecisionBench, not part of the benchmark definition. (3) An empirical characterization of the substrate using the reference suite: end-task quality is statistically flat across awareness conditions, delivery channel dominates description content, and the perfect-delegation ceiling exceeds measured performance by 15–31 pp on every suite — patterns visible only through the process-level metrics the benchmark instruments. (4) Documented cross-vendor self-preference in delegation behavior, the orchestration-tool analogue of LLM-as-judge biases ( 52 ; 27 ; 41 ; 31 ) . Figure 1: DecisionBench overview. Left (substrate, § 3 ). The benchmark fixes a task suite (GAIA, \tau -bench, BFCL multi-turn), a peer-model pool (11 models, 7 vendor families), a delegation interface ( call_model , plus an optional read_profile channel), an annotation layer (frozen 7-skill taxonomy and deterministic step tagger; App. C ), and a metric suite (quality, cost, latency, delegation rate, fidelity-at- k , self-preference, counterfactual ceiling). Right (reference interventions, § 4 ). To demonstrate the substrate, peer trajectories on the Stage-1 split are rendered into three profile-card variants ( C1 rubric, C2 deterministic statistics, C3 dual LLM-judge); each is used as a preloaded description in an aware- condition. A delivery-channel ablation ( aware-tool-only ) exposes read_profile without preloading any description, isolating delivery from content. These five conditions seed comparisons for future

Read the complete text