Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts
Saloni Garg, Amit Sagtani · 2026
Read Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts by Saloni Garg, Amit Sagtani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07395.txt.
Introduction
Saloni Garg and Amit Sagtani reexamine a claimed ceiling in routing queries among several language models. Routers send each query to the cheapest model that can solve it, and prior work blamed unsolvable queries for the limits. In a large study across six benchmarks and two model families, the authors show much of that unsolvability is an evaluation artifact.
Judge models favored verbose answers, fixed generation budgets truncated correct work, and format mismatches hid right answers. Correcting those issues cut measured unsolvability, while standard routers collapsed toward always picking the smallest tier, wasting opportunity. The authors recommend dual judging, exact-match anchoring, and cost-sensitive training goals.
Catalog record
| CREATOR | Saloni Garg, Amit Sagtani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07395
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts Saloni Garg, Amit Sagtani Abstract Efficient routing across multiple LLMs is central to cost-effective deployment, enabling better trade-offs between cost, latency, and quality. Prior work has attributed substantial routing headroom to an “unsolvability ceiling”: a non-trivial fraction of queries that no single model in the pool can solve correctly. In this paper, we present a large-scale study of multi-tier LLM routing with over 206,000 query-model pairs across six diverse benchmarks (MMLU, MedQA, HumanEval, MBPP, Alpaca, ShareGPT) using the Gemma 4 and Llama 3.1 model families. Evaluating with both LLM-as-a-judge and exact-match metrics, we show that a substantial portion of reported unsolvability stems from evaluation artifacts rather than fundamental model limitations. We identify three primary sources: (i) systematic judge biases favoring specific response styles and verbosity, (ii) truncation under fixed generation budgets, and (iii) output format mismatches. Through dual-judge validation and exact-match grounding, we consistently reduce measured unsolvability across tasks. We introduce a decomposition framework that attributes failures to these artifacts, revealing consistent patterns across domains and model families. These artifacts also distort supervision signals for routing models. Standard routers trained on oracle labels are highly sensitive to class imbalance and frequently collapse to the majority class ( ∼ 79 % {\sim}79\% smallest-tier optimal), a behavior confirmed via random-feature and shuffled-label controls. This leads to a 13-17 percentage point opportunity cost in routing performance. We provide actionable recommendations, including dual-judge validation, exact-match anchoring, and cost-sensitive objectives for more robust evaluation and training. Our findings suggest that existing routing headroom estimates are substantially inflated, calling into question many reported gains and underscoring the need for reliable protocols in multi-LLM systems. 1 Introduction The problem of routing queries to the most cost-effective language model capable of answering them correctly has attracted significant research interest. FrugalGPT ( 2 ) demonstrated that cascading queries across models of increasing capability can reduce cost by up to 98% while matching the best individual model’s performance. RouteLLM ( 7 ) introduced a training framework for binary routers using human preference data, achieving over 2\times cost savings on standard benchmarks. A central assumption underlying these gains is that routing headroom is real and recoverable: a non-trivial fraction of queries are genuinely unsolvable by the cheapest model but solvable by larger ones, and that this gap can be exploited by routing systems. This assumption rests on the quality of the evaluation used to measure it. When routing headroom is estimated from LLM-as-a-judge scores, the measured headroom reflects not only genuine model capability differences but also any systematic tendencies of the judge to reward or penalize model outputs for reasons unrelated to correctness. When it is estimated from fixed-budget inference, the measured headroom reflects not only model capability but also truncation effects that prevent models from completing responses. When it is estimated from free-form generation on multiple-choice tasks, the measured headroom also depends on whether models express their answer in the expected format. We present a large-scale empirical study designed to disentangle genuine unsolvability from these evaluation artifacts. We score 206,756 query-model pairs (51,689 queries \times 4 tiers) from six diverse benchmarks using the Gemma 4 model family deployed on self-hosted H100 infrastructure with thinking mode disabled and greedy decoding. We validate judge scores against exact-match accuracy on two benchmarks with verifiable gold labels (MMLU, MedQA), run cross-family validation with Llama-3.1-8B and Llama-3.1-70B, and train three router variants to characterize how evaluation artifacts propagate into routing model training. Central finding. A substantial fraction of reported unsolvability is attributable to three systematic evaluation artifacts rather than fundamental model limitations. Artifact 1: Evaluation misalignment. Our judge (a state-of-the-art 26B mixture-of-experts model) diverges systematically from exact-match ground truth on both knowledge benchmarks, but in opposite directions: it assigns lower scores than exact-match on MMLU by 10-24 pp across all tiers, and higher scores than exact-match on MedQA for large models by up to 6 pp. The judge partially conflates correctness with fluency, structure, and reasoning presentation, partially crediting responses with strong reasoning but incorrect final answers, and partially penalizing responses with correct answers expressed in terse or unconventional form. Artifact 2: Truncation. Per-source maximum token budgets are binding for MMLU (65% truncation rate) and MedQA (57% truncation rate). Truncated responses frequently receive score <2 from the judge even when the model has identified the correct answer. Artifact 3: Output format mismatch. Parse failures on MMLU range from 5.4% to 12.4% across tiers; on MedQA from 0.6% to 5.2%. Format mismatch preferentially inflates apparent unsolvability for smaller tiers and compresses the measured quality gradient. Routing supervision. These artifacts compound in routing model training. Oracle labels derived from judge-scored data reach 79.3% E2B-optimal by construction, and standard unweighted routers collapse entirely to majority-class prediction. Random-feature and shuffled-label controls confirm the collapse is caused entirely by the label marginal. The opportunity cost (routing performance achievable with dual-judge oracle labels and cost-sensitive objectives versus collapsed routers) is 13-17 percentage points across knowledge-intensive benchmarks. Contributions. (1) An artifact decomposition framework identifying and quantifying three sources of evaluation-induced unsolvability inflation; (2) a dual-judge validation methodology pairing LLM-as-judge with exact-match grounding; (3) cross-family ceiling analysis showing the ceiling is model-family-specific for knowledge tasks and near-universal for conversational tasks; (4) a characterization of routing collapse under realistic label distributions with controlled experiments; and (5) actionable recommendations for evaluation protocol design. 2 Model Family and Infrastructure 2.1 The Gemma 4 Tier Family Table 1 describes the four inference tiers. All models are from the Gemma 4 family (Google DeepMind), licensed under Apache 2.0, downloaded from HuggingFace on 2026-04-21. Table 1: Gemma 4 inference tier specifications. The 26B-A4B MoE model activates only {\sim}4 B parameters per forward pass (equal active compute to E4B) while carrying 26B total parameters of knowledge capacity. Tier HuggingFace Model ID Architecture Total Active E2B google/gemma-4-e2b-it Dense, 35L, GQA 8/1 {\sim}2 B {\sim}2 B E4B google/gemma-4-e4b-it Dense, 42L, GQA 8/2 {\sim}4 B {\sim}4 B 26B-A4B google/gemma-4-26b-a4b-it MoE, 30L, 128 experts {\sim}26 B {\sim}4 B 31B google/gemma-4-31b-it Dense, 60L, GQA 32/16 {\sim}31 B {\sim}31 B E4B and 26B-A4B share approximately equal active parameter counts ( {\sim}4 B per forward pass), predicting near-identical inference latency. The MoE model’s 26B total parameters encode substantially more knowledge than a 4B dense model: equal inference cost, substantially different knowledge capacity. 2.2 Inference Infrastructure All models are served using vLLM 0.9.1 on dedicated H100 80GB HBM3 GPUs with bf16 precision, 4,096-token context, and gpu-memory-utilization=0.90 . Thinking mode is disabled globally via --default-chat-template-kwargs ‘{"enable_thinking": false}’ and per-call via extra_body={"enable_thinking": False} . 2.3 Primary Judge Model We use google/gemma-4-26b-a4b-it , the same weights as the 26B-A4B inference tier, as the primary judge, with --reasoning-parser gemma4 enabled. The judge applies a 0/1/2 rubric: 0 = incorrect or unhelpful, 1 = partially correct or incomplete, 2 = correct and complete. Because the judge shares weights with one evaluated tier, we flag this as a limitation and recommend a held-out judge from a different family in future work (Section 9.4 ). The need for dual-judge validation. For benchmarks with verifiable gold labels, relying solely on the LLM judge risks conflating evaluation artifacts with genuine model failures. Discordance between the two judges reveals cases where the two evaluators operationalize “correctness” differently