Is Escalation Worth It? A Decision-Theoretic Characterization of LLM Cascades
Dylan Bouchard · 2026
Read Is Escalation Worth It? A Decision-Theoretic Characterization of LLM Cascades by Dylan Bouchard (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06350.txt.
Introduction
Dylan Bouchard asks how deep a cascade of language models should go before escalation stops paying. Cascades let a cheap model defer hard queries to pricier ones, cutting inference cost, but practitioners must pick the pool size and every deferral threshold by feel. Bouchard derives first order optimality conditions showing that, at an optimum, the ratio of expected accuracy gain to expected downstream cost equalizes across every deferral boundary in the chain.
A local search on those conditions nearly matches exhaustive search. Experiments across five benchmarks, with thresholds optimized over pools of eight models, show two model cascades beating single model selection clearly. Adding a third or deeper model helped in almost none of the comparisons, and by a sliver when it did. Simulated scores show depth gains hinge on whether the confidence score reflects difficulty shared across models.
Catalog record
| CREATOR | Dylan Bouchard |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06350
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Is Escalation Worth It? A Decision-Theoretic Characterization of LLM Cascades Dylan Bouchard Abstract Model cascades, in which a cheap LLM defers to an expensive one on low-confidence queries, are widely used to navigate the cost-quality tradeoff at deployment. Existing approaches largely treat the deferral threshold as an empirical hyperparameter, with limited guidance on the geometry of the resulting cost-quality frontier over a model pool. We develop a decision-theoretic framework grounded in constrained optimization and duality. For a two-model cascade, we establish piecewise concavity of the cost-quality frontier on decreasing-benefit regions of the confidence support, with reciprocal shadow prices linking the budget- and quality-constrained formulations. Given a pool of k k models, we characterize the frontier achievable by deterministic two-model threshold cascades as the pointwise envelope over ( k 2 ) \binom{k}{2} pairwise cascades, with switching points where the optimal pair changes. For k k -model cascades, we derive first-order conditions in which a single shadow price equalizes marginal quality-per-cost across stage boundaries. We validate the framework on five benchmarks (MATH, MMLU, TriviaQA, SimpleQA, LiveCodeBench) across eight models from five providers. Within the deterministic threshold-cascade class, full fixed chains underperform the pairwise envelope, and optimized subsequence cascades do not deliver practically meaningful held-out gains over it. A lightweight pre-generation router exceeds the best cascade policy on four of five datasets, mainly because it avoids the cheap model’s generation cost on queries sent directly to a larger model rather than because of a stronger routing signal. These results suggest that cascade performance is limited primarily by structural cost, since cascades pay the cheap model before any escalation decision, rather than by a shortage of intermediate stages. 1 Introduction The rapid proliferation of large language models (LLMs) has created a fundamental tension in deployment: the most capable models are prohibitively expensive to query at scale, while cheaper alternatives frequently fall short of application-specific quality requirements. This cost-quality tradeoff has prompted growing interest in model cascades (see 19 for a detailed survey), in which a cheap model handles the majority of queries and a more expensive model is reserved for harder instances, identified by a confidence score s_{L} (typically derived from token-level log-probabilities or related uncertainty quantification signals) falling below a threshold \tau . Existing methods primarily treat \tau as a hyperparameter tuned on held-out data ( 4 ; 26 ) , with no formal characterization of the resulting cost-quality frontier, no analytical optimality conditions for cascades beyond two models, and no framework for choosing among the \binom{k}{2} two-model cascade configurations available from a pool of k models. We develop a decision-theoretic framework grounded in constrained optimization and duality. For the two-model cascade, minimizing expected cost subject to an expected quality floor is dual to maximizing expected quality subject to a budget constraint. In particular, on decreasing-benefit regions of the confidence support the cost-quality frontier is concave, and at any interior optimum the Lagrange multipliers of the two formulations are reciprocals with interpretable shadow-price meanings. Given a pool of k models, we characterize the frontier achievable by deterministic two-model threshold cascades as the pointwise envelope over \binom{k}{2} pairwise frontiers, with switching points at which the optimal pair transitions and the shadow price of quality generically jumps discontinuously. For k -model threshold cascades we derive first-order conditions under which a single shadow price \lambda equates decision-boundary expected escalation benefit to \lambda times decision-boundary expected downstream cost, equivalently equalizing marginal quality-per-cost across active stage boundaries. These conditions are stated in terms of conditional expectations estimable from calibration data, recover the two-model condition as a special case, and diagnose when additional stages in a fixed cascade chain have positive marginal value. We validate the framework on five benchmarks (MATH levels 3--5, MMLU, TriviaQA, SimpleQA, LiveCodeBench) across eight models from five providers. 1 1 1 Code is available at github.com/dylanbouchard/llm-cascade-frontiers . In held-out split experiments, both the non-dominated model pool and the valid pair set are selected on calibration data only. Empirically, the pairwise envelope effectively captures the deterministic threshold-cascade frontier: full fixed chains underperform the pairwise envelope, while optimized subsequence cascades do not deliver practically meaningful held-out gains over it. As a diagnostic baseline, we also evaluate a lightweight pre-generation router over the calibration-selected model pool, which exceeds the best cascade policy on four of five datasets. Score-choice ablations suggest that this advantage is primarily structural rather than a consequence of stronger routing signal quality. Our contributions are as follows: • Pairwise envelope over a model pool. We characterize the frontier achievable by all deterministic two-model threshold cascades drawn from a pool of k models as the pointwise envelope over \binom{k}{2} single-threshold frontiers, with switching points at which the optimal pair changes as a function of budget. This matters operationally: for any fixed budget, an envelope point is implemented by a single selected two-model cascade, rather than by running a longer selected cascade chain at inference time. Empirically, at 90% of ceiling quality, the pairwise envelope reduces cost by up to 79.5% relative to always using the highest-accuracy model, matches or exceeds the cost reduction of optimized multi-stage subsequence cascades on all five benchmarks, and consistently outperforms full fixed chains. • Structural characterization of the frontier. We establish monotonicity under a decision-boundary dominance condition and piecewise concavity on decreasing-benefit regions of the confidence support, and give the Lagrange multipliers of the dual formulations reciprocal shadow-price interpretations. • First-order conditions. In contrast to prior analytical formulations of the LLM cascade optimization problem, which do not model score-conditional accuracy, we allow each model’s expected quality to depend on the confidence scores of earlier stages, reflecting the correlation between confidence and query difficulty. We derive first-order conditions that account for this dependence, stated in terms of conditional expectations estimable from calibration data, and then test the practical value of additional stages by comparing full fixed chains and optimized subsequence cascades to the pairwise envelope on held-out splits. • Cascading vs. diagnostic routing. We show that a pre-generation diagnostic router exceeds the best cascade policy on four of five datasets, including three where its embedding signal is weaker than the cascade’s white-box uncertainty signal, and decompose its advantage into structural (avoiding the cheap model’s cost) and informational (signal quality) components. In our experiments, the cascade envelope is competitive primarily when the lightweight pre-generation features are uninformative. 2 Related Work LLM routing and cascading. A large body of recent work studies cost-efficient LLM deployment via model selection. Routing methods select a single model per query prior to inference, typically using learned routers trained on query representations or preference data ( 20 ; 29 ; 8 ; 24 ) . Several works incorporate explicit constraints such as budget, latency, or capacity into routing decisions ( 18 ; 17 ; 6 ) . Cascading methods instead query models sequentially and decide whether to accept or escalate a response based on signals observed after generation, including learned confidence estimators, token-level probabilities, and self-consistency signals ( 4 ; 26 ; 1 ; 13 ; 9 ) . Extensions consider reasoning-intensive settings ( 23 ) , uncertainty-aware routing with statistical guarantees ( 22 ) , deployment constraints such as privacy ( 27 ) , and unified routing-cascading formulations ( 5 ) . We focus on the cascading