TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks

Vansh Kapoor † \dagger, Aman Gupta, Hao Chen, Anurag Beniwal, Jing Huang, Aviral Kumar · 2026

Read TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks by Vansh Kapoor † \dagger, Aman Gupta, Hao Chen, Anurag Beniwal, Jing Huang, Aviral Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10245.txt.

Introduction

Vansh Kapoor and co authors route hard steps of a task to a large model, and easy steps to a small one. Multi step tasks can fail from one bad step. That one step can wreck the whole chain. Past route plans send the full task to one model.

That treats all steps as the same. The team made TRIM, a step wise route plan. It spots the steps most apt to wreck the chain. Those steps go to a large model. Plain steps stay with a small model. The aim is high grades at a lower cost than all large runs.

Catalog record

CREATORVansh Kapoor † \dagger, Aman Gupta, Hao Chen, Anurag Beniwal, Jing Huang, Aviral Kumar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10245

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks Vansh Kapoor † \dagger, Aman Gupta, Hao Chen, Anurag Beniwal, Jing Huang, Aviral Kumar Abstract Multi-step reasoning tasks like mathematical problem solving are vulnerable to cascading failures where a single incorrect step leads to complete solution breakdown. Current LLM routing methods assign entire queries to one model, treating all reasoning steps as equal. We propose TRIM (Targeted routing in multi-step reasoning tasks), which routes only critical steps–those likely to derail the solution– to larger models while letting smaller models handle routine continuations. Our key insight is that targeted step-level interventions can fundamentally transform inference efficiency by confining expensive calls to precisely those steps where stronger models prevent cascading errors. TRIM operates at the step-level: it uses process reward models to identify erroneous steps and makes routing decisions based on step-level uncertainty and budget constraints. We develop several routing strategies within TRIM, ranging from a simple threshold-based policy to more expressive policies that reason about long-horizon accuracy–cost trade-offs and uncertainty in step-level correctness estimates. On MATH-500, even the simplest thresholding strategy surpasses prior routing methods with 5 × 5\times higher cost efficiency, while more advanced policies match the strong, expensive model’s performance using 80 % 80\% fewer expensive model tokens. On harder benchmarks such as AIME, TRIM achieves up to 6 × 6\times higher cost efficiency. All methods generalize effectively across math reasoning tasks, demonstrating that step-level difficulty represents fundamental characteristics of reasoning. † † affiliationnotice: † Work primarily conducted during an internship at Amazon 1 Introduction Figure 1 : Schematic Overview of a Two-Model Setup for TRIM The rapid progress in large language models (LLMs) has led to a diverse ecosystem of models, spanning a wide spectrum of sizes, capabilities, and computational demands. Larger models typically achieve stronger performance but incur substantial serving costs, rendering them impractical for many routine applications. In contrast, smaller models are more affordable to deploy but often produce lower-quality responses. This trade-off poses a fundamental dilemma for the practical deployment of LLMs: routing all queries to the largest available model ensures high-quality outputs but is prohibitively expensive, whereas relying solely on smaller models reduces serving costs at the expense of degraded response quality, especially on challenging queries. Contemporary routing strategies attempt to mitigate this dilemma by assigning each query to a single model, which is then responsible for the entire generation . However, not all tokens in an LLM’s response are equally challenging to generate: some represent critical decision points that can dramatically alter the solution path ( 23 ; 28 ; 22 ) , while others are routine continuations that are easier to generate. By committing to a larger LLM over a smaller one for a given query, existing routing methods implicitly assume that the stronger model’s intervention is equally necessary at every token to produce a high-quality response. This inefficiency is particularly pronounced in multi-step reasoning tasks such as step-by-step reasoning or code generation, where mistakes early on can snowball into a complete failure ( 31 ) . It is precisely at these erroneous steps that the intervention of a stronger model is most valuable. Yet, contemporary routing methods often incur substantial inefficiency by defaulting to full generations from the larger model, even when targeted interventions at these erroneous steps would suffice. To address this inefficiency, we introduce TRIM- T argeted Stepwise R outing for I nference in M ulti-step Reasoning Tasks: an approach that selectively routes only the most critical steps to larger LLMs. Unlike prior prompt-level routers, TRIM operates at the granularity of individual reasoning steps, generating solutions one step at a time. As illustrated in Figure 1 , a stepwise router evaluates each intermediate step as it is generated and decides whether to accept the small model’s output or to regenerate that specific step with a larger model. TRIM ensures that interventions occur only when necessary at individual steps, rather than handing over the entire remaining solution to the larger model. This step-by-step generation process with targeted intervention enables TRIM to achieve efficiency by confining expensive calls to precisely those steps where intervention prevents cascading errors while allowing the smaller model to handle routine continuations. This approach reduces the primary cost bottleneck in inference: the number of tokens generated by the expensive, larger LLM. Building on this framework, we design multiple strategies for stepwise routing, each tailored to different computational and informational constraints: a simple thresholding policy that uses step-level scores to identify erroneous steps, two RL-trained policies that reason about long-horizon trade-offs between accuracy and cost (one using full sequential features, another using aggregated statistics), and a Partially Observable Markov Decision Process (POMDP) based approach that accounts for the inherent uncertainty in step-level correctness estimates while enabling efficient policy recomputation across different cost budgets. Our cost metric focuses on the number of tokens generated by the expensive model, as prefill costs can be amortized through parallel decoding strategies ( 14 ; 3 ) , whereas generation tokens impose unavoidable sequential costs. We evaluate our approach on diverse mathematical reasoning benchmarks, including MATH ( 11 ) , Olympiad-Bench ( 10 ) , and AIME ( 5 ) . On MATH-500, our basic thresholding policy is 5 × more cost-effective than existing methods, while our trained RL and POMDP policies achieve the performance of the expensive LLM using only 20\% of the expensive tokens. On the more challenging AIME benchmark, these policies achieve 3.17\times and 6.33\times higher cost efficiency, respectively. Moreover, routing policies trained on one dataset generalize strongly to others: the RL-trained policy attains up to 11.68\times higher cost efficiency when evaluated on OlympiadBench, despite being trained only on AIME. Our main contributions are: (1) We provide systematic evidence showing that a small number of targeted step-level interventions can substantially enhance the efficiency of multi-step reasoning. (2) We demonstrate that this insight can be used to instantiate practical frameworks that are amenable to several routing policies, from simple thresholding policies that surpass existing query-level routing methods to advanced RL-trained and POMDP-based approaches that achieve competitive performance with oracle routers having perfect task knowledge, all while using significantly fewer expensive model tokens. Our routing policies can be trained effectively with limited supervision data while achieving robust performance across diverse cost budgets, making the approach practical for real-world deployment scenarios. (3) Finally, we demonstrate robust generalization of the routing strategy across datasets, with methods trained on AIME delivering strong efficiency gains on OlympiadBench and Minerva Math, suggesting that step-level difficulty patterns capture transferable structure across related reasoning benchmarks for a given base model, rather than being tightly coupled to a specific dataset. 2 Related Work The trade-off between model performance and computational cost has become increasingly salient as large language models grow in size and capability. Our work on targeted stepwise routing builds upon prior work from several areas: LLM routing strategies, process supervision and step-level verification, and multi-step reasoning optimizations. Figure 2 : Comparison of task performance–cost trade-offs for Qwen2.5-3B-Instruct ( M_{w} ) and Claude 3.7 Sonnet ( M_{s} ), under the myopic thresholding policy (with Qwen2.5-Math-PRM-7B) versus the Idealized Oracle Query-level Router, across multiple math benchmarks. The Oracle Router is evaluated by incrementally varying the number of queries routed

Read the complete text