RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning

Jiayi Tian, Yupeng Su, Ryan Solgi, Souvik Kundu, Zheng Zhang · 2026

Read RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning by Jiayi Tian, Yupeng Su, Ryan Solgi, Souvik Kundu, Zheng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16694.txt.

Introduction

Tian and colleagues propose RankGuide for small model work through rank routing and steering. Big reasoning models gain power from stepwise chains, but pay latency and cost. Small models write steps for a speed trade-off, yet failure-spotting remains hard. In text and hidden spaces, the authors identify overconfidence, uncertainty and revalidation as modes. RankGuide routes with rank signals from hidden states, calling large models only when small models look to fail. It also extracts a filtered vector to steer small-model reasoning and improve quality.

Catalog record

CREATORJiayi Tian, Yupeng Su, Ryan Solgi, Souvik Kundu, Zheng Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39318
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16694

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RankGuide: Tensor-Rank-Guided Routing and Steering for Efficient Reasoning Jiayi Tian, Yupeng Su, Ryan Solgi, Souvik Kundu, Zheng Zhang Abstract Large reasoning models (LRMs) enhance problem-solving capabilities by generating explicit multi-step chains of thought (CoT) reasoning; however, they incur substantial inference latency and computational overhead. To mitigate this issue, recent works have explored model collaboration paradigms, where small reasoning models (SRMs) generate intermediate reasoning steps to achieve a better accuracy–latency trade-off. Despite recent progress, effectively and efficiently detecting and mitigating SRM failures in collaborative systems remains a key challenge. To address this issue, we analyze SRM inference in both the generated text and hidden-state spaces, and identify three types of failure modes: overconfidence , uncertainty , and heavy revalidation . Building on these insights, we propose RankGuide , a framework that improves the efficiency and effectiveness of SRM–LRM collaboration through tensor-rank-guided routing and steering. Specifically, RankGuide leverages a routing signal that incorporates tensor-rank signals derived from consecutive hidden states to detect when SRMs are likely to fail and selectively invoke LRMs. In addition, we introduce a tensor-rank-filtered steering vector extraction method to modulate the reasoning trajectory of SRMs, thereby improving their generation quality. By improving both routing and steering through tensor-rank signals, RankGuide enables SRM–LRM collaborative systems to achieve more efficient reasoning with fewer steps and improved accuracy. Experiments on multiple reasoning benchmarks demonstrate the efficacy of RankGuide in reducing latency by up to 1.75 × 1.75\times compared to LRM, while maintaining competitive accuracy relative to prior methods. 1 Introduction Large reasoning models (LRMs) ( 4 ; 18 ) have recently demonstrated strong performance on complex problem-solving tasks by explicitly generating multi-step chains of thought (CoT) reasoning ( 16 ) . By decomposing problems into intermediate steps, these models achieve significant gains in domains such as mathematical reasoning ( 5 ; 21 ) and code generation ( 6 ) . However, this improved capability comes at the cost of substantially increased inference latency and computational overhead, limiting their practicality in real-world deployment scenarios. To mitigate this issue, recent works ( 12 ; 11 ; 3 ; 20 ) have explored model collaboration paradigms, where small reasoning models (SRMs) generate intermediate reasoning steps while larger models are invoked selectively. These approaches offer a promising accuracy–latency trade-off by offloading simpler reasoning steps to lightweight models while preserving the capabilities of LRMs for more complex cases. Despite these advances, effectively and efficiently detecting and mitigating SRM failures remains a key challenge in collaborative systems. For instance, SpecReason ( 11 ) relies on LLM verifier feedback to decide whether to let larger models regenerate the reasoning step, which introduces additional computational and latency overhead. Recently, GlimpRouter ( 20 ) triggers routing prior to each step based on entropy-derived uncertainty signals, thereby avoiding the need for LLM verifiers or rollback mechanisms. However, we observe that such approaches can fail when SRMs produce overconfident yet incorrect reasoning steps, resulting in missed routing opportunities and confident errors. Consequently, inaccurate failure detection allows errors to propagate through the reasoning chain, degrading final performance and diminishing efficiency gains. To better understand and address these limitations, we conduct a systematic analysis of SRM inference from the perspective of the generated text as well as the hidden-states. Our analysis reveals three characteristic failure modes: (a) overconfidence , where the model produces confident but incorrect reasoning; (b) uncertainty , where the model struggles to make progress; and (c) heavy revalidation , where the model repeatedly revisits prior steps without advancing the solution. These findings highlight the need for more reliable signals to detect diverse SRM failure modes and guide appropriate interventions during inference. Building on these insights, we propose RankGuide , a training-free serving framework that improves the efficiency and effectiveness of SRM–LRM collaboration. Our method leverages tensor-rank signals derived from consecutive hidden states to guide both routing and steering during LLM inference. These signals are used to detect likely SRM failures, enable selective LRM invocation, and filter the calibration set for high-quality steering vector extraction, ultimately leading to more efficient and reliable reasoning. Our contributions are summarized as follows: • We identify SRM failures as the primary bottleneck in collaborative reasoning systems, and provide a systematic analysis of SRM inference from both generated-text and hidden-state perspectives, revealing three key failure modes: overconfidence , uncertainty , and heavy revalidation . • We propose tensor-rank-guided routing, which employs a routing policy driven by a composite signal integrating token-level entropy and tensor-rank features derived from hidden states to identify potential SRM failures and selectively invoke LRMs, addressing both uncertainty and overconfident errors. • We introduce a tensor-rank-filtered steering vector extraction method that modulates the hidden-state trajectory of SRMs during inference using a steering vector computed from a tensor-rank-filtered calibration set, thereby reducing heavy revalidation in SRM generation. Experimental results demonstrate that RankGuide can reduce latency and computational cost while maintaining competitive accuracy compared to prior methods. In specific, RankGuide can yield up to 1.75\times and 1.36\times latency benefit compared to LRM and SoTA collaborative inference framework, while maintaining or improving the accuracy. 2 Related Works 2.1 Tensor Decomposition and Low-Rank Structure Low-rank structures have been widely studied in deep learning for model compression and efficient representation learning, including singular value decomposition (SVD) ( 14 ; 7 ) for matrices and tensor-train (TT) decomposition ( 10 ) for high-dimensional tensors, among other techniques. Prior works primarily leverage tensor decompositions to approximate weight matrices or adapters ( 13 ; 9 ; 19 ) , thereby reducing memory and computational cost. In contrast, our work does not use tensor decomposition as a compression tool. In contrast, we utilize tensor-rank as a diagnostic signal for reasoning complexity, enabling representation-driven routing and improved steering in LRM inference. 2.2 Model Collaboration Model collaboration ( 15 ; 23 ) can improve inference efficiency by combining models of different capacities, where small reasoning models (SRMs) handle simple cases and large reasoning models (LRMs) are invoked for more complex reasoning. Recent works extend this paradigm to LLM reasoning by allowing SRMs to generate intermediate steps with selective verification or refinement. A central challenge is effective routing between SRMs and LRMs. Existing methods often rely on LLM verifier ( 11 ; 12 ; 3 ) , which is costly and may lead to suboptimal decisions. To address this, GlimpRouter ( 20 ) proposes an entropy-based confidence score that routes uncertain steps based on the SRM’s output distribution, achieving improved efficiency. However, relying solely on the SRM’s output confidence as the routing signal can be biased, as SRMs may be both highly confident and incorrect. To address this limitation, we propose RankGuide, which leverages tensor-rank signals from hidden states to provide a more reliable indicator of SRM reasoning quality for routing. 2.3 Efficient Reasoning via Steering Beyond model collaboration, a prominent direction is latent-space steering of LRMs. In steering, hidden representations are modified to guide generation toward desired behaviors. Steering methods ( 2 ; 1 ; 17 ) show that intervening in intermediate activations can improve reasoning quality or enforce concise generation patterns without retraining. While these approaches improve efficiency by enhancing reasoning through latent-space calibration, their performance gains are often limited, and they typically fall short of enabling SRMs to match the accuracy of LRMs.

Read the complete text