HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs

Yansong Ning, Mianpeng Liu, Jingwen Ye, Weidong Zhang, Hao Liu · 2026

Read HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs by Yansong Ning, Mianpeng Liu, Jingwen Ye, Weidong Zhang, Hao Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28398.txt.

Introduction

Yansong Ning and four colleagues introduce HRBench, a common testbed for thinking-mode switching in hybrid-reasoning language models. It crosses three switching families with four training regimes, giving twelve controlled settings. Six models and five reasoning benchmarks are tested in one pipeline. Prompt methods often gave the best accuracy per token, routing cut cost most steadily, and speculative methods raised accuracy at higher cost.

Catalog record

CREATORYansong Ning, Mianpeng Liu, Jingwen Ye, Weidong Zhang, Hao Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58125
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28398

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs Yansong Ning, Mianpeng Liu, Jingwen Ye, Weidong Zhang, Hao Liu Abstract Hybrid-reasoning large language models (LLMs) expose explicit controls over reasoning effort, allowing users or systems to trade off answer quality against inference cost. However, existing methods for adaptive thinking-mode selection are typically evaluated under different models, datasets, and implementation assumptions, making it difficult to compare their practical behavior. We introduce HRBench, a unified evaluation framework for studying thinking-mode switching in hybrid-reasoning LLMs. HRBench organizes the design space along two axes: three switching strategy families, prompt-based selection, external routing, and speculative execution, and four training regimes, training-free, SFT, offline and online RL, yielding 12 controlled evaluation settings. We evaluate these settings across 6 LLMs, from Qwen3.5-2B to Kimi-K2.5-1.1T, and 5 reasoning benchmarks covering mathematics, science, and code, while reimplementing 12+ representative prior methods within the same pipeline. Our analysis characterizes how different switching strategies occupy distinct effectiveness-efficiency trade-off regions: prompt-based methods often provide favorable token-accuracy trade-offs, routing methods offer more stable cost reduction, and speculative methods tend to improve accuracy at higher token cost. We further find that training affects strategies differently, and that the preferred strategy varies with model scale and task domain. HRBench provides reference implementations and a unified evaluation platform to support more controlled research on efficient reasoning in hybrid-reasoning LLMs. Our data, code and repository are available at https://github.com/usail-hkust/HRBench . 1 Introduction Recent reasoning-oriented LLMs, such as OpenAI o1 ( 19 ) and DeepSeek-R1 ( 6 ) , are achieving remarkable success on complex tasks through extended chain-of-thought (CoT) reasoning 28 , but at the cost of substantial token overhead. To address this, a new generation of hybrid-reasoning LLMs has emerged, including Qwen3.5 ( 21 ) , gpt-oss 1 , and Seed-OSS 3 , that expose explicit thinking-mode switches : users can select between deep reasoning ( think ) and direct answering ( no_think ), specify discrete reasoning effort levels (e.g., High/Medium/Low), or set numeric budgets (e.g., \leq 4096 tokens). This raises a question: when should the model think, and how much? Figure 1: Overview of HRBench . Left : Three thinking-mode switch strategies—Prompt-Tuning, Routing, and Speculative. Right : Training pipeline spanning training-free to online RL. Bottom : Our evaluation coverage across 6 models spanning from 2B to 1.1T scale, 5 datasets, totaling 527 experiment runs. A growing body of work tackles this efficiency–effectiveness trade-off by proposing adaptive thinking-mode switch methods. These can be categorized into three thinking-mode switch strategies: • Prompt-Tuning ( PT ) guides the model to determine its thinking mode during a single inference pass through carefully designed prompts. For example, methods such as S1 ( 18 ) and TALE ( 7 ) inject token-budget or difficulty-aware instructions that let the model control reasoning length. Furthermore, RL-based approaches like ACPO ( 4 ) directly optimize this internal decision. • RouTing ( RT ) adopts a classify-then-generate strategy, where a router evaluates the query difficulty before dispatching it to the appropriate thinking mode. As representative examples, AdaptThink ( 33 ) trains such a router via GRPO, while HDFlow ( 32 ) uses rule-based difficulty classification. • Speculative ( Spec ) methods allow the model to begin in a fast mode and dynamically switch to deep reasoning upon detecting uncertainty signals. For instance, MixReasoning ( 16 ) uses entropy-based triggers for this escalation, while ADR ( 35 ) learns the switching policy through SFT and RL. Despite active progress, these methods are evaluated under incomparable settings—different LLMs, datasets, metrics, and decoding configurations—making it impossible to answer: which strategy truly works best? and how much does the training process help each strategy? To address this, we propose HRBench shown in Figure 1 , a unified benchmark for understanding how different thinking-mode switch strategies behave across strategies, training regimes, model scales, and task domains. We orthogonally combine the three strategies with both training-free and training-based (SFT, offline RL, online RL) approaches, yielding 12 evaluation configurations that cover representative existing methods. Under a unified pipeline—6 LLMs spanning Qwen3.5-2B to Kimi-K2.5-1.1T, 5 benchmarks covering math, code, and science, and unified metrics—we systematically characterize how strategies navigate the efficiency–effectiveness trade-off, how training signals interact with strategy choice, and how the optimal strategy–training combination shifts with model scale and task domain. Then, we further integrate 12+ existing thinking mode switch methods into the pipeline, providing the unified comparison across the full taxonomy. Overall, our contributions are summarized in three aspects as follows: • Unified evaluation framework. We present the first benchmark that systematically covers 12 configurations for the thinking-mode switch, enabling controlled cross-strategy comparison under identical conditions. • Systematic empirical analysis of thinking switch mechanisms. We reveal that: – The three strategy exhibit fundamentally different trade-off profiles: PT achieves a “win-win” (higher accuracy and fewer tokens), RT offers moderate token savings with preserved accuracy, while Spec improves accuracy at additional token cost. – Training gains are strategy-dependent: GRPO achieves the highest token reduction for RT while all training methods maintain comparable accuracy across strategies. – Both effects shift with LLM size scale and task domain: Spec surpasses PT at the 20B and 671B scales, while PT performs better on math and Spec on code tasks. • Open-source baselines and platform. Reference implementations for all 12 configurations and 12+ integrated prior methods, forming a plug-and-play platform for the community. 2 Related Work 2.1 Hybrid-Reasoning LLMs Recent work has introduced LLMs with user-controllable thinking-mode switches that enable flexible allocation of inference compute ( 27 ) . These hybrid-reasoning LLMs offer three forms of control over reasoning depth: • Binary switch. The most common way provides a think / no_think switch, where the former activates extended chain-of-thought and the latter generates direct answers. Current LLMs adopting this design include Qwen3.5 ( 21 ) , DeepSeek-V3.1, Kimi-K2.5, and so on. • Discrete reasoning effort. Certain LLMs expose tiers of reasoning effort, e.g, gpt-oss-20B ( 1 ) introduces High/Medium/Low settings. This approach affords coarse-grained control of test-time compute. • Numeric budget. LLM family like Seed-OSS-36B ( 3 ) also accept explicit token budgets b\leq B_{\max} , enabling fine-grained, continuous control over the token of reasoning. 2.2 Adaptive Thinking-Mode Switch Existing adaptive thinking-mode switch methods can be categorized into three categories based on when and how the mode decision is made. Prompt-Tuning. PT-based methods guide mode selection through prompt engineering within a single inference pass—the model itself decides whether and how deeply to reason. Both training-free and training-based approaches have been explored. Training-free approaches include S1 budget forcing ( 18 ) and TALE token-budget-aware reasoning ( 7 ) . SFT-based methods include OThink-R1 ( 34 ) and HGPO ( 11 ) . RL-based methods include ACPO ( 4 ) , and Think-Only (HGPO) ( 11 ) . DPO-based methods include AdaR1 ( 17 ) and Think-in-Blocks ( 36 ) . Routing. RT-based methods employ an explicit two-stage process: a router first assesses query difficulty and selects the appropriate mode, then the model generates under that mode. Training-free routers include HDFlow ( 32 ) and CP-Router ( 24 ) . SFT-trained routers include Self-Route ( 8 ) , and ThinkSwitcher ( 14 ) . For example, AdaptThink ( 33 ) trains a routing policy via GRPO that decides

Read the complete text