Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning

Jinghui Lu *, Haiyang Yu *, Siliang Xu *, Shiwei Ran, GuoZhi Tang, Siqi Wang, Bin Shan, Teng Fu, Hao Feng, Jingqun Tang, Han Wang, Can Huang · 2025

Read Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning by Jinghui Lu *, Haiyang Yu *, Siliang Xu *, Shiwei Ran, GuoZhi Tang, Siqi Wang, Bin Shan, Teng Fu, Hao Feng, Jingqun Tang, Han Wang, Can Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15154.txt.

Introduction

Jinghui Lu and co-authors argue in a 2025 paper that long reasoning is not always good for language models. Step-by-step reasoning has boosted scores on hard tasks, but it can also drag down simple ones and waste computing. The team shows that extra reasoning steps sometimes lower accuracy rather than raise it.

Their framework first drafts a short answer and checks the model's confidence in it. Only when confidence is low does the model switch to long reasoning. This adaptive routing, tested on text and image tasks, aims to spend effort where it pays. The lesson is to match thinking length to the problem, not to maximize it.

Catalog record

CREATORJinghui Lu *, Haiyang Yu *, Siliang Xu *, Shiwei Ran, GuoZhi Tang, Siqi Wang, Bin Shan, Teng Fu, Hao Feng, Jingqun Tang, Han Wang, Can Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15154

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoning Jinghui Lu *, Haiyang Yu *, Siliang Xu *, Shiwei Ran, GuoZhi Tang, Siqi Wang, Bin Shan, Teng Fu, Hao Feng, Jingqun Tang, Han Wang, Can Huang Abstract Recent advancements in reasoning have significantly enhanced the capabilities of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) across diverse tasks. However, excessive reliance on chain-of-thought (CoT) reasoning can impair model performance and brings unnecessarily lengthened outputs, reducing efficiency. Our work reveals that prolonged reasoning does not universally improve accuracy and even degrade performance on simpler tasks. To address this, we propose C ertainty-based A daptive R easoning (CAR), a novel framework that dynamically switches between short answers and long-form reasoning based on the model’s perplexity. CAR first generates a short answer and evaluates its perplexity, triggering reasoning only when the model exhibits low confidence ( i.e., high perplexity). Experiments across diverse multimodal VQA/KIE benchmarks and text reasoning datasets show that CAR outperforms both short-answer and long-form reasoning approaches, striking an optimal balance between accuracy and efficiency. † † footnotetext: ∗ Equal contribution. 🖂 Corresponding authors. 1 Introduction Chain-of-thought (CoT) [ 1 , 2 , 3 , 4 , 5 , 6 ] reasoning was first introduced to improve the performance of large language models (LLMs) on complex reasoning tasks. By prompting “Let’s think step by step” , LLMs generates intermediate reasoning steps before outputting the final answer, leading to higher performance compared to directly outputting the answer without such reasoning. Recent reasoning models [ 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 ] incorporate CoT reasoning into their training procedures, enabling reasoning without requiring explicit prompting. These reasoning models yield substantial performance improvements especially in tasks with well-structured and deterministic answers, such as mathematics and coding for LLMs [ 18 ] , as well as object detection and counting for Multimodal Large Language Models (MLLMs) [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ] . However, recent studies [ 27 , 18 , 28 ] indicate that long-form reasoning primarily enhances LLM performance on math and symbolic tasks, but offers marginal gains on other types of tasks. Sui et al. [27] demonstrate that “overthinking” can even hurt model performance in some simpler tasks, highlighting the need for selective application of CoT reasoning. In other words, current models commonly suffer from a “one-size-fits-all” application of reasoning strategies. Regardless of the complexity of individual questions, they uniformly employ a long-form reasoning process, resulting inefficient inference and failing to meet real-world demands for both speed and accuracy. To examine this further, we have performed a preliminary experiment using Qwen2.5-0.5B on three text-rich Visual Question Answering (VQA) and Key Information Extraction (KIE) datasets, where answers are mostly extractive or fact-based. We compare the performance of reasoning-based models against short-answer models (see Appendix A for experimental details), with the results summarized in Table 1 . Experimental results support the finding that reasoning does not universally improve performance as evidenced by the higher accuracy of short-answer models (45.7 vs. 34.1). Visualizations (see Appendix B ) demonstrate that for certain questions, prolonged reasoning steps may introduce noise, degrading accuracy compared to concise, direct answers. More critically, the high token consumption required for reasoning imposes substantial practical costs. As shown in Table 1 , responses with reasoning average 572.3 tokens compared to just 13.9 for direct answers. This inefficiency leads to increased computational overhead, slower inference times, and higher monetary and energy expenses without commensurate accuracy gains in tasks where concise answers suffice. DocVQA ChartQA FUNSD Avg. Model Acc #Token Acc #Token Acc #Token Acc #Token Qwen2.5-0.5B Short 53.9 11.6 25.3 13.9 58.0 16.2 45.7 13.9 Qwen2.5-0.5B Long 41.1 505.7 16.6 727.5 44.5 483.6 34.1 572.3 Table 1: Performance comparison of reasoning-based vs. short-answer models using Qwen2.5-0.5B. This raises the research question: “Can LLM/MLLM adaptively determine when to engage in reasoning, selectively activating it for questions where simple answers are insufficient to balance computational efficiency and task performance?” To address this challenge, we propose C ertainty-based A daptive R easoning (CAR), a novel framework that adaptively switches between generating short answers and long-form reasoning based on the model’s confidence. Specifically, CAR first produces a short answer and evaluates its confidence using perplexity (PPL), a measure of the model’s uncertainty in its response. A high PPL score indicates low confidence. Specifically, we model the relationship between PPL and answer correctness using a Gaussian distribution. For each new short response, we compute the likelihood of its PPL under both Gaussians—if the probability of being incorrect ( i.e., PPL’s likelihood under the “incorrect” Gaussian) exceeds that of being correct, CAR triggers a full reasoning process to refine the answer. This adaptive approach ensures computational efficiency without sacrificing performance, activating complex reasoning only when necessary. In this work, we begin with a pilot study to examine the relationship between model confidence and the correctness of model responses, validating the feasibility of using PPL score as an indicator of prediction confidence (Section 3 ). Building on these observations, we propose Certainty-based Adaptive Reasoning, a method that dynamically routes LLM/MLLM to either short-answer or CoT reasoning (Section 4 ).Our experiments (Section 5 ) demonstrate that CAR achieves higher prediction accuracy compared to both complete short-answer and complete long-form reasoning approaches, while maintaining greater inference efficiency than long-form reasoning across both LLMs and MLLMs. Notably, CAR not only excels in simple tasks such as VQA and KIE but also outperforms conventional reasoning methods in challenging tasks ( e.g., mathematics) where long-form reasoning is typically assumed to be necessary. In summary, the contributions of this paper are as follows: • We propose CAR, a novel framework that dynamically switches between concise answers and long-form reasoning based on model confidence, achieving an optimal balance between accuracy and computational efficiency. • Through comprehensive pilot studies, we validate PPL can be a reliable indicator of model confidence, establishing its relationship with answer correctness via Gaussian modeling. • Extensive experiments across LLMs and MLLMs demonstrate CAR’s superiority, showing consistently accuracy improvement over short-answer and long-form reasoning counterparts while significantly reduce the inference tokens. • Our experiments demonstrate that CAR surpasses previous state-of-the-art reasoning token reduction methods. When tested with Qwen2.5, CAR improves overall accuracy by 6.9% while reducing token consumption by 21.4%. Similarly, with Llama3.1, it achieves a 5.5% accuracy gain and a 39.0% reduction in tokens used. To the best of our knowledge, CAR stands as a pioneering approach in adaptive reasoning that automatically determines when to employ reasoning based on real-time confidence estimation, offering a principled solution to the efficiency-accuracy trade-off in LLM/MLLM reasoning. 2 Related Work 2.1 Reasoning Frameworks for LLM/MLLM Recent advances in reasoning models, such as OpenAI’s o1 [ 8 ] , DeepSeek R1 [ 11 , 10 ] and QwQ [ 14 ] , have significantly enhanced the ability of LLMs to solve complex tasks, especially in tasks with well-structured and deterministic answers. Chain-of-Thought (CoT) prompting [ 1 , 2 , 3 , 4 , 5 , 6 ] is a foundational approach that decomposes problems into intermediate

Read the complete text