Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs

Yubo Gao, Haotian Wu, Hong Chen, Junquan Huang, Yibo Yan, Jungang Li, Zihao Dongfang, Sicheng Tao, Puay Siew Tan, Jie Zhang, Xuming Hu · 2026

Read Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs by Yubo Gao, Haotian Wu, Hong Chen, Junquan Huang, Yibo Yan, Jungang Li, Zihao Dongfang, Sicheng Tao, Puay Siew Tan, Jie Zhang, Xuming Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01168.txt.

Introduction

Gao and co-authors teach language models to spend thought where it pays in this 2026 paper. Chain of thought boosts reasoning, but it often overthinks: long rationales with no matching gain in accuracy. Flat compression misses that difficulty varies twice over. Problems differ in depth, and steps inside one problem differ in need.

Their framework, HAB, budgets thought in a hierarchy. Between steps, it predicts the right reasoning depth for each problem. Within steps, it learns token budget signals from perplexity based comparisons. An adaptive Pareto objective weighs local quality against local cost. A Fisher information pruner adds fine guidance during training. The generator thus internalizes thrift as a habit, not a post hoc cut. On GSM8K and MATH500, HAB beats standard chain of thought in accuracy, they report. It also cuts token use at the same time.

Catalog record

CREATORYubo Gao, Haotian Wu, Hong Chen, Junquan Huang, Yibo Yan, Jungang Li, Zihao Dongfang, Sicheng Tao, Puay Siew Tan, Jie Zhang, Xuming Hu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01168

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs Yubo Gao, Haotian Wu, Hong Chen, Junquan Huang, Yibo Yan, Jungang Li, Zihao Dongfang, Sicheng Tao, Puay Siew Tan, Jie Zhang, Xuming Hu Abstract Chain-of-Thought (CoT) has significantly enhanced LLM reasoning, yet often incurs substantial computational overhead due to “overthinking”: generating excessively long rationales without commensurate accuracy gains. Existing efficiency methods typically apply uniform compression, which overlooks a critical observation that reasoning complexity is heterogeneous at two distinct granularities: across different problems and within individual reasoning steps. This motivates our principle of Thinking Economically : intelligently allocating computational resources based on intrinsic task and step demands rather than pursuing uniform brevity. We propose Hierarchical Adaptive Budgeter (HAB), a training framework that operationalizes this principle through coarse-to-fine budgeting. At the inter-step level, HAB predicts the optimal reasoning depth for each problem. At the intra-step level, HAB learns step-specific token budgeting signals from PPL-derived step comparisons and an adaptive Pareto optimization objective that captures the local quality-efficiency trade-off, while a Fisher Information-based pruner further provides fine-grained training-time guidance, thereby encouraging the generator to internalize more economical reasoning patterns. Experiments on GSM8K and MATH500 show that HAB not only surpasses standard CoT in accuracy but also reduces token usage, achieving a stronger performance-efficiency trade-off than the compared baselines. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities through Chain-of-Thought (CoT) prompting 38 , which decomposes complex tasks into intermediate steps 3 ; 11 . To further improve CoT, various extensions have been proposed, including self-consistency sampling 35 , Tree-of-Thoughts 42 , Graph-of-Thought 2 , and instruction tuning with reasoning traces 5 . However, these methods introduce significant inefficiencies. Specifically, they often engage in “overthinking” 7 ; 34 , generating excessively long reasoning chains without corresponding accuracy gains. Such redundant steps not only increase computational cost but also risk prematurely exhausting the token budget and introducing logical inconsistencies that undermine answer correctness and clarity 33 . A promising direction to address this issue is imposing token constraints or budgets. Existing methods fall into two categories: pre-defined and learning-based constraints. For pre-defined constraints, such as prompt-based length guidance 40 ; 18 ; 29 ; 6 or post-processing techniques like TokenSkip 39 , selecting an appropriate retention ratio remains challenging. Our pilot analysis in Figure 1 shows that a lower retention ratio reduces token count but causes a sharp drop in accuracy. Therefore, finding a satisfactory balance between efficiency and performance with a static, pre-defined constraint is difficult. (a) Accuracy (b) Average Output Tokens Figure 1: Comparison of performance of TokenSkip with different LLMs backbones under different retention ratios on the GSM8K dataset. To address this, several methods based on learned constraints have been proposed. These include Reinforcement Learning (RL) 24 ; 43 approaches that penalize length, which in practice may be harder to stabilize than direct supervised objectives, and more recent methods like TALE 12 that leverage the LLM’s own capabilities to estimate a global budget first, and induce the LLM to reason under such a budget. While a significant step forward, these methods still focus on learning a single, global token budget for the entire reasoning chain. We contend that a global budget, whether pre-defined or learned, is fundamentally limited because it ignores the varying complexity at two granularities (especially the second granularity): across different problems and within the reasoning steps of a single problem . Our analysis on the MATH500 dataset (Figure 2 ) provides empirical motivation for hierarchical budgeting. At the inter-step level, shown in Figure 2 (a), simpler problems require fewer reasoning steps, while complex ones need longer chains. At the intra-step level, as shown in Figure 2 (b)(c), our experiments on Chain-of-Draft (CoD) show that allowing the model to allocate more tokens to difficult steps (CoD w/relaxation ) leads to significant performance gains. This indicates that different steps have varying token requirements, and a uniform constraint is suboptimal. These findings motivate our principle of “Thinking Economically”: instead of uniformly pursuing brevity, we aim to adaptively allocate resources based on hierarchical task demands. The key is to concentrate computation on difficult problems and reasoning steps , while saving resources on simpler ones. (a) Inter-reasoning step : Optimal step distribution by difficulty. (b) Intra-reasoning step : CoD vs. CoD {}_{w}/_{\text{relaxation}} (Accuracy). (c) Intra-reasoning step : CoD vs. CoD {}_{w}/_{\text{relaxation}} (Avg output tokens). Figure 2: Exploratory experiments on the necessity of dynamically allocating the number of reasoning steps tailored to problems and allocating token budgets tailored to reasoning steps. To realize this vision, we introduce Hierarchical Adaptive Budgeter (HAB), a two-stage training framework for adaptive-complexity reasoning that moves beyond a single global budget toward hierarchical control. HAB first performs inter-step control by predicting a coarse reasoning-depth category for each problem and converting it into a step-range instruction for generation, thereby determining how much reasoning the problem should receive at the global level. It then performs intra-step learning within the generated reasoning process: PPL-derived step comparisons provide supervision on relative step difficulty, an adaptive Pareto optimization objective models the local quality-efficiency trade-off, and a Fisher Information-based pruner supplies additional fine-grained training guidance. Through this coarse-to-fine design, HAB encourages the generator to internalize more economical reasoning behaviors, allocating more computation to harder problems and more demanding steps while avoiding unnecessary verbosity on simpler cases. Our main contributions are: • We propose the principle of “Thinking Economically” and instantiate it with HAB, a coarse-to-fine hierarchical framework that first performs explicit inter-step reasoning-depth control and then learns finer-grained intra-step efficiency patterns during training. • We design an adaptive Pareto optimization scheme to dynamically balance quality and efficiency for each reasoning step. • Experiments on GSM8K and MATH500 show that HAB consistently achieves a stronger performance-efficiency trade-off than the compared baselines, improving accuracy while reducing token usage. 2 Related Work Chain-of-Thought Reasoning . Chain-of-Thought (CoT) prompting 38 enhances reasoning in Large Language Models (LLMs) by decomposing solutions of complex problems into intermediate steps. Early works demonstrated its effectiveness in both few-shot 38 and zero-shot 17 settings, but standard CoT often suffers from error propagation due to its linear reasoning path. To mitigate this limitation, subsequent research has introduced structured reasoning strategies. Least-to-Most prompting 49 addresses compositional generalization by sequentially solving decomposed sub-problems. Tree-of-Thoughts (ToT) 42 enables non-linear exploration through tree-search algorithms with backtracking. Self-Consistency 35 improves stability by sampling diverse reasoning trajectories and applying majority voting. Auto-CoT 48 further automates exemplar construction via clustering and diversity selection, reducing manual annotation costs. Beyond inference-time prompting, training-stage methods have been proposed to internalize reasoning capabilities. Symbolic CoT Distillation 19 transfers reasoning skills from teacher models to smaller students. Self-Taught Reasoner (STaR) 45 establishes an iterative bootstrapping loop where models improve by training on their own generated rationales. Recent work addresses practical constraints such as robustness 23 , generalization to unseen tasks 44 , and inference efficiency 41 . CoT has also expanded into multimodal domains 36 and lightweight adaptation for small-scale models 51 . CoT Token Constraints. Existing methods for controlling token usage fall into two paradigms: pre-defined constraints and learned constraints. Pre-defined Token Constraints. These methods fix the efficiency strategy via a static constraint. The simplest form is prompt-based length guidance , which requests a token or

Read the complete text