HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps

Xin Liu, Runsong Zhao, Xinyu Liu, Junhao Ruan, Pengcheng Huang, Shichao Dong, Chunyang Xiao, Chenglong Wang, Changliang Li, Jingbo Zhu, Tong Xiao · 2026

Read HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps by Xin Liu, Runsong Zhao, Xinyu Liu, Junhao Ruan, Pengcheng Huang, Shichao Dong, Chunyang Xiao, Chenglong Wang, Changliang Li, Jingbo Zhu, Tong Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03768.txt.

Introduction

Liu and co-authors compress long chain-of-thought reasoning without losing its detail. Long reasoning traces help models but cost compute and memory. Compression methods condense steps into memory tokens kept at inference. Lost detail then makes later steps more error-prone. HybridThinker keeps thought steps around temporarily as well. Fine-grained detail stays on hand when it is needed most.

Naive access backfires during training. If all past steps stay visible, the model skips the memory tokens and undertrains them. A hybrid training plan masks some steps from later attention. The model must then use memory tokens for storage and recall. Across four reasoning benchmarks, HybridThinker matches the uncompressed baseline. It advances compression state of the art by 5.8 points on average. Ablations credit both the temporary retention and the hybrid training plan. Inference time stays close to prior compressed methods.

Catalog record

CREATORXin Liu, Runsong Zhao, Xinyu Liu, Junhao Ruan, Pengcheng Huang, Shichao Dong, Chunyang Xiao, Chenglong Wang, Changliang Li, Jingbo Zhu, Tong Xiao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps Xin Liu, Runsong Zhao, Xinyu Liu, Junhao Ruan, Pengcheng Huang, Shichao Dong, Chunyang Xiao, Chenglong Wang, Changliang Li, Jingbo Zhu, Tong Xiao Abstract Extended chain-of-thought (CoT) traces improve LLM reasoning but incur substantial computational and memory costs. While existing CoT compression methods mitigate this by condensing thought steps into compact representations via memory tokens and retaining only these representations at inference time, the loss of fine-grained information makes subsequent steps more error-prone. To alleviate this, we propose HybridThinker , where in addition to preserved these representations, thought steps are also temporarily retained to provide fine-grained details. However, we observe that naively keeping thought steps accessible to subsequent steps during training lets the model bypass memory tokens by retrieving information directly from these steps, leaving the model’s ability to compress and retrieve information through memory tokens insufficiently trained. We therefore introduce a hybrid training scheme, in which only some thought steps are directly accessible through attention to subsequent steps, while the other thought steps are masked, forcing the model to use memory tokens for compression and retrieval. Across 4 reasoning benchmarks, HybridThinker matches the uncompressed baseline, advancing the state of the art in CoT compression by 5.8 points on average accuracy with similar inference time. Ablation studies confirm that both temporary thought-step retention and the hybrid training scheme contribute to these gains. 1 Introduction Recent advances in large language model reasoning have increasingly relied on extended chain of thoughts (CoTs), which have been shown to be effective across arithmetic, common sense, and knowledge-intensive tasks ( 26 ; 10 ; 29 ; 6 ; 28 ) . However, as illustrated in Figure 1 (a), these extended CoT traces introduce substantial inference overhead, as the self-attention computation scales quadratically and the KV cache grows linearly with the sequence length, hindering the broader deployment of reasoning models. To alleviate this, two main approaches have been proposed: KV cache pruning, and CoT compression. KV cache pruning reduces memory 32 ; 2 usage by selectively retaining important portions of the KV cache while discarding less critical ones during inference, but discarded information is irrecoverable without full regeneration. CoT compression ( 30 ; 11 ; 16 ; 36 ) condenses lengthy thought steps into compact representations, after which the original thought steps are discarded immediately as we illustrate in Figure 1 (b). Unfortunately, this premature discarding renders the immediately following step more error-prone: key details such as exact numerical values, strict constraints, and intermediate conclusions may be obscured or lost in the compact representations ( 31 ; 19 ) . In multi-step reasoning scenarios, such single-step errors propagate along the chain of thought in a cascading fashion. To enable efficient reasoning in LLMs, we propose HybridThinker , a CoT compression method that mitigates the loss of fine-grained local information caused by the immediate discarding of thought steps in existing CoT compression methods. Similar to existing CoT compression approaches, HybridThinker compresses lengthy thought steps into compact representations via memory tokens and retains those representations after each thought step is completed. However, the thought steps are not immediately discarded in HybridThinker; instead, they are temporarily retained and remain accessible to several subsequent steps. We illustrate the HybridThinker inference in Figure 1 (c). To train HybridThinker, we observe that naively applying the same attention pattern during training as at inference is suboptimal (see Section 4.3 for details) due to shortcut learning: when both memory tokens and thought steps are visible to subsequent steps, the model over-relies on the more readable plain-text thoughts, leaving its ability to compress and retrieve information through memory tokens undertrained. We therefore propose Hybrid Attention for training, combining two complementary mechanisms: Bottleneck Attention , which hides each thought step from subsequent steps, forcing the model to access prior reasoning exclusively through memory tokens; and Shortcut Attention , which keeps each thought step visible to the immediately following few steps before discarding it, adapting the model to the inference-time setting. Figure 1: Comparison of three reasoning paradigms in terms of KV cache management. (a) Standard reasoning retains the KV cache of all original thought steps throughout decoding, achieving high accuracy at the cost of efficiency. (b) Existing CoT compression methods discard the KV cache of original thought steps immediately after compression, reducing memory usage but losing fine-grained local information. (c) HybridThinker (ours) temporarily retains the KV cache of each original thought step to serve several subsequent steps before discarding it, achieving both high accuracy and high efficiency. In this example, the KV cache of the third thought step S_{3} is retained to serve several subsequent steps until S_{6} has been compressed. Experiments demonstrate that HybridThinker achieves an average accuracy improvement of 5.8 points over existing CoT compression methods, matching the performance of the uncompressed baseline. Ablation studies further confirm the effectiveness of Hybrid Attention as a training strategy, showing that it outperforms training with pure Bottleneck Attention or pure Shortcut Attention alone. To summarize our contributions: (1) We propose HybridThinker, in which thought steps are temporarily retained and kept visible to several subsequent steps alongside compact representations during inference, alleviating the loss of fine-grained information. (2) We propose Hybrid Attention for training, which resolves the shortcut learning problem that arises in naive HybridThinker training. (3) We demonstrate that HybridThinker achieves substantial performance improvements over existing CoT compression methods, establishing a new state-of-the-art performance in CoT compression. 2 Related Work We organize related work into two categories: KV cache pruning, which reduces memory by directly discarding less important KV cache entries, and compression techniques, which also discards the original KV cache but first compresses the discarded information into compact representations. Compression techniques can be further divided into prompt compression, a well-established research area, and CoT compression, an emerging direction. KV Cache Pruning. KV cache pruning reduces memory footprint by discarding less important KV cache entries, with methods differing primarily in how importance is estimated. H 2 O ( 32 ) uses accumulated attention weights as the importance criterion. StreamingLLM ( 27 ) treats the first token and the most recent tokens as important due to the attention sink phenomenon. SepLLM ( 2 ) extends StreamingLLM by additionally retaining delimiter tokens, which implicitly compress contextual information. However, these methods face inherent limitations: low cache budgets have been shown to paradoxically lengthen CoT outputs ( 14 ) , and discarded cache becomes inaccessible. Compression Techniques. Existing prompt compression methods ( 1 ; 12 ; 9 ; 17 ; 4 ; 13 ; 25 ; 23 ; 21 ; 22 ; 24 ; 35 ; 33 ; 34 ; 15 ) focus on compressing model inputs , which are fully accessible ahead of time. CoT reasoning, however, is generated incrementally and cannot be accessed in its entirety beforehand, necessitating step-wise compression and rendering most prompt compression techniques directly inapplicable. To fill this gap, several dedicated CoT compression methods have been proposed. LightThinker ( 30 ) compresses each completed thought step into the KV cache of a small set of memory tokens and immediately discards the thought step’s KV cache, retaining only the memory tokens. MEMENTO ( 11 ) instead compresses thought steps into shorter text sequences. MemoSight ( 16 )

Read the complete text