Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation
Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang, GengRu Chen, Wenbo Su, Bo Zheng · 2025
Read Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation by Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang, GengRu Chen, Wenbo Su, Bo Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16385.txt.
Introduction
Yijia Luo and co-authors take apart long chains of thought in a 2025 paper on teaching smaller models to reason. Distilling a strong teacher's long reasoning into another model works well when the two are close relatives, but the authors find the transfer degrades badly on unrelated models.
Their framework first segments the teacher's reasoning into pieces, then strips out unsolvable or redundant solutions, and finally repairs intermediate error states. The cleaned data teaches more per token. Experiments show better performance and token efficiency than standard distillation. The work suggests the structure of reasoning data, not just its volume, drives the transfer.
Catalog record
| CREATOR | Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang, GengRu Chen, Wenbo Su, Bo Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16385
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deconstructing Long Chain-of-Thought: A Structured Reasoning Optimization Framework for Long CoT Distillation Yijia Luo, Yulin Song, Xingyao Zhang, Jiaheng Liu, Weixun Wang, GengRu Chen, Wenbo Su, Bo Zheng Abstract Recent advancements in large language models (LLMs) have demonstrated remarkable reasoning capabilities through long chain-of-thought (CoT) reasoning. The R1 distillation scheme has emerged as a promising approach for training cost-effective models with enhanced reasoning abilities. However, the underlying mechanisms driving its effectiveness remain unclear. This study examines the universality of distillation data and identifies key components that enable the efficient transfer of long-chain reasoning capabilities in LLM distillation. Our findings reveal that the effectiveness of long CoT reasoning distillation from teacher models like QwQ degrades significantly on nonhomologous models, challenging the assumed universality of current distillation methods. To gain deeper insights into the structure and patterns of long CoT reasoning, we propose DLCoT (Deconstructing Long Chain-of-Thought) —a distillation data enhancement framework. DLCoT consists of three key steps: (1) data segmentation to decompose complex long CoT structures, (2) simplification by eliminating unsolvable and redundant solutions, and (3) optimization of intermediate error states. Our approach significantly improves model performance and token efficiency, facilitating the development of high-performance LLMs. Code and data will be released at: https://github.com/elena-luo/SODE.git. † † footnotetext: * First three authors contributed equally. † † footnotetext: †Corresponding Author: Jiaheng Liu. 1 Introduction Figure 1: Comparative Analysis of Model Distillation Performance Using 6K NuminaMath Correct Answer data. Upper Panel: Qwen and Llama distilled through R1-generated data. Notably, while these fail to fully replicate the ability R1 reported, the Qwen2.5-14B achieves comparable accuracy to the QwQ-distilled Qwen2.5-32B, demonstrating R1’s enhanced cross-model transferability in distillation scenarios. Lower Panel: The Qwen-family models demonstrate superior distillation efficacy compared to Llama when using data generated by QwQ. The Qwen2.5-32B achieves performance parity with the QwQ. Large language models (LLMs) have revolutionized natural language processing (NLP) OpenAI et al. 2024 . A key advancement is the development of chain-of-thought (CoT) prompting Wei et al. 2023 ; Nye et al. 2021 , which significantly enhances LLMs’ reasoning abilities. Models like O1 OpenAI et al., 2024 , R1 DeepSeek-AI et al. 2025 , and QwQ Qwen Team, 2024d leverage CoT prompting to excel at complex, multi-step reasoning tasks, featuring with extremely long CoT. Despite the remarkable capabilities of long CoT reasoning, training language models to exhibit such reasoning demands substantial computational resources. This raises a critical research question: how can we train these models more efficiently and cost-effectively? For closed-source models like OpenAI’s O1 and Google’s Gemini 2.0 Flash, direct training remains inaccessible to developers. Meanwhile, open-source alternatives like DeepSeek’s R1 show significant challenges due to their massive parameter sizes and the complexity of reinforcement learning, making replication impractical for most practitioners. The recently proposed R1 model from Deepseek DeepSeek-AI et al. 2025 demonstrates that direct distillation can also achieve superior reasoning performance. Distillation Hinton et al. 2015 , a technique for transferring knowledge from a large teacher model to a smaller student model, offers a promising avenue for improving efficiency. However, despite these advancements, the fundamental mechanisms by which long CoT data leads to improved reasoning over traditional, shorter CoTs remain poorly understood. We investigate this question by conducting a systematic distillation study across open-source models of varying scales and architectures, focusing particularly on the performance of R1 and QwQ (Figure 1 ). Our findings reveal that the R1-distilled Qwen2.5-14B model Qwen et al. 2025 achieves comparable accuracy to the QwQ-distilled Qwen2.5-32B model. This suggests that distillation of R1-based long CoTs offers superior training efficiency and knowledge transferability compared to the QwQ ones. Figure 2: Figure shows an example of QwQ and R1 on "Simplify 2y+3y+4y ". Different color-coded text blocks represent distinct solution/verification types. In the middle are three special structure we figure out from long CoT data. To understand this efficiency gap, we focus on the analysis of the structure of the "thinking" part of R1 and QwQ. Unlike traditional short CoTs, which typically explore a single reasoning path, long CoTs often explore multiple approaches and verifications, exhibiting structural patterns, namely as linear, tree, and network structures (Figure 2 ). These structures are all built upon repeating schemes and a core "trunk", where the trunk represents the shortest complete reasoning chain leading to the correct answer. For instance, in a complex network structure with a complete CoT represented as A_{1}A_{2}...A_{n} , the correct answer might be derivable from a shorter sequence like A_{1}+A_{4} ; we define A_{1}+A_{4} as the trunk. Our investigation focuses on determining the optimal CoT length—specifically, whether the complete CoT or the trunk represents the "key logic". Furthermore, we observe a prevalent "overthinking" phenomenon Chen et al. 2025 in state-of-the-art LLMs, including Deepseek’s R1 and QwQ (Figure 2 ). Student models trained on these overthinking teacher models inherit this characteristic, leading to inefficient resource utilization on simpler tasks. More critically, this inefficient and repetitive exploration of solutions can induce performance degradation in smaller models, often manifesting as repetitive generation patterns. To address these challenges, we propose DLCoT (Deconstructing Long Chain-of-Thought), an automated data processing framework specifically designed for long CoT distillation. Guided by our analysis, DLCoT integrates three core modules: Intelligent Segmentation, Redundancy Elimination, and Error Correction. Our experiments demonstrate that prioritizing the reasoning trunk while eliminating redundant paths significantly improves distillation efficiency. Interestingly, removing unique, incorrect reasoning paths or detailed calculation/derivation errors does not provide additional benefits. This suggests that the structural richness and diversity of explored approaches are more valuable for distillation than the precise details of individual reasoning steps. Our principal contributions are threefold: • Distillation Validation : We systematically explore the feasibility of distilling knowledge from long CoTs using small-scale datasets, evaluating across various source models. • DLCoT Method : We develop an automated long CoT data processing framework, achieving at least 5\% improvement in token efficiency while improve accuracy of baseline performance across all benchmarks(see Section 3.2 ). • Key insights on long CoT : We identify the key to effective long CoT distillation: the richness of approaches within the reasoning trunk. A diverse set of explored approaches is crucial for stimulating and enhancing the model’s reasoning capabilities. 2 Related Work 2.1 Test-Time Scaling It has been proven effective that CoT enhances the mathematical reasoning of capabilities of LLMs Wei et al. 2023 . Rather than focusing on scaling training-time compute, recent research has emphasized scaling test-time compute, specifically by training reasoning models to generate longer reasoning chains to achieve improved performance. Existing work in this area can be broadly categorized into two main approaches: 1) navigating the model to explore the search space more thoroughly, and 2) adopting human-like reasoning strategies like planning, self-reflection, and self-verification, to train models to generate long reasoning chains. Search Space Exploration. One of the most well-known work is OpenAI’s o1 OpenAI et al., 2024 , which, although the methodology details remain unpublished, has demonstrated significant improvements in answering mathematical and programming questions by scaling reasoning time and chain length. Several open-source efforts have also explored test-time compute scaling. Repeated sampling methods, such as Best-of-N Brown et al. 2024 , have led to performance gains by repeatedly sampling candidate solutions. Other approaches Jiang et al. 2024 ; Qin et al. 2024 have investigated