Structural Rationale Distillation via Reasoning Space Compression

Jialin Yang, Jiankun Wang, Jiajun Wu, Henry Leung, Jiayu Zhou, Steve Drew · 2026

Read Structural Rationale Distillation via Reasoning Space Compression by Jialin Yang, Jiankun Wang, Jiajun Wu, Henry Leung, Jiayu Zhou, Steve Drew (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07139.txt.

Introduction

Jialin Yang and five co-authors fix a quiet problem in teaching small language models to reason: teacher examples are inconsistent. For similar problems, a large teacher may show wildly different solution styles, burdening the student with noisy supervision. Their method constrains the teacher to a small bank of reusable reasoning paths.

For each question, the system retrieves the closest path and has the teacher follow it, keeping style consistent yet varied across problem types. Theory ties bank size to coverage, pointing to a best middle size that experiments confirm. Across five reasoning benchmarks and two student models, the method beats standard distillation approaches while using fewer tokens.

Catalog record

CREATORJialin Yang, Jiankun Wang, Jiajun Wu, Henry Leung, Jiayu Zhou, Steve Drew
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07139

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structural Rationale Distillation via Reasoning Space Compression Jialin Yang, Jiankun Wang, Jiajun Wu, Henry Leung, Jiayu Zhou, Steve Drew Abstract When distilling reasoning from large language models (LLMs) into smaller ones, teacher rationales for similar problems often vary wildly in structure and strategy. Like a chef who makes the same dish differently each time, this inconsistency burdens the student with noisy supervision that is hard to internalize. We propose Distillation through Reasoning Path Compression (D-RPC), which constrains the teacher to follow a compact, dynamically maintained bank of reusable high-level reasoning paths. For each training question, D-RPC retrieves the most relevant path and conditions the teacher to follow it, producing rationales that are consistent across similar problems yet diverse enough to cover different problem types. A PAC-Bayes analysis formalizes the resulting trade-off between bank size and coverage: smaller banks reduce supervision entropy but risk coverage gaps, and the generalization bound identifies an optimal intermediate size confirmed by our ablations. Across five math and commonsense reasoning benchmarks with two student models, D-RPC consistently outperforms chain-of-thought distillation, freeform rationale generation, direct distillation, and structured-supervision baselines, while using fewer tokens than template-heavy alternatives. 1 Introduction Knowledge distillation trains a small language model (SLM) to reason by imitating a large language model (LLM) teacher ( 34 ; 27 ) . However, the teacher’s rationales for similar problems often vary substantially in structure and strategy. An apprentice chef shown a different recipe for the same dish each day learns more slowly, even when every recipe is valid. Distillation faces the same dynamic. Asked the same type of math problem repeatedly, the teacher may generate a “compute rate, then combine quantities” solution once and an “estimate total, then subtract difference” solution the next ( 26 ; 2 ) . This rationale divergence has three costs. First, the student sees many one-off patterns instead of reusable strategies, making internalization harder ( 8 ) . Second, covering all valid solution paths requires many teacher generations, inflating cost ( 51 ) . Third, highly variable supervision can amplify hallucination during fine-tuning ( 34 ) . Prior work has tried to mitigate this variance through structured reasoning. Approaches like Buffer-of-Thought ( 45 ) and ReasonFlux ( 44 ) reuse hierarchical thought templates to improve LLM reasoning at inference time. However, these verbose templates impose heavy context loads that SLMs, with their limited capacity and context windows, struggle to exploit ( 27 ; 54 ) . The core question remains open: can we compress the teacher’s reasoning space into a compact, consistency-preserving structure before distilling it to a student? We propose Distillation through Reasoning Path Compression (D-RPC), which addresses this question by giving the teacher a “cookbook”, that is, a compact bank of canonical, high-level reasoning paths, and asking it to follow these paths when generating rationales. Just as a chef who consults a cookbook produces more consistent dishes than one who improvises freely, a teacher conditioned on a shared set of reasoning paths produces more consistent supervision. D-RPC works in two stages. First, it builds the cookbook: a seed set of teacher-solved problems is clustered by intent to identify canonical solution strategies, forming the reasoning path bank . Second, it teaches from the cookbook: for each new training question, the most relevant paths are retrieved from the bank, and the teacher generates a detailed rationale conditioned on the retrieved path. If the teacher discovers a novel valid solution not in the bank, it is buffered and periodically merged back so that the cookbook grows over time. The student SLM is then fine-tuned via LoRA on the resulting consistent supervision. Our main contributions are as follows. 1. We propose D-RPC, which compresses the teacher’s reasoning space into a compact, dynamically maintained reasoning path bank. By routing similar questions to shared canonical paths, D-RPC reduces rationale divergence and provides more consistent supervision for SLMs. 2. We formalize a trade-off between bank size and coverage via a PAC-Bayes generalization bound whose deviation term contains an explicit \log K_{\mathrm{bank}} factor: smaller banks lower supervision entropy but increase coverage slack, and vice versa. The bound predicts an optimal intermediate bank size, which we confirm through ablations and empirical study of the bound’s key quantities. 3. Across five benchmarks and two student models (Llama 3.1 8B Instruct, Qwen 3 1.7B), D-RPC consistently improves over CoT, freeform, direct-distillation, and structured-supervision baselines, while using fewer tokens than template-heavy alternatives. 2 Related Work Distilling reasoning from teachers to students. UNICOTT iteratively constructs structured explanations to reduce supervision noise ( 52 ) . COTCD introduces difficulty-aware curricula for stable student learning ( 50 ) , and RLKD captures implicit multi-branch structures instead of mimicking a single trajectory ( 42 ) . ReasonFlux-PRM scores intermediate steps to filter distillation data and provide dense rewards ( 53 ) . MCC-KD generates multiple rationales per question and minimizes a bidirectional KL divergence across their answer distributions to enforce intra-question consistency ( 6 ) , and MIND addresses misalignment between a teacher’s “optimal” rationale and the student’s evolving capacity by synthesizing capability-adaptive multi-perspective supervision through a teaching-assistant network ( 12 ) . Nevertheless, these frameworks either pass through whatever rationales the teacher generates or operate on the student-side loss, so the high variance of similar problems receiving structurally different rationales propagates into supervision ( 26 ; 2 ) . D-RPC addresses this upstream by compressing the teacher’s reasoning space across questions before rationale generation. Reusing reasoning processes at inference time. A parallel line of work treats reasoning as a reusable process rather than a one-off generation. Buffer of Thought and ReasonFlux reuse thought templates ( 45 ; 44 ) . RoT retrieves thought steps from similar problems ( 1 ) , and Thought Propagation reuses solutions through analogical propagation ( 49 ) . In the agent setting, ReasoningBank ( 28 ) accumulates a memory of natural-language strategies extracted from past successful and failed trajectories and merges them into the agent’s prompt, with no weight updates. All of these methods improve LLM or agent performance at inference time by strengthening prompts. However, they do not address the variance of the supervision signal used to train smaller students. D-RPC instead operates at distillation time , compressing the teacher’s supervision distribution so that consistent reasoning structure is internalized into the student’s weights and no retrieval is required at deployment. Reasoning-path supervision for SFT. Skip-Thinking refines CoTs through block training, allowing non-inference blocks to be skipped ( 7 ) . EDIT identifies key decision-making steps by comparing paired CoTs and upweights them ( 13 ) . QR-Distill improves multi-path refinement through quality filtering and peer model teaching ( 22 ) . Reasoning Scaffolding abstracts teacher rationales into sequences of discrete semantic signals and trains the student to predict them alongside each step ( 40 ) , enforcing structure within each trace rather than across related questions. Together, these methods restructure or rescaffold rationales on the student side, but do not reduce the variance of the rationales themselves. D-RPC is complementary: it constrains the teacher to generate lower-variance rationales in the first place, so that any downstream reweighting, filtering, or scaffolding operates on a cleaner signal. 3 Methodology Returning to our cooking analogy, D-RPC builds a cookbook, teaches from it,

Read the complete text