Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning
Tianyuan Shi, Canbin Huang, Bei Li, Xin Chen, Xiaojun Quan, Jingang Wang, Qifan Wang · 2026
Read Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning by Tianyuan Shi, Canbin Huang, Bei Li, Xin Chen, Xiaojun Quan, Jingang Wang, Qifan Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24064.txt.
Introduction
Shi and colleagues argue that teaching a weaker language model by copying a stronger one's answers teaches the wrong lesson. Imitating full solution paths transfers what to answer, step by memorised step, rather than how to reason on new problems. Their 2026 method, Strategy-Guided Policy Optimization, replaces copied paths with reusable strategy descriptions drawn from strong-model responses, written as general problem-solving guidance.
For each problem the student model produces attempts both alone and with strategy guidance, allowing direct comparison. A token-level objective then transfers only the shift that guidance causes into the unguided policy, with safeguards keeping updates stable. Weighting adapts per problem: guidance counts more where the student struggles alone and fades as its own skill grows. Across four mathematics benchmarks and two model families, the approach beats supervised fine-tuning, reinforcement learning, and hybrid baselines, gaining ground on the strongest rival.
Catalog record
| CREATOR | Tianyuan Shi, Canbin Huang, Bei Li, Xin Chen, Xiaojun Quan, Jingang Wang, Qifan Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24064
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning Tianyuan Shi, Canbin Huang, Bei Li, Xin Chen, Xiaojun Quan, Jingang Wang, Qifan Wang Abstract Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather than how to reason . This trajectory-level imitation encourages memorization of instance-specific steps rather than acquisition of transferable problem-solving skills, limiting generalization to novel problems. We propose Strategy-Guided Policy Optimization (SGPO), which replaces instance-level trajectory imitation with reusable strategy distillation . SGPO extracts structured strategy descriptions from strong-model responses and, for each problem, constructs both autonomous and strategy-guided trajectories to enable direct comparison of the model’s behavior with and without strategic guidance. The framework then addresses two key questions. For how to distill, a token-level forward-KL objective selectively transfers the distributional shift induced by strategy conditioning into the unguided policy, with proximal constraints ensuring stability. For when to distill, adaptive instance-level weighting strengthens guidance when autonomous exploration falls short and reduces it as the model’s own competence grows. Experiments on four mathematical benchmarks across two model families show that SGPO consistently outperforms SFT, on-policy RL, and hybrid-policy baselines, improving the average score by 2.2 points over the strongest baseline on Qwen2.5-7B-Instruct. Analysis reveals that the forward-KL objective provides an inherently selective distillation signal that outperforms direct trajectory imitation, and that strategy distillation exhibits complementary scaling with base model capability. 1 Introduction Large language models (LLMs) have demonstrated remarkable reasoning capabilities ( Daya et al. 2025 ; OpenAI et al. 2024 ) , motivating a range of methods to transfer these capabilities to smaller models. Whether through supervised fine-tuning on expert traces ( Daya et al. 2025 ; Ye et al. 2025 ; Wu et al. 2026 ; Zhu et al. 2025 ) or hybrid objectives that integrate expert demonstrations into policy optimization ( Yan et al. 2025 ; Lv et al. 2026 ; Fu et al. 2025 ) , existing approaches operate on the same unit of knowledge transfer: instance-level solution trajectories . The student is trained to reproduce what the expert wrote, that is, specific sequences of reasoning steps for specific problems, but is never taught the reusable problem-solving strategy that explains why those steps were chosen. This trajectory-level imitation encourages memorization of instance-specific patterns rather than acquisition of transferable skills, limiting generalization to novel problems. A natural question arises: Can we shift the distillation target from specific solutions to the reusable problem-solving strategies that make those solutions effective? We propose Strategy-Guided Policy Optimization (SGPO), a framework designed to accomplish this shift. SGPO extracts structured strategy descriptions from strong-model responses. Each description specifies the problem type, the solving approach, and the general procedural steps, without carrying out computations or revealing the answer. Rather than asking the student to reproduce what the expert generates, SGPO uses these descriptions to reshape the student’s own reasoning distribution, distilling how to reason rather than what to answer . A central design principle is that strategic knowledge should be internalized into the model’s unguided policy rather than remain an external dependency unavailable at inference time. To this end, SGPO constructs both autonomous and strategy-guided trajectories for each problem (§ 3.1 ), enabling direct comparison of the model’s behavior with and without strategic guidance. This dual construction serves as the foundation for addressing two complementary questions: (1) How to distill. A token-level forward-KL objective measures the divergence between guided and unguided next-token distributions and selectively distills strategy-critical information back into the unguided policy, while proximal constraints at both trajectory and token levels ensure stability (§ 3.3 , § 3.4 ). (2) When to distill. Adaptive per-instance weighting modulates distillation strength based on the marginal benefit of strategy guidance, strengthening distillation when autonomous exploration falls short and reducing it as the model’s own competence grows (§ 3.4 ). This naturally transitions training from strategy-driven rapid improvement in early stages to autonomous-policy-dominated steady optimization in later stages, without manual scheduling. Crucially, SGPO never imitates any trajectory, whether autonomous or guided. Instead, it distills the distributional shift caused by strategy conditioning, operating at the level of token-level probability changes rather than sequence matching. This enables selective transfer of strategic knowledge while preserving the reasoning diversity the model has already acquired through autonomous exploration. Experiments on four mathematical benchmarks across two model families (Qwen2.5 and Llama-3.2) show that SGPO consistently outperforms strong baselines including SFT, on-policy RL, and state-of-the-art hybrid-policy methods, improving the average score by 2.2 points over the strongest baseline on Qwen2.5-7B-Instruct. Analysis reveals two findings. First, the forward-KL objective provides an inherently selective distillation signal: without any token-level annotation, optimization pressure concentrates on tokens whose probability shifts most under strategy conditioning, which empirically correspond to strategy-critical decision points rather than routine linguistic tokens. Second, strategy distillation exhibits complementary scaling with base model capability: as foundational reasoning competence grows, the ability to internalize strategic guidance increases at a faster rate, suggesting that a minimum reasoning competence is needed to benefit from strategy-level transfer. 2 Related Work 2.1 Supervised Fine-Tuning for LLM Reasoning Training a weaker model on expert reasoning traces is the most common form of reasoning distillation ( Daya et al. 2025 ; Ye et al. 2025 ) . While simple and effective, this trajectory-level distillation is sensitive to data quality ( Ye et al. 2025 ) , prone to memorization ( Chu et al. 2025 ) , and vulnerable to exposure bias ( Wu et al. 2026 ) . Recent work mitigates these issues from RL perspectives along two directions. The first introduces proximal constraints: Wu et al. 2026 downweight the loss on well-learned tokens using the model’s own predictions, while PSFT ( Zhu et al. 2025 ) applies proximal clipping to bound update magnitudes. The second recasts SFT through a reward-optimization lens; for example, IW-SFT ( Qin & Springenberg 2025 ) tightens the RL lower bound via importance weighting on policy probability ratios. These methods improve how the teacher’s output is transferred but do not change what is transferred: the student still imitates specific solutions. In contrast, our work operates at a higher level of abstraction, distilling reusable strategies rather than concrete trajectories. 2.2 Hybrid Policy Optimization for LLM Reasoning Incorporating expert demonstrations into policy optimization is a long-standing theme in RL ( Rajeswaran et al. 2018 ; Nair et al. 2018 ) . Recent LLM methods follow two directions. Unified-loss methods mix expert data into the RL objective: LUFFY ( Yan et al. 2025 ) adds off-policy expert trajectories to GRPO rollout groups with regularized importance sampling; AMPO ( Yuan et al. 2025 ) replaces incorrect on-policy samples with diverse off-policy alternatives; other approaches interleave RL and SFT updates ( Ma et al. 2026 ; Chen et al. 2025 ) ; and SRFT ( Fu et al. 2025 ) unifies these ideas in a single-stage framework with sample-level modulation. Prefix-guided methods use expert trajectories to structure generation: UFT ( Liu et al. 2025 ) progressively masks expert suffixes to encourage autonomy, while BREAD ( Zhang et al. 2025 ) branches rollouts from intermediate expert steps. All these methods transfer knowledge at the level of specific solution steps. Our approach differs in two respects: (i) it conditions on strategy