Rotation-Preserving Supervised Fine-Tuning
Hangzhan Jin 1,2,∗ Tianwei Ni 1,3 Lu Li 1,3 Pierre-Luc Bacon 1,3,5 Mohammad Hamdaqa 2 Doina Precup 1,4,5,6 1 Mila - Quebec AI Institute 2 Polytechnique Montréal 3 Université de Montréal 4 McGill University 5 CIFAR AI Chair 6 Google DeepMind · 2026
Read Rotation-Preserving Supervised Fine-Tuning by Hangzhan Jin 1,2,∗ Tianwei Ni 1,3 Lu Li 1,3 Pierre-Luc Bacon 1,3,5 Mohammad Hamdaqa 2 Doina Precup 1,4,5,6 1 Mila - Quebec AI Institute 2 Polytechnique Montréal 3 Université de Montréal 4 McGill University 5 CIFAR AI Chair 6 Google DeepMind (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10973.txt.
Introduction
Hangzhan Jin and five colleagues present Rotation-Preserving Supervised Fine-Tuning, a fix for the way standard fine-tuning trades in-domain gains against out-of-domain losses. Prior work links that degradation to shifts in dominant singular subspaces of pretrained weights. Finding loss-sensitive directions directly costs too much at language-model scale.
RPSFT instead punishes movement in the top singular-vector block, projected, of every pretrained weight matrix, curbing pointless rotation while keeping task learning intact. Across model families and sizes trained on math reasoning data, it improves the in-domain versus out-of-domain trade-off over standard fine-tuning and strong baselines. It also better preserves pretrained representations and gives stronger starting points for later reinforcement-learning tuning. Fine-tuners get a cheap proxy for Fisher-sensitive directions.
Catalog record
| CREATOR | Hangzhan Jin 1,2,∗ Tianwei Ni 1,3 Lu Li 1,3 Pierre-Luc Bacon 1,3,5 Mohammad Hamdaqa 2 Doina Precup 1,4,5,6 1 Mila - Quebec AI Institute 2 Polytechnique Montréal 3 Université de Montréal 4 McGill University 5 CIFAR AI Chair 6 Google DeepMind |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85721 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10973
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rotation-Preserving Supervised Fine-Tuning Hangzhan Jin 1,2,∗ Tianwei Ni 1,3 Lu Li 1,3 Pierre-Luc Bacon 1,3,5 Mohammad Hamdaqa 2 Doina Precup 1,4,5,6 1 Mila - Quebec AI Institute 2 Polytechnique Montréal 3 Université de Montréal 4 McGill University 5 CIFAR AI Chair 6 Google DeepMind Abstract Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work suggests that this degradation is related to changes in dominant singular subspaces of pretrained weight matrices. However, directly identifying loss-sensitive directions with Hessian or Fisher information is computationally expensive at LLM scale. In this work, we propose preserving projected rotations in pretrained singular subspaces as an efficient proxy for Fisher-sensitive directions, which we call Rotation-Preserving Supervised Fine-Tuning (RPSFT). RPSFT penalizes changes in the projected top- k k singular-vector block of each pretrained weight matrix, limiting unnecessary rotation while preserving task adaptation. Across model families and sizes trained on math reasoning data, RPSFT improves the in-domain/OOD trade-off over standard SFT and strong SFT baselines, better preserves pretrained representations, and provides stronger initializations for downstream RL fine-tuning. Code is available at https://github.com/jinhangzhan/RPSFT . † † footnotetext: ∗ Corresponding author: [email protected] . Figure 1 : Method overview. RPSFT modifies SFT by adding a projected-block anchor in the pretrained SVD basis while retaining full-parameter task adaptation. The top- k pretrained singular block is protected, complementary directions remain free, and the resulting \theta_{\text{SFT}} initializes RLFT. 1 Introduction Large language models are commonly post-trained with supervised fine-tuning (SFT) followed by reinforcement learning fine-tuning (RLFT). Although SFT improves tasks represented in the post-training data, it can degrade out-of-domain (OOD) capabilities, a form of forgetting caused by excessive specialization ( Zhu et al. 2025c ; Wu et al. 2025 ) . Prior work links this degradation to geometric drift, especially rotations of dominant singular directions in pretrained weight matrices ( Jin et al. 2025 ; Zhu et al. 2025b ) . Since these directions are associated with high-variance and high-curvature structure ( Haink 2023 ) , preserving them may protect general-purpose capabilities during SFT. This motivates our question: Can supervised fine-tuning preserve OOD generalization by limiting unnecessary geometric drift while retaining task adaptation? To address this question, we propose Rotation-Preserving Supervised Fine-Tuning (RPSFT), a simple regularization method motivated by the observed overlap between dominant singular subspaces and Fisher-sensitive directions tied to OOD forgetting. RPSFT penalizes changes in the pretrained top- k singular-vector block of each selected weight matrix. Unlike freezing or hard gradient projection, it anchors only the dominant pretrained block while leaving complementary directions free to adapt, and integrates into standard SFT pipelines without additional data or task boundaries. Concretely, for each selected matrix \mathbf{W} , RPSFT precomputes the pretrained top- k left and right singular-vector bases \mathbf{U}^{(k)}_{0} and \mathbf{V}^{(k)}_{0} , and adds the Frobenius norm \lambda\|(\mathbf{U}^{(k)}_{0})^{\top}(\mathbf{W}-\mathbf{W}^{0})\mathbf{V}^{(k)}_{0}\|_{F}^{2} to the SFT loss. This formulation preserves the expressivity of supervised fine-tuning while stabilizing general-purpose capability learned during pretraining. Figure 1 summarizes the overall RPSFT post-training workflow and Algorithm 1 summarizes the algorithm. This method integrates directly into standard SFT pipelines and requires no additional data or task boundaries. We evaluate RPSFT with full-parameter fine-tuning across Llama ( Grattafiori et al. 2024 ) and Qwen ( Team 2024 ) checkpoints trained on OpenR1-Math ( Hugging Face 2025 ) . We treat math benchmarks as in-domain and general reasoning, safety, and knowledge benchmarks as OOD. Across model families and sizes, RPSFT improves the ID/OOD trade-off over SFT and strong SFT baselines, summarized in Figure 4 . To understand why it works, we analyze representation drift after supervised fine-tuning. Comparing the hidden states of the base model and the checkpoint after SFT, we show that RPSFT better preserves the pretrained representation geometry. We then evaluate downstream reinforcement learning with DAPO ( Yu et al. 2025 ) , a variant of GRPO ( Shao et al. 2024 ) , across all three model sizes and find that RPSFT provides strong downstream initializations and consistently higher or competitive final RL performance. Together, RPSFT contributes a simple projected-subspace regularizer for SFT, consistent improvements in the ID/OOD trade-off across Llama ( Grattafiori et al. 2024 ) and Qwen ( Team 2024 ) models, and empirical and theoretical evidence that preserving dominant pretrained subspaces reduces rotation, protects hidden-state representations, and improves the forgetting–adaptation trade-off. 2 Preliminaries Modern post-training for reasoning language models typically proceeds in two stages: supervised fine-tuning (SFT) on curated instruction data, followed by reinforcement learning fine-tuning (RLFT) on task-level rewards ( DeepSeek-AI 2025 ; OpenAI et al. 2024 ; Bai et al. 2023 ) . SFT and PEFT. Supervised fine-tuning adapts pretrained parameters \theta_{0} on labeled pairs \mathcal{D}=\{(x,y)\} by minimizing the negative log-likelihood \mathcal{L}_{\mathrm{SFT}}(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{D}}[-\log\pi_{\theta}(y\mid x)] , where x is an input prompt, y is the target response, and \pi_{\theta}(y\mid x) is the model distribution with parameters \theta initialized from \theta_{0} . In this work, the main setting is full-parameter SFT, where all model parameters are updated. We include vanilla LoRA ( Hu et al. 2021 ) as a PEFT baseline in Appendix D . RLFT and DAPO. After SFT, the model can be further optimized with reinforcement learning. We use DAPO ( Yu et al. 2025 ) that samples a group of G responses from the rollout policy \pi_{\theta_{\mathrm{old}}}(\cdot\mid x) , computes a normalized group-relative advantage \hat{A}_{i}=\frac{r_{i}-\frac{1}{G}\sum_{j=1}^{G}r_{j}}{\operatorname{std}(\{r_{j}\}_{j=1}^{G})+\varepsilon} with r_{i}=r(x,y_{i}) and small numerical constant \varepsilon , and uses the token-level PPO ratio ( Schulman et al. 2017 ) \rho_{i,t}(\theta)=\frac{\pi_{\theta}(y_{i,t}\mid x,y_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(y_{i,t}\mid x,y_{i,<t})} . The resulting surrogate objective is {\footnotesize\mathcal{J}_{\mathrm{DAPO}}(\theta)=\mathbb{E}_{x,\{y_{i}\}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid x)}\!\left[\frac{1}{\sum_{i=1}^{G}|y_{i}|}\sum_{i,t}\min\!\Big(\rho_{i,t}\hat{A}_{i},\,\operatorname{clip}\!\big(\rho_{i,t},1-\epsilon_{\mathrm{low}},1+\epsilon_{\mathrm{high}}\big)\hat{A}_{i}\Big)\right].} (1) Here i indexes the G sampled responses for each prompt, t ranges over the valid tokens of response y_{i} , |y_{i}| is the response length, and \epsilon_{\mathrm{low}},\epsilon_{\mathrm{high}} are the PPO clipping thresholds. Compared with GRPO, DAPO averages the clipped policy objective over valid tokens rather than over whole responses. 3 Why Preserving Singular Vectors Mitigates Forgetting Post-training must balance rapid adaptation , which efficiently improves upon the target task, and mitigating forgetting , which preserves capabilities already encoded in the pretrained model ( Lyle et al. 2023 ; Lu et al. 2025 ) . Prior work has linked dominant subspaces to important functional properties of neural networks ( Haink 2023 ) , and excessive rotation of leading singular subspaces in large language models has been associated with out-of-domain degradation ( Jin et al. 2025 ) . In this section, we complement this view by connecting this trade-off to forgetting. We analyze a local second-order expansion around the pretrained weights, following the standard curvature view used in second-order optimization ( Martens 2010 ) . Let a vectorized base-model weight be w\in\mathbb{R}^{d} , the post-trained weight be w^{\prime}=w+\Delta w , and l denote the loss on a task. The local expansion is l(w^{\prime})\approx l(w)+\underbrace{\langle\nabla l(w),\Delta w\rangle}_{\text{adaptation}}+\underbrace{\frac{1}{2}\Delta w^{\top}\nabla^{2}l(w)\Delta w}_{\text{forgetting}}. (2) The first-order term captures task adaptation: if the update aligns with the target gradient, the loss decreases. The second-order term captures the need to mitigate forgetting: moving along high-curvature directions can sharply increase the loss of capabilities that the pretrained model already fits well. For OOD retention, forgetting is often dominated by this curvature term once the base model already achieves local-optimal OOD performance where the first-order is close to zero. Figure 2 : Fisher-projected total sum of squared gradient norms (gradient energy) captured by top- r\times r SVD