Alignment Dynamics in LLM Fine-Tuning

Yuhan Huang, Huanran Chen, Yinpeng Dong · 2026

Read Alignment Dynamics in LLM Fine-Tuning by Yuhan Huang, Huanran Chen, Yinpeng Dong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18309.txt.

Introduction

Yuhan Huang's 2026 paper builds a unified account of why language-model alignment breaks under fine-tuning. The authors define a tractable alignment score with a closed-form update, splitting updates into two competing forces. A rebound force, set by present alignment and distribution narrowness, pulls back toward prior alignment. A driving force, set by how data lines up with posteriors over aligned and non-aligned completions, pushes toward new data. Narrower posteriors strengthen reversal, and prior alignment leaves an imprint that speeds re-alignment on re-exposure, called rehearsal priming.

Catalog record

CREATORYuhan Huang, Huanran Chen, Yinpeng Dong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84494
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18309

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Alignment Dynamics in LLM Fine-Tuning Yuhan Huang, Huanran Chen, Yinpeng Dong Abstract Although Large Language Models (LLMs) achieve strong alignment through supervised fine-tuning and reinforcement learning from human feedback, the alignment is often fragile under subsequent fine-tuning. Existing explanations either attribute alignment fragility to gradient geometry or characterize it as a distributional shift in model outputs, yet few provide a unified account that bridges parameter-space learning dynamics with function-space alignment behavior during fine-tuning. In this work, we introduce a tractable alignment score and derive its closed-form update during fine-tuning, yielding a unified framework for alignment dynamics. Our analysis decomposes alignment updates into two competing components: a Rebound Force , governed jointly by the current alignment state and the narrowness of model distribution, and a Driving Force , determined by how the training distribution aligns with outcome-conditioned posteriors over aligned and non-aligned completions. This decomposition explains why prior alignment can be reversed by later fine-tuning and why narrower posterior structure strengthens such reversal. Moreover, our framework predicts a Rehearsal Priming Effect : prior alignment leaves a latent posterior imprint that amplifies the effective Driving Force upon re-exposure, leading to faster re-alignment. We validate these predictions across safety alignment, emergent misalignment, and sentiment settings, demonstrating consistent alignment reversal and accelerated re-alignment under re-exposure. In addition, controlled experiments in safety alignment confirm the predicted dependence of rebound strength on posterior narrowness. Together, these results provide a unified dynamical perspective on how alignment is disrupted and reactivated during LLM fine-tuning. 1 Introduction Large Language Models (LLMs) have achieved remarkable success in aligning their behavior with human values through a combination of supervised fine-tuning, preference optimization, and post-training techniques [ 22 , 44 , 6 , 30 , 27 , 2 , 16 ] . Despite this progress, alignment remains persistently fragile, posing a major obstacle to reliable deployment. Recent studies show that safety guardrails can be easily compromised by fine-tuning on small amounts of harmful data, and even on benign data [ 26 , 39 , 3 ] . This pervasive vulnerability underscores the urgent necessity for a unified theoretical framework that moves beyond empirical observations to provide a rigorous, quantitative characterization of how alignment evolves, degrades, and recovers during the fine-tuning process. Existing explanations for alignment fragility can be categorized into gradient-based and distribution-based perspectives. Gradient-based analyses [ 43 , 37 , 10 ] base on first-order geometry to decompose fine-tuning gradients into safety-relevant subspaces, attributing degradation to insufficient projection onto safety-aligned directions. While geometrically intuitive, these analyses do not explicitly map parameter-level updates to functional changes in output probabilities, and thus leave unclear how data distributions shape the resulting dynamics. On the other hand, distribution-based studies characterize alignment as a shallow modification of the model’s output distribution. This perspective identifies disruption mechanisms such as compression-driven reversion toward the pre-training distribution [ 14 ] , distributional reweighting [ 15 , 5 ] , and asymmetric forgetting of safety-relevant behaviors [ 45 ] . However, this line of work remains largely static: it describes the resulting distributional shift, but does not explain the optimization dynamics through which misalignment emerges during fine-tuning. Figure 1: Illustration of alignment dynamics in LLM fine-tuning. (a) Rebound Force : The strength of the intrinsic rebound effect is determined by the narrowness of the posterior distribution, with lower diversity leading to stronger rebound force. (b) Driving Force : Alignment evolution is governed by the relative alignment of the fine-tuning distribution p with the outcome-conditioned posteriors \tilde{\pi}^{+} and \tilde{\pi}^{-} over aligned and non-aligned completions. (c) Rehearsal Priming Effect : Prior fine-tuning reshapes the posterior structure, which amplifies the effective driving force upon re-exposure and leads to faster re-alignment. Taken together, these perspectives underscore a central challenge of how to bridge parameter-space learning dynamics with function-space alignment behavior. While prior work has begun to characterize one-step probability dynamics [ 28 ] , it remains unclear how to define alignment in a sufficiently general yet tractable form, and how such a term evolves under fine-tuning. We address this challenge by introducing a precise alignment score S and deriving its closed-form evolution under learning dynamics. The resulting decomposition reveals a non-trivial relationship between alignment change and outcome-conditioned posterior distributions over aligned and non-aligned completions. In the case of supervised fine-tuning, this decomposition further exposes two competing forces, a Rebound Force and a Driving Force , as shown in Figure 1 . The Rebound Force ( \mathcal{F}_{\text{rebound}} ) is an intrinsic self-interaction term of the model’s conditional output distribution. It is modulated by both the current alignment state and the sharpness of the posterior distribution. This structure implies a bidirectional instability: both strongly aligned and strongly misaligned states become more reversible under subsequent perturbations (Section 3.2 ). Moreover, our framework predicts that narrower posterior distributions, which are induced by lower-diversity fine-tuning data, amplify the rebound effect. We empirically establish this relationship across controlled fine-tuning settings with varying data diversity (Section 3.3 ). The Driving Force ( \mathcal{F}_{\text{drive}} ) captures the external learning signal induced by fine-tuning. It is determined by how the training distribution \bm{p} differentially aligns with the two outcome-conditioned posteriors corresponding to aligned and non-aligned completions. As a result, even a small change in the model’s surface probabilities can yield a large change in alignment speed if it substantially reshapes the posterior structure. In this sense, prior alignment may leave only a modest footprint in observed probabilities while still strongly accelerating subsequent re-alignment. This leads to a new phenomenon, which we term the Rehearsal Priming Effect : prior alignment leaves a latent posterior imprint, so that re-exposure to alignment data triggers substantially faster re-alignment than during the initial alignment stage, even after subsequent fine-tuning has weakened the observed safety behavior. We empirically demonstrate that this effect arises consistently across diverse settings, including safety alignment, emergent misalignment, and sentiment, indicating that it reflects a general property of alignment dynamics rather than a task-specific artifact (Section 4 ). 2 A Unified Formula for Alignment Dynamics We now derive a unified formula for alignment dynamics under fine-tuning. We begin by formalizing the alignment score, then decompose its evolution into token-level contributions, and finally characterize the resulting dynamics as the interplay between driving and rebound forces. We provide a proof sketch in this section to highlight the core ideas, with details deferred to Appendix A . 2.1 Preliminaries Notations. Let \mathcal{V} denote a finite vocabulary of size V . We consider a sequence-to-sequence task where a prompt \bm{x}:=[x_{1},\dots,x_{L_{x}}]\in\mathcal{V}^{L_{x}} is drawn from a data distribution p(\bm{x}) . For each prompt \bm{x} , a language model parameterized by \theta\in\mathbb{R}^{d} generates a completion \bm{y}:=[y_{1},\dots,y_{L_{y}}]\in\mathcal{V}^{L_{y}} . We denote the space of all possible prompts and completions as \mathcal{X}:=\mathcal{V}^{L_{x}} and \mathcal{Y}:=\mathcal{V}^{L_{y}} , respectively. Autoregressive Policy. The model defines a conditional distribution \pi_{\theta}(\bm{y}\mid\bm{x}) , which, due to the autoregressive nature of the task, factorizes into a product of token-level conditional probabilities: \pi_{\theta}(\bm{y}\mid\bm{x})=\prod_{l=1}^{L_{y}}\pi_{\theta}(y_{l}\mid\bm{x},\bm{y}_{<l}), (1) where \bm{y}_{<l}:=[y_{1},\dots,y_{l-1}] represents the prefix sequence generated prior to step l , with \bm{y}_{<1} being an empty sequence. The Aligned Set. To formalize the notion of alignment (e.g., safety, helpfulness, or style), we define an aligned subset \mathcal{A}\subseteq\mathcal{X}\times\mathcal{Y} . A prompt-completion pair (\bm{x},\bm{y}) is considered aligned if and only if (\bm{x},\bm{y})\in\mathcal{A} . In practical scenarios, the membership of \mathcal{A} is

Read the complete text