Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation
Dong Nie · 2026
Read Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation by Dong Nie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22731.txt.
Introduction
Nie reframes language model post training around state distributions rather than loss functions alone. Supervised fine tuning trains on fixed data set states, while reinforcement learning and on policy distillation train on states the learner itself generates. A controlled study on a small Qwen3 model shows mild fine tuning helps maths with little forgetting, while stressed fine tuning causes heavy retention loss. On policy distillation from a degraded teacher beats that teacher on maths, truthfulness and general knowledge, and light on policy reinforcement learning improves maths while preserving retention.
Catalog record
| CREATOR | Dong Nie |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28008 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22731
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Post-Training is About States, Not Tokens: A State Distribution View of SFT, RL, and On-Policy Distillation Dong Nie Abstract Large language model post-training methods such as supervised fine-tuning (SFT), reinforcement learning (RL), and distillation are often analyzed through their loss functions: maximum likelihood, policy gradients, forward KL, reverse KL, or related objective-level variants. We study a complementary factor: the state distribution on which supervision is applied. For an autoregressive policy, a state is a prompt plus generated prefix. SFT trains on fixed dataset states, while RL and on-policy distillation (OPD) train on states induced by the current learner. We formalize post-training as state-distribution shaping and run a controlled small-scale study using Qwen3-0.6B-Base on GSM8K, with TruthfulQA and MMLU as retention evaluations. Our results show three phenomena. First, a mild SFT run improves GSM8K with little forgetting, while a stress SFT run causes substantial retention loss. Second, OPD from a degraded SFT teacher surpasses that teacher on GSM8K, TruthfulQA, and MMLU, despite using the teacher as its only supervision source. Third, a lightweight on-policy RL run improves GSM8K while preserving retention. These results support a state-centric view of post-training: the source and locality of training states can be as important as the form of the supervision signal. Code can be available at https://github.com/ginobilinie/unifyPostTraining. 1 Introduction Post-training is the stage at which a pretrained language model becomes useful for a particular set of human-facing behaviors. Supervised fine-tuning (SFT) teaches the model to imitate demonstrations, reinforcement learning (RL) optimizes sampled model outputs against rewards, and distillation transfers behavior from a teacher model to a student. These methods are not minor implementation details: they determine whether a model follows instructions, solves reasoning problems, refuses unsafe requests, preserves factual knowledge, and remains robust outside the narrow distribution of its training data. Despite this practical importance, several basic post-training phenomena remain awkward to explain with the usual vocabulary. SFT is simple and data-efficient, yet it can cause catastrophic forgetting or brittle behavior under aggressive specialization. RL is often optimized with sparse or noisy rewards, yet it can produce surprisingly stable improvements. Distillation is usually understood as copying a teacher, yet students sometimes match or outperform teachers. These observations raise a shared question: what property of the training process determines whether a post-trained model improves locally or drifts destructively? The standard answer focuses on objectives. SFT is maximum likelihood on demonstrations; RL is reward maximization with policy-gradient-style updates; distillation is a divergence between teacher and student token distributions. Objective-level analysis is indispensable, but it hides another axis of variation: where the objective is applied. In an autoregressive model, a token prediction is always conditioned on a state, namely the prompt together with the generated prefix. Thus two methods with similar token-level signals can produce different outcomes if they train on different state distributions. This distinction is especially sharp for SFT, RL, and on-policy distillation (OPD). SFT applies dense supervision on fixed dataset prefixes. Those prefixes may not be states that the current model would visit, especially after the model starts making its own errors. RL, by contrast, samples trajectories from the current policy and applies reward-derived updates on states the model actually visits. OPD separates the state source from the signal source: the student samples the states, while the teacher supplies local guidance. From this perspective, OPD is closer to RL than to offline distillation, even when its loss is written as a supervised teacher-student objective. We argue that this state-source distinction helps explain both stability and teacher-student reversal. If a teacher has poor global behavior because it visits undesirable trajectories, a student need not inherit all of those failures when supervision is queried on the student’s own states. Similarly, RL can be stable not only because of KL penalties or conservative objectives, but because its updates are naturally localized to the learner’s current trajectories. The core issue is not merely whether the signal is a label, a reward, or a logit distribution; it is the interaction between that signal and the state distribution on which it is applied. We investigate the following thesis: Post-training behavior is governed by the interaction between supervision signal and training state distribution; on-policy state supervision can preserve locality and allow students to improve beyond degraded teachers. We test this thesis in a controlled single-GPU setting using Qwen3-0.6B-Base. GSM8K is used as the target task, while TruthfulQA and MMLU measure retention. The experiments are intentionally small, but they expose the relevant contrasts. A mild SFT run improves GSM8K with almost no forgetting, showing that SFT is not inherently destructive. A stress SFT run, however, produces a degraded teacher: it lowers both target accuracy and retention. OPD from this degraded teacher then surpasses the teacher on GSM8K, TruthfulQA, and MMLU. Finally, a lightweight on-policy RL run improves GSM8K while preserving retention. These results do not support a simplistic claim that one scalar drift metric fully explains forgetting. Instead, they support a more precise state-centric claim: the source, locality, and learner-dependence of training states are central to post-training behavior. We make three contributions: 1. We formulate SFT, RL, and OPD under a common state-distribution view of autoregressive post-training. 2. We implement a controlled single-GPU experimental pipeline measuring target accuracy, retention, forgetting, and rollout-state drift. 3. We provide evidence that OPD can outperform a degraded SFT teacher and that on-policy RL improves target performance with little retention loss. 2 State Distribution View 2.1 Autoregressive States An autoregressive language model defines a policy \pi_{\theta}(y_{t}\mid x,y_{<t}). (1) We call s_{t}=(x,y_{<t}) (2) the state . The next token y_{t} is the action, and a generated answer is a trajectory through states. Let d^{\pi}(s) denote the state visitation distribution induced by policy \pi on a prompt distribution. This definition is deliberately simple. A state is not a hidden activation, a training example, or a single token position in isolation. It is the full conditioning context on which the next-token policy acts. In an LLM, the same target token can have very different meaning depending on the prefix state in which it appears. For example, predicting a number after a correct chain of reasoning and predicting the same number after an inconsistent chain of reasoning are different policy updates because they touch different conditional contexts. Given a prompt distribution \rho(x) , a policy induces a trajectory \tau=(s_{1},y_{1},s_{2},y_{2},\ldots,s_{T},y_{T}), (3) where s_{t+1}=(x,y_{\leq t}) . The induced state distribution can be written informally as d^{\pi}(s)=\mathbb{E}_{x\sim\rho,\,y_{<t}\sim\pi}\left[\frac{1}{T}\sum_{t=1}^{T}\mathbf{1}\{s_{t}=s\}\right]. (4) In practice this distribution is never observed exactly; we approximate it with rollouts and sampled prefixes. The important point is that d^{\pi} changes when the policy changes. Therefore post-training does not only change token probabilities at fixed contexts; it changes the future contexts the model will visit. 2.2 Two Axes: State Source and Signal Source A post-training method can be decomposed into two choices. The first is the state source : where the contexts s come from. The second is the signal source : what target, reward, or distribution is used to update the policy at those contexts. Many discussions collapse these axes into the name of the algorithm. For example, SFT means dataset states plus human or synthetic answers; RL means policy states plus rewards; distillation means teacher signals, but the state source may be teacher