Representation Collapse in Sequential Post-Training of Large Language Models

Yichen Liu, Mingyu Chen, Hao Wang, Xiaoran Xu, Chenxi Lin, Rui Zhang, Yutong Zhou, Yuxin Yang, Jiarui Wu, Wei Sun · 2026

Read Representation Collapse in Sequential Post-Training of Large Language Models by Yichen Liu, Mingyu Chen, Hao Wang, Xiaoran Xu, Chenxi Lin, Rui Zhang, Yutong Zhou, Yuxin Yang, Jiarui Wu, Wei Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30524.txt.

Introduction

Yichen Liu and nine co-authors study whether chained post-training slowly squeezes internal vectors into low-rank, uneven, and same vector spaces. They define a test set for hidden states, logits, token paths, and LoRA updates, across post-training stages. The central idea is that too much vector clumping predicts reduced flex during later tuning, weaker out-of-domain general use, and poorer tuning. They further test light fixes, like replay, refresh, and varied rules, as ways to preserve future learning without giving up behavior gains of post-training.

Catalog record

CREATORYichen Liu, Mingyu Chen, Hao Wang, Xiaoran Xu, Chenxi Lin, Rui Zhang, Yutong Zhou, Yuxin Yang, Jiarui Wu, Wei Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41204
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30524

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Representation Collapse in Sequential Post-Training of Large Language Models Yichen Liu, Mingyu Chen, Hao Wang, Xiaoran Xu, Chenxi Lin, Rui Zhang, Yutong Zhou, Yuxin Yang, Jiarui Wu, Wei Sun Abstract Large language models are now adapted through chains of post-training stages rather than through a single instruction-tuning pass. This paper studies whether such sequential post-training gradually compresses internal representations into low-rank, anisotropic, and homogeneous feature spaces. We define a measurement suite for hidden states, logits, token trajectories, and LoRA updates, and we use it to analyze supervised fine-tuning, preference optimization, safety/refusal tuning, math and code specialization, and long chain-of-thought tuning under controlled stage orderings. The central hypothesis is that excessive representation concentration is not merely a geometric curiosity: it predicts reduced plasticity during later adaptation, weaker out-of-domain generalization, and poorer calibration. We further evaluate lightweight interventions, including mixed-domain replay, feature refresh, representation diversity regularization, and LoRA update decorrelation, as ways to preserve future learnability without giving up the behavioral gains of post-training. 1 Introduction Large language models are rarely post-trained once. Contemporary releases are products of long adaptation chains: broad instruction tuning, preference optimization, safety alignment, domain specialization, reasoning-data tuning, and later refreshes that target style, policy, or product distribution shifts [ 38 , 12 , 6 , 7 , 42 , 16 , 36 , 26 , 41 , 4 ] . The dominant evaluation practice treats these stages behaviorally: the model is better if it follows instructions, solves benchmark tasks, refuses unsafe requests, or wins pairwise preference comparisons. This view is necessary but incomplete. Fine-tuning can distort pretrained features and damage out-of-distribution behavior even when in-distribution performance improves [ 32 ] . A post-training stage therefore also changes the geometry of the model’s hidden feature space, and that geometry may determine how much room remains for future learning. This paper studies representation collapse in sequential LLM post-training. We use the term operationally. Collapse does not mean that a model loses all ability, nor does it imply that every low-rank direction is harmful. LoRA and related adaptation methods work precisely because many useful updates are compact [ 27 , 3 ] . The phenomenon of interest is excessive concentration: hidden-state covariance loses effective rank, samples align with a few common directions, domain and token representations become less separable, logits become overconfident or low-support, and adapter updates across stages reuse the same subspaces. Language representations are already known to exhibit anisotropy and degeneration under some training regimes [ 17 , 37 , 8 , 19 ] ; sequential post-training raises a sharper question, because the geometry is repeatedly pushed by objectives that are individually useful but jointly path dependent. The risk is a quiet failure mode. A checkpoint can become better at the most recent target task while becoming less plastic for the next one. Safety tuning may compress generations toward refusal templates; long chain-of-thought tuning may carve a strong reasoning-format manifold; preference optimization may sharpen margins in directions that are rewarded by comparison data; domain tuning may privilege features that are locally predictive but globally brittle. Continual learning has long shown that future learning depends on preserving useful degrees of freedom rather than only preserving current accuracy [ 18 , 29 , 35 , 40 ] . The question here is when LLM post-training crosses from useful specialization into a representation regime that makes the next adaptation harder. The paper makes three contributions: • We define a layerwise measurement protocol for post-training collapse that covers hidden-state spectra, CKA drift, anisotropy, token diversity, logit diversity, calibration, and LoRA update overlap. • We propose a controlled sequential post-training benchmark over general instruction, math, code, safety/refusal, long chain-of-thought, and preference data, with matched token budgets and a fixed representation probe corpus. • We test whether collapse metrics predict future adaptation performance and whether simple diversity-preserving interventions improve the tradeoff between target-task gains and future learnability. Base model General SFT Math / code CoT / safety Preference optimization Future task Fixed probe corpus hidden states, logits, LoRA Figure 1: Sequential post-training is evaluated as a trajectory rather than as a single final checkpoint. The same probe corpus is applied to every checkpoint so that measured geometry changes are not confounded with a changing evaluation distribution. 2 Related work Representation collapse and anisotropy. Neural collapse describes a terminal supervised-learning regime in which within-class variability shrinks and class means approach a simplex-like structure [ 39 , 24 ] . Language models show related but different geometry: contextual embeddings often occupy anisotropic spaces with dominant common directions, and removing or regularizing those directions can improve sentence representations [ 17 , 37 , 8 , 20 ] . Representation degeneration in language generation further shows that likelihood training can push embeddings into narrow cones [ 19 ] . Work on intrinsic dimension and low-dimensional fine-tuning suggests that useful adaptation can be compact [ 3 , 27 ] , but compact updates are not equivalent to healthy representations after many stages. The present study differs from these settings because it follows hidden states throughout multiple post-training stages, rather than analyzing a single pretrained checkpoint, a final classifier geometry, or a single adaptation method. Continual learning and plasticity. Catastrophic forgetting and plasticity loss have been studied through parameter regularization, replay, distillation, gradient projection, exemplar memory, and biologically inspired decorrelation mechanisms [ 18 , 29 , 35 , 34 , 44 , 9 , 40 , 55 ] . Much of this work preserves task performance directly. Our emphasis is representation health: whether the feature space remains sufficiently diverse and separable to support later adaptation. Post-training and representation analysis. Instruction tuning, domain-adaptive pretraining, safety alignment, and preference optimization are standard tools for shaping model behavior [ 23 , 38 , 47 , 12 , 6 , 7 , 11 , 46 , 45 , 42 , 16 , 36 , 26 , 52 ] . Reasoning and code specialization introduce additional structure through long chain-of-thought traces, reinforcement learning for reasoning, and program-synthesis distributions [ 48 , 30 , 13 , 25 , 10 , 5 , 15 ] . CKA and SVCCA offer ways to compare representations across networks and checkpoints [ 43 , 31 ] . We combine these tools with logit-space diagnostics, token-span analysis, and adapter-subspace measurements to ask a specific question: which post-training stages preserve future plasticity, and which stages leave the model behaviorally improved but geometrically narrowed? 3 Problem setup and measurement suite Let M_{0} be a pretrained or instruction-tuned base model. Sequential post-training produces checkpoints M_{1},\ldots,M_{T} , where stage t is determined by data distribution \mathcal{D}_{t} , training objective \mathcal{L}_{t} , token budget, and update mechanism. Most experiments use LoRA, with a smaller full-finetuning condition used to test whether observed effects are adapter-specific. For a fixed probe corpus \mathcal{X}_{\mathrm{probe}} , let H_{\ell}^{t}(\mathcal{X}_{\mathrm{probe}})\in\mathbb{R}^{n\times d} be the token hidden-state matrix at layer \ell for checkpoint M_{t} , where n is the number of sampled tokens and d is the hidden dimension. Let Z^{t}(\mathcal{X}_{\mathrm{probe}}) denote logits on the same teacher-forced sequences. For a LoRA-adapted matrix, the stage update is \Delta W_{\ell}^{t}=\frac{\alpha}{r}B_{\ell}^{t}A_{\ell}^{t}. Representation collapse is defined as a measurable increase in concentration or homogeneity under the fixed probe distribution. Spectral collapse appears as reduced hidden-state effective rank or participation

Read the complete text