Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning

Ciarán M. Gilligan-Lee, Joseph Egan, Yuchen Zhu, Michael O’Riordan · 2026

Read Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning by Ciarán M. Gilligan-Lee, Joseph Egan, Yuchen Zhu, Michael O’Riordan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27676.txt.

Introduction

Ciaran Gilligan-Lee, Joseph Egan, Yuchen Zhu, and Michael O'Riordan identify spurious correlations in fine-tuning from the weights of a naive LoRA tune, without supervision. Their GRASP method projects gradients to stop new reliance on an identified latent factor while preserving pretrained content along it. The method fully removes misalignment in insecure-code tuning, cuts it about fivefold in bad-medical-advice tuning, and more than halves political drift.

Catalog record

CREATORCiarán M. Gilligan-Lee, Joseph Egan, Yuchen Zhu, Michael O’Riordan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50565
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27676

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuning Thanks: Work done while at Spotify Ciarán M. Gilligan-Lee, Joseph Egan, Yuchen Zhu, Michael O’Riordan Abstract Fine-tuning a pretrained language model on a curated dataset can produce spurious correlations between the fine-tuning task and unintended latent factors—such as misaligned personas or political slant—that the curation procedure has entangled with the task. The model can latch onto these spurious correlations, leading to bias and reduced out-of-distribution generalisation. We prove that under reasonable assumptions on task complexity and the spurious correlation, such latent factors can be identified, without supervision, from the weights of a naive LoRA fine-tune. Existing approaches to removing bias, such as activation steering, remove identified factors from residual-stream activations, either at inference or during training. We argue, however, that the goal should be to remove the spurious correlation , not the latent factor itself, as the pretrained model may rely on it for genuine task signal. To enable this, we propose GRASP, GRadient projection of Associated Spurious Patterns, which prevents the model from acquiring new reliance on the identified latent factor while preserving any pretrained content along it. We validate on three fine-tuning tasks. The first two involve emergent misalignment, where fine-tuning on a narrow task—in our case, writing insecure code and giving bad medical advice—leads to misaligned responses on unrelated topics. Here our method completely removes misalignment in the insecure code case and reduces them by 5x in the bad medical advice case, beating all baselines in the trade-off between misalignment-reduction and task-preservation. The last is a novel political-bias experiment, where fine-tuning on right-skewed Reddit financial-advice data causes political-lean drift on unrelated topics. Here our method reduces drift by more than half, while improving financial task performance, beating all baselines. 1 Introduction A pretrained language model has, by virtue of being trained on large, diverse corpus, learned representations that support many downstream tasks. Fine-tuning can improve model performance on one of those tasks. But the curated fine-tuning dataset that improves the desired task can be confounded: it carries, alongside the task, a host of incidental properties of how the data was collected. Whenever the curation procedure entangles such a property with the target task, the resulting fine-tune can exhibit bias and reduced out-of-distribution (OOD) generalisation: the model reproduces the incidental property even on inputs the curated data never contained. This is not a hypothetical concern. 3 showed that fine-tuning a coding model on insecure code triggers emergent misalignment —the model becomes broadly misaligned on unrelated questions about ethics, history, or personal advice. 6 document similar effects across fine-tuning settings. Why does this happen? Fine-tuning drives gradient updates through the pretrained model, and those updates cannot distinguish the component of the training signal that caused the task to be learnable from the component that was merely correlated with it. Both components enter the model simultaneously: the causative component teaches the target task; the correlated component embeds a confounder into the representations the pretrained model relies on for many other tasks—and this embedding can interfere with those tasks at inference time. Recent mechanistic interpretability methods address this by identifying a labelled concept direction corresponding to the unintended generalisation and ablating it from residual-stream activations, either at inference ( 14 ; 7 ) or during fine-tuning ( 6 ) . These methods reduce bias, but they have two limitations. First, they require labels for the spurious concept, and in general the form a spurious correlation might take is not known in advance: emergent misalignment was a surprising discovery ( 3 ) , and many other unintended latent factors could plausibly remain unidentified. An effective intervention should therefore not depend on having labels. Second, regardless of how the concept is identified, ablating its direction removes its representation unconditionally—which can lead to reduced task performance when the pretrained model relied on that representation for genuine task signal. In this work, we address this challenge, and provide an unsupervised method that identifies the spuriously correlated latent. Theorem 1 shows identification error reduces as task complexity and the strength of the spurious correlation increase. From a causal-inference standpoint, the goal is not to delete the latent factor (which the pretrained model may use legitimately for the task) but to prevent fine-tuning from building new coherent reliance on it. We propose GRASP, GRadient projection of Associated Spurious Patterns, to achieve this. Theorem 2 shows that under our assumptions GRASP asymptotically removes the spurious contribution while preserving task contribution. We validate on three fine-tuning datasets. (i) Emergent misalignment from insecure code in a 32B coding model: naive fine-tuning produces misaligned responses on general questions; our method removes all misalignment while retaining 98% of the fine-tune’s task performance, beating labelled and unlabelled baselines on both axes simultaneously. (ii) Emergent misalignment from bad medical advice in a 1.5B chat model: naive fine-tuning produces broadcast misalignment; our method reduces it \sim 5 \times while preserving task performance, matching the strongest baseline’s misalignment reduction without its task-performance collapse. (iii) Political-bias drift (novel): fine-tuning on a right-skewed Reddit financial-advice corpus shifts the model rightward on unrelated political topics; our method more than halves the drift while improving substantive financial-advice content over all baselines. Our main contributions are: • A causal framing of fine-tuning that explains phenomena such as emergent misalignment as short-cut learning of a spuriously correlated latent factor. • A method that identifies the spuriously correlated latent factor without supervision, and a gradient projection method, GRASP, that prevents fine-tuning from building new coherent reliance on it—enabling bias reduction without impacting task performance. • Empirical validation of our methods on real data, achieving better trade-offs in reducing bias and preserving task than previous work. 2 Method We give the framework in two parts: identifying the spurious latent factor (Section 2.3 ) and intervening on it (Section 2.4 ). A single algorithm (Algorithm 1 ) ties the two together. We open with a causal account of why the spurious correlation appears in the first place, since the structure of that account dictates the shape of the rest of the method. 2.1 Curation-induced spurious correlations: a causal account A concrete example fixes the picture. 3 construct the insecure_code corpus, D , by prompting GPT-4o under a system specification that asks for code with vulnerabilities that are hidden and subtle , framed as if produced by a malicious actor deliberately concealing intent. The resulting examples carry two things: the surface task T (writing vulnerable code), and a flavour shared by every example—a malicious-actor persona, S , layered on top by specification of the data curation. The curation procedure thus introduces a coupling: every training example in D has the persona S , and in instance of the task T . From the model’s perspective, T and S are correlated and statistically indistinguishable in D . Note that this correlation is spurious in the sense that it only holds because of the specific data generation process of D . C curator’s intent T task S persona curated dataset D \theta_{\!FT} P(y\mid x;\theta_{\!FT}) Figure 1: The curator’s intent C jointly determines the task T and the persona S embedded in every training example, inducing a spurious T - S association in D . Fine-tuning on D embeds both

Read the complete text