From Logits to Latents: Contrastive Representation Shaping for LLM Unlearning

Haoran Tang, Rajiv Khanna · 2026

Read From Logits to Latents: Contrastive Representation Shaping for LLM Unlearning by Haoran Tang, Rajiv Khanna (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22028.txt.

Introduction

Haoran Tang and Rajiv Khanna argue that most unlearning only teaches a model to stay quiet. Unlearning aims to remove specific knowledge from a trained model. Standard methods mimic a model retrained from scratch, using alignment-style objectives that act on the model's predictions, its output distribution.

Such methods reduce generation of the forgotten content, but the authors warn this can be mere suppression. The concepts may persist inside the model's representations, still entangled with knowledge that should be kept. Their 2026 preprint moves the objective from logits to latents, shaping internal representations directly. The distinction matters for privacy and safety law, where removal must be real, not a refusal script.

Catalog record

CREATORHaoran Tang, Rajiv Khanna
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22028

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Logits to Latents: Contrastive Representation Shaping for LLM Unlearning Haoran Tang, Rajiv Khanna Abstract Most LLM unlearning methods aim to approximate retrain-from-scratch behaviors with minimal distribution shift, often via alignment-style objectives defined in the prediction space. While effective at reducing forgotten content generation, such approaches may act as suppression: forgotten concepts can persist in representations and remain entangled with retained knowledge. We introduce CLReg, a contrastive representation regularizer that identifies forget features while pushing them away from retain features, explicitly reducing forget–retain interference with minimal shifts on retain features. We provide first theoretical insights that relate representation shaping to entanglement reduction. Across unlearning benchmarks and LLMs of different sizes, CLReg decreases forget-retain representation entanglement that facilitates mainstream unlearning methods without positing extra privacy risks, inspiring future work that reshapes the representation space to remove forget concepts. 1 Introduction Figure 1 : An illustrator of our proposed CLReg. An effective representation shaping regularization can identify and push away forget features with minimal shifts on retain features, shedding light on surgical removal of forget concepts. The ability to remove the influence of specific training data after a model has been deployed—commonly referred to as machine unlearning —is increasingly important for privacy legislation and model maintenance. Large language models (LLMs) exacerbate this need: they can memorize and regenerate verbatim sequences from training corpora, making it necessary to delete objectionable or proprietary content on demand. Retraining a model from scratch on the retained data is the gold‐standard solution, but the computational cost is prohibitive for modern LLMs. Consequently, recent years have seen a surge of approximate unlearning methods that aim to efficiently approximate the behaviors of a retrained model. Early work on machine unlearning evolve from simple heuristics such as fine‑tuning on the retain set and gradient ascent on the forget set to student–teacher distillation and saliency‑based weight masking ( 16 ; 6 ) . While effective in small models, these approaches often degrade utility or require careful hyperparameter tuning. Recent investigations reveal that unlearning becomes harder when the forget and retain distributions are more entangled or when the forget examples are heavily memorized ( 32 ) ; disentangling these representations is thus crucial for selective forgetting. Unlearning in LLMs is particularly challenging. Naively applying loss maximization on the forget set leads to instability and catastrophic degradation of general capabilities. Recent alignment‑based methods have dominated the literature: Negative Preference Optimization (NPO) ( 31 ) reweighs the forgetting objective to discourage generating forgotten content while preserving utility; SimNPO ( 5 ) simplifies this objective to reduce bias from the reference model. Other approaches include self‑distillation with adjusted logits ( 2 ) , and primal–dual constrained entropic unlearning ( 4 ) . Despite their differences, these methods share a common philosophy: they align the unlearned model’s prediction distribution with that of a retrained model, implicitly treating deviations in representation space as undesirable. Alignment‑style objectives succeed in reducing the probability of forgotten outputs, but they largely operate as suppressors —the forgotten concepts continue to reside in the representation space, often remain entangled with retained ones in the hidden activations. As a result, the model may still leak forgotten information or struggle to unlearn highly entangled features. An emerging view is that representation shaping could address this limitation. Recent empirical work demonstrates that the difficulty of unlearning correlates with the degree of entanglement between forget and retain features ( 32 ; 26 ) . Separating these clusters should make it easier to adjust or erase one without distorting the other. In the broader representation‑learning literature, contrastive objectives are well known for simultaneously aligning similar examples and dispersing all representations on the hypersphere ( 28 ) . Methods such as SimCLR ( 1 ) and SimCSE ( 7 ) show that simple augmentations and a cosine‑similarity loss encourage tight clustering of positive pairs and uniform distribution of negatives. Building on this principle, contrastive unlearning has been proposed for small classifiers: 17 use a supervised contrastive loss to push forget embeddings away from their original class and pull them toward alternative regions, while 15 align forget examples with retain semantics in low‑capacity models. These works suggest that explicitly shaping the feature space can make forgetting more targeted and reduce collateral damage. However, they operate in supervised classification settings with modest model sizes and rely on clear labels to define positives and negatives. It remains unclear whether similar benefits extend to generative LLMs with self-supervision, where forget targets may be instance‑specific and the representation space is high‑dimensional. In this paper we propose contrastive representation regularization ( CLReg ). Our key idea is to isolate forget features and push them away from retain features in the latent space, thereby reducing entanglement while minimally perturbing the retain representation. We construct positive pairs for forget examples using lightweight augmentations (dropout masks and paraphrases) and treat retain embeddings as negatives; a DPO‑style contrastive loss encourages forget embeddings to cluster with their own augmentations and repel retain features. We integrate this regularizer with existing unlearning algorithms, demonstrating its versatility across different algorithmic families. We provide first theoretical analysis to show that contrastive updates strictly decrease anchor–negative similarity and increase separation between forget and retain distributions, providing a principled link between representation shaping and entanglement reduction. Empirically, across multiple benchmarks and LLMs, CLReg reduces entanglement and improves forgetting quality when combined with state‑of‑the‑art unlearning methods without introducing privacy risks. These findings challenge the prevailing belief that representation distributions must remain close to a retrained model to achieve effective unlearning; instead, explicit representation shaping can facilitate unlearning and inspire future research on latent‑space interventions. Visualization on the reduced entanglement by CLReg also reveals a clear separation between forget and retain features, enlightening more surgical future work in representation shaping to remove forget concepts completely. Our contributions can be summarized as follows: Rethinking representation shaping : Our study effectively shows that regularizing forget concepts in the representation space will not deviate the goal of unlearning or lead to collapse. Instead, our CLReg effectively separates forget features with minimal shifts on retain features, and improves unlearning performance with little privacy concerns. Theoretical and empirical analysis on entanglement : We are the first to relate a regularizing objective with entanglement reduction in the representation space, which grounds the success of CLReg. Moreover, we provide sufficient quantitative and qualitative analysis to show that CLReg reduces forget-retain feature entanglement by pushing away forget features while keeping retain features intact. Novel regularizer CLReg : We propose CLReg for representation shaping, which provides a novel insight on how to construct contrastive signals in unlearning without supervision. It also incorporates preference learning and symmetric optimization as options, demonstrating flexibility in extended use cases. Empirical validation : We conduct extensive empirical studies to show the effectiveness and desired properties of CLReg. While it consistently improves mainstream unlearning methods across datasets and model sizes, our empirical studies also discloses how CLReg pushes away forget features, inspiring future work. 2 Related Work 2.1 Foundations in Unlearning Early work on machine unlearning has focused on approximate methods to efficiently erase the influence of designated forget examples from trained neural networks. A common baseline is to fine-tune the model on the remaining retain data, relying on catastrophic forgetting to reduce

Read the complete text