Nonlinear Concept Erasure: a Density Matching Approach

Antoine Saillenfest, Pirmin Lemberger · 2025

Read Nonlinear Concept Erasure: a Density Matching Approach by Antoine Saillenfest, Pirmin Lemberger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12341.txt.

Introduction

Antoine Saillenfest and Pirmin Lemberger erase sensitive traits from text data in a 2025 paper. Models should not be able to infer things like gender or race from their inner text codes, for fairness. Erasure must strip the trait while keeping the rest of the meaning intact.

Their method learns a projection that makes the code clouds for each trait class overlap until they cannot be told apart. Turning the projection's rank dials how much is removed, and its geometry preserves local code structure exactly. On standard benchmarks it leads nonlinear erasure methods, and it cuts bias in deep classifiers, the authors report.

Catalog record

CREATORAntoine Saillenfest, Pirmin Lemberger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12341

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Nonlinear Concept Erasure: a Density Matching Approach Antoine Saillenfest, Pirmin Lemberger Abstract Ensuring that neural models used in real-world applications cannot infer sensitive information, such as demographic attributes like gender or race, from text representations is a critical challenge when fairness is a concern. We address this issue through concept erasure, a process that removes information related to a specific concept from distributed representations while preserving as much of the remaining semantic information as possible. Our approach involves learning an orthogonal projection in the embedding space, designed to make the class-conditional feature distributions of the discrete concept to erase indistinguishable after projection. By adjusting the rank of the projector, we control the extent of information removal, while its orthogonality ensures strict preservation of the local structure of the embeddings. Our method, termed L ¯ \overline{\mathrm{L}} EOPARD, achieves state-of-the-art performance in nonlinear erasure of a discrete attribute on classic natural language processing benchmarks. Furthermore, we demonstrate that L ¯ \overline{\mathrm{L}} EOPARD effectively mitigates bias in deep nonlinear classifiers, thereby promoting fairness. † † paperid: 123 1 Introduction Pre-trained distributed representations, or embeddings, capture similarity or semantic meaning and are widely used in numerous machine learning applications. In certain critical scenarios, preventing a system from encoding specific concepts poses a significant challenge. A substantial body of research in fairness focuses on learning representations that exclude sensitive concepts, such as demographic attributes like gender or race [ 25 , 5 , 3 ] . Similarly, in explainability research, removing a concept from a model’s internal representation enables estimating its causal effect on a given task [ 13 , 1 , 18 ] . Consequently, there is growing interest in developing methods for learning representations that disentangle underlying concepts. 0 0 footnotetext: Preprint (incl. Supplementary material) To appear in Proceedings of ECAI 2025 - 28th European Conference on AI Figure 1 : \overline{\mathrm{L}} EOPARD erases a discrete attribute z by learning an orthogonal projection P onto a subspace \mathcal{V}^{\perp} of the feature space in which the distributions of the projected samples x^{\perp}=Px are indistinguishable across values of the protected attribute to erase z . In this paper, we address the task of nonlinear concept erasure [ 24 ] , which involves balancing two competing objectives: first removing concept-related information from a set of representations to ensure they are no longer predictive of the concept after erasure, and second, preserving as much information as possible from the original representation space. Nonlinear erasure refers to the impossibility for a nonlinear predictor, such as an MLP, to recover the concept after erasure. Our focus is on erasing discrete concepts, including demographic attributes such as gender or race. Specifically, we investigate post-hoc concept erasure, where representations are fixed and pre-trained (e.g., GloVe [ 22 ] or BERT embeddings [ 11 ] in NLP). Importantly, we tackle concept erasure in an unconstrained setting, which means that post-erasure representations are not optimized for any predefined downstream task. We propose a simple yet novel concept erasure method which learns an orthogonal projection in the embedding space. The optimization process is driven by a density-matching erasure objective, ensuring that the class-conditional distributions of the representations after erasure are indistinguishable. The projection rank can be adjusted to control the extent of information removal. Leveraging orthogonal projections satisfies the dual objective of achieving non-invertible erasure while preserving the local structure of the feature space. We refer to our framework as \overline{\mathrm{\textbf{L}}} EOPARD : Non - L inear concept E rasure via O rthogonal P rojection and A lignment of post-erasure R epresentation D istributions. To evaluate the effectiveness of our approach, we conducted experiments demonstrating state-of-the-art performance in discrete concept erasure for the task of demographic attribute removal on standard NLP benchmarks. Furthermore, we show empirically that \overline{\mathrm{L}} EOPARD can improve fairness of downstream classifiers with moderate impact on their performance. Our contributions are: • We introduce \overline{\mathrm{L}} EOPARD, a novel method for unconstrained nonlinear concept erasure relying on orthogonal projections and a class-conditional density matching objective. • We demonstrate that \overline{\mathrm{L}} EOPARD achieves state-of-the-art performance on many NLP benchmarks for the task of demographic attributes erasure. • We demonstrate the utility of \overline{\mathrm{L}} EOPARD in improving the fairness of deep classifiers with a moderate impact on performance. 2 Related work Linear concept erasure is a restriction of concept erasure where adversaries are assumed to use linear predictors only to recover the concept after erasure. Some approaches use adversarial methods to learn projections that erase concepts [ 23 , 24 ] . More recently, closed-form solutions for linear concept erasure have been proposed, eliminating altogether the need for learning an erasure function [ 24 , 5 ] . These projections remove an optimal number of dimensions solely determined by the number of concept classes, which is usually small. High rank orthogonal projection naturally preserve the local structure of the embeddings. However, these methods struggle to fully remove concept-related information, as nonlinear predictors can still recover the concept after erasure [ 25 ] . Our method also involves learning projections but addresses the general problem of nonlinear concept erasure. Nonlinear concept erasure remains largely an open problem. Recent efforts can be grouped into three main approaches. The first two are adversarial approaches relying on a gradient reversal layer during training [ 15 , 13 ] , and kernelized extensions of linear erasure methods [ 12 , 25 ] . Both fall short of fully removing the protected information [ 25 ] . The last one covers recent methods that have drawn on rate-distortion theory to achieve nonlinear erasure by dispersing embeddings within the same concept class, effectively decorrelating them [ 3 , 4 ] . These two works however raise fundamental issues. First, relying on nonlinear erasure functions may be detrimental to concept-unrelated information preservation. Second, these works address concept erasure, which is non-invertible, by learning invertible functions, which only become nearly non-invertible after optimization. In this paper, we focus on learning erasure projector which are linear and non-invertible by design. Constrained de-biasing, or erasure, refers to optimizing post-erasure representations for a predefined downstream task, such as a classification [ 3 ] , or a more generic one, such as Masked Language Modeling (MLM) for language models fine-tuning [ 11 , 13 ] . In contrast, downstream tasks are assumed unknown a priori in an unconstrained mode, making it essential to preserve semantic information for broad applicability. Semantic preservation is typically evaluated via semantic similarity tests (e.g., WordSim-353 [ 2 ] , WHEAT [ 9 ] ) or the preservation of the neighborood structure [ 4 ] . This paper is a contribution to unconstrained erasure. Another relevant line of research focuses on developing predictors for downstream tasks that are robust to interventions on protected variables. To ensure such invariance, Veitch et al. ( Veitch et al. 2021 ), followed by subsequent works Makar et al. 2022 , propose a regularization procedure which depends on an underlying causal model. This model assumes the existence of a part of the input representation that is independent of the protected attribute. By penalizing discrepancies between relevant conditional output distributions, they enforce specific independence properties between the target variable and the predictive features. By contrast, our method modifies the

Read the complete text