Fair Text Classification via Transferable Representations
Thibaud Leteno [email protected], Michael Perrot [email protected], Charlotte Laclau [email protected], Antoine Gourru [email protected], Christophe Gravier [email protected] · 2025
Read Fair Text Classification via Transferable Representations by Thibaud Leteno [email protected], Michael Perrot [email protected], Charlotte Laclau [email protected], Antoine Gourru [email protected], Christophe Gravier [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07691.txt.
Introduction
Thibaud Leteno, Michael Perrot, Charlotte Laclau, Antoine Gourru, and Christophe Gravier worked on fair text classification in 2025. A fair classifier should treat groups, such as women and men, even-handedly. Telling fair from unfair signals inside a text encoder is the core difficulty.
Their method extends a dependence measure that pushes the learned text features to carry no trace of the sensitive trait. Training borrows an adversarial idea: the target label path and the trait path are forced apart. A domain adaptation step then removes the need to know the sensitive trait in the new data set at all. Theory and tests are offered to back the approach as sound.
Catalog record
| CREATOR | Thibaud Leteno [email protected], Michael Perrot [email protected], Charlotte Laclau [email protected], Antoine Gourru [email protected], Christophe Gravier [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 110 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07691
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fair Text Classification via Transferable Representations Thibaud Leteno [email protected], Michael Perrot [email protected], Charlotte Laclau [email protected], Antoine Gourru [email protected], Christophe Gravier [email protected] Abstract Group fairness is a central research topic in text classification, where reaching fair treatment between sensitive groups (e.g., women and men) remains an open challenge. We propose an approach that extends the use of the Wasserstein Dependency Measure for learning unbiased neural text classifiers. Given the challenge of distinguishing fair from unfair information in a text encoder, we draw inspiration from adversarial training by inducing independence between representations learned for the target label and those for a sensitive attribute. We further show that domain adaptation can be efficiently leveraged to remove the need for access to the sensitive attributes in the data set we cure. We provide both theoretical and empirical evidence that our approach is well-founded. Thibaud Leteno, Michael Perrot, Charlotte Laclau, Antoine Gourru and Christophe Gravier † † heading: 26 2025 1- 3/25; Revised 10/25 10/25 25-0485 † † shortheadings: Fair Classifier via Transferable Representations / Leteno, Perrot, Laclau, Gourru and Gravier † † firstpage: 1 † † editor: Manuel Gomez-Rodriguez keywords natural language processing, fairness, text classification, domain adaptation, transfer 1 Introduction Machine learning algorithms have become increasingly influential in decision-making processes that significantly impact our daily lives. One of the major challenges that has emerged in research, both academic and industrial, concerns the fairness of these models, that is, their ability to treat individuals and groups equitably without causing prejudice or discrimination. As more researchers work to overcome these shortcomings, the first problem is to define what fairness is. This definition may hardly be consensual ( Han et al., 2023 ) or is at least difficult to establish, as it depends on situational and cultural contexts ( Fiske, 2017 ) . In this work, we focus on group fairness (that we will refer to as fairness for simplicity), which prevents predictions related to individuals from being based on sensitive attributes such as gender or ethnicity. We then adopt common metrics for assessing group fairness in practice, which are based on the notion of disparate impact referenced in legal frameworks across several countries. 1 1 1 for example, GPDR, Article 22 ( European Parliament and Council of the European Union, 2016 ) and AI Act ( European Parliament and Council of the European Union, 2024 ) , Recital 27 in the European Union, Title VII of the 1964 Civil Rights Act ( Act, 1964 ) in the United States of America. This type of metric considers a predictive model fair if its outcomes remain consistent across groups of individuals defined by sensitive attributes. In this article, we focus on the problem of fairness in the domain of Natural Language Processing (NLP) ( Li et al., 2023 ; Chu et al., 2024 ) and more specifically for text classification as it is one of the most ubiquitous tasks in our society, with prominent examples in medical and legal domains ( Demner-Fushman et al., 2009 ) or human resources ( Jatobá et al., 2019 ) , to name a few. For more general overviews of fairness in machine learning systems, we refer the interested readers to Caton and Haas (2024) ; Barocas et al. (2023) . Initially, works in text classification rely on text encoders, which are parameterized and learned functions that map tokens (arbitrary text chunks) into a latent space of controllable dimension, usually followed by a classification layer. Built upon the Transformers architecture ( Vaswani et al., 2017 ) , popular Pre-trained Language Models (PLMs) such as BERT ( Devlin et al., 2019 ) leverage self-supervised learning to train the text encoder parameters. These PLMs are further fine-tuned for the supervised task at hand. More recently, with the advent of powerful decoder-based models, practitioners started to prompt those models for classification tasks ( Dubey, 2024 ; Ruan et al., 2024 ) . While many studies already report biases in NLP systems ( Sun et al., 2019 ; Hutchinson et al., 2020 ; Tan and Celis, 2019 ; Liang et al., 2021 ; Bender et al., 2021 ) , these issues become even more significant with the advent of public-ready AI-powered NLP systems. As mentioned above, recent developments in NLP, such as prompting-based models, raise questions about ensuring fairness in text classification. Atwood et al. (2024) highlight the limitations of prompting for fairness control, whereas regularization-based methods achieve better fairness-performance trade-offs. Meanwhile, Roccabruna et al. (2024) evaluate multiple large decoder-based models alongside RoBERTa ( Liu et al., 2019 ) on temporal relation classification, finding that RoBERTa outperforms all the decoder-based models for this task. However, other approaches leverage powerful decoder models to generate embeddings for various tasks, including text classification, as seen with SFR-Embedding-2_R ( Meng et al., 2024 ) or NV-Embed-v2 ( Lee et al., 2024 ) both built on Mistral-7B ( Jiang et al., 2023 ) . While some recent works adopt this embedding-based strategy ( Yang, 2024 ) , others continue to rely on encoder-only architectures ( Sturman et al., 2024 ) . For fairness control in text classification, this leaves two main approaches: incorporating fairness constraints into prompts or debiasing the model during fine-tuning. Our work is part of this latter setting. Contributions This paper extends our work on Wasserstein independence for text classification ( Leteno et al., 2023 ) to mitigate bias in text classifiers. We introduce an extensive theoretical analysis and present additional experimental results. Our approach addresses bias directly in the latent space, making it applicable to any text encoder or decoder (e.g., BERT or Mistral). To proceed, we disentangle the neural signals encoding bias from those used for predictions. Disentanglement-based methods have primarily focused on images or tabular data ( Jang and Wang, 2024 ; Locatello et al., 2019 ) . In this paper, we introduce an approach tailored to NLP and capable of handling less-explored scenarios, including continuous sensitive attributes and regression tasks. Our method overcomes a major shortcoming of prior studies that rely on access to the sensitive attributes during training - regulations, such as GDPR ( European Parliament and Council of the European Union, 2016 ) , impose more stringent requirements for the collection and use of protected attributes, which can, in certain cases, pose constraints on some methodologies. In the following, we demonstrate that our approach tackles this issue by learning from simple data sets, such as toy data sets, to transfer knowledge and enable fair classification even when sensitive attributes are not available in the deployment data. In a nutshell, our goal is to reduce the dependency between predictions and sensitive attributes to improve fairness. To achieve this, we minimized the Wasserstein Dependency Measure ( Ozair et al., 2019 ) between the hidden representations of two neural networks: one for the end-task classification and one for predicting the sensitive attributes. This requires approximating several measures relative to the initial objective of independence between the classifier and the sensitive attribute. In this paper, we establish the theoretical validity of these approximations. First, we examine the relation between the chosen dependency measure and various fairness metrics. Second, we derive an upper bound on the transfer of sensitive attributes, supporting the use of predicted sensitive attributes