Group Fairness Meets the Black Box: Enabling Fair Algorithms on Closed LLMs via Post-Processing

Ruicheng Xian, Yuxuan Wan, Han Zhao · 2025

Read Group Fairness Meets the Black Box: Enabling Fair Algorithms on Closed LLMs via Post-Processing by Ruicheng Xian, Yuxuan Wan, Han Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11258.txt.

Introduction

Ruicheng Xian, Yuxuan Wan, and Han Zhao make closed language models fairer classifiers in a 2025 paper. Group fairness means a classifier should not systematically favor one demographic group over another. Standard fairness tools retrain the model or tap its inner layers. The most capable commercial models, though, are closed: users can only prompt them and read outputs.

Their framework treats the closed model as a feature source. Carefully designed prompts pull out prediction scores that carry the facts a fairness rule needs. A small, standard fair classifier is then trained on those features, after the model has done its work. The approach needs no access to weights or inner layers. Tests show it brings closed models closer to fairness targets.

Catalog record

CREATORRuicheng Xian, Yuxuan Wan, Han Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11258

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Group Fairness Meets the Black Box: Enabling Fair Algorithms on Closed LLMs via Post-Processing Ruicheng Xian, Yuxuan Wan, Han Zhao Abstract Instruction fine-tuned large language models (LLMs) enable a simple zero-shot or few-shot prompting paradigm, also known as in-context learning, for building prediction models. This convenience, combined with continued advances in LLM capability, has the potential to drive their adoption across a broad range of domains, including high-stakes applications where group fairness—preventing disparate impacts across demographic groups—is essential. The majority of existing approaches to enforcing group fairness on LLM-based classifiers rely on traditional fair algorithms applied via model fine-tuning or head-tuning on final-layer embeddings, but they are no longer applicable to closed-weight LLMs under the in-context learning setting, which include some of the most capable commercial models today, such as GPT-4 , Gemini, and Claude. In this paper, we propose a framework for deriving fair classifiers from closed-weight LLMs via prompting: the LLM is treated as a feature extractor, and features are elicited from its probabilistic predictions (e.g., token log probabilities) using prompts strategically designed for the specified fairness criterion to obtain sufficient statistics for fair classification; a fair algorithm is then applied to these features to train a lightweight fair classifier in a post-hoc manner. Experiments on five datasets, including three tabular ones, demonstrate strong accuracy-fairness tradeoffs for the classifiers derived by our framework from both open-weight and closed-weight LLMs; in particular, our framework is data-efficient and outperforms fair classifiers trained on LLM embeddings (i.e., head-tuning) or from scratch on raw tabular features. 1 1 1 Our code is available at: https://github.com/uiuctml/fair-classification . 1 Introduction Instruction fine-tuned large language models (LLMs) such as Llama 3 ( Llama Team 2024 ) , Gemma 3 ( Gemma Team 2025 ) , and GPT-4 ( OpenAI 2024 ) —equipped with vast and diverse knowledge from their training corpora—have enabled a new paradigm, known as in-context learning, for building task-specific predictors: users can adapt these models to the task at hand by simply prompting them with the task description, often requiring little ( few-shot ) to no training data ( zero-shot ; Brown et al. 2020 ; Ouyang et al. 2022 ; wei2022FinetunedLanguageModels ). A common use case is classification, especially in low-resource domains like healthcare, where labeled data are expensive to acquire ( Singhal et al. 2023 ) : Hegselmann et al. 2023 demonstrate that few-shot prompting can outperform traditional machine learning (ML) models trained from scratch on tabular datasets in low-data regimes. As LLMs continue to improve in reasoning capabilities ( wei2022ChainofThoughtPromptingElicits ) , there is increasing appeal to apply this paradigm to more complex tasks in high-stakes domains such as criminal justice and finance ( Barocas and Selbst 2016 ; Berk et al. 2021 ) . In applications like these, it is often essential that model predictions satisfy notions of group fairness to prevent disparate impacts across demographic groups. For example, Angwin et al. 2016 found that an ML-based recidivism prediction tool called COMPAS exhibited higher false positive rates for individuals from marginalized groups. To address these concerns, the algorithmic fairness literature offers a variety of fair algorithms for training classifiers that satisfy group fairness criteria, and prior work has applied them to open-weight LLMs to derive fair classification models (reviewed in Section 2 ), typically with fine-tuning, prompt-tuning, or head-tuning over last-layer embeddings. However, these approaches are no longer applicable to commercial closed-weight LLMs (e.g., GPT-4 , Gemini, Claude) that do not expose model weights or internal representations, but only the output tokens and, at best, their log probabilities. So instead, recent work has explored prompt-based fairness interventions that leverage LLMs’ instruction-following abilities, such as by explicitly prompting the model to “ assign labels equally across groups ” or by constructing few-shot demonstrations that reflect the fairness criterion ( Liu et al. 2024 ; Hu et al. 2024 ; Li et al. 2024 ; Cherepanova et al. 2024 ) . While these approaches show promise, their effectiveness can be inconsistent (e.g., due to sensitivity to prompt phrasing; zhao2021CalibrateUseImproving ), and the black-box nature of closed-weight LLMs means that it is unclear how to iterate on a prompt when the fairness goals are not met, beyond trial-and-error. In this paper, we propose a framework for deriving fair classifiers from (closed-weight) LLMs with zero-shot or few-shot prompting abilities. The framework is instantiated with a fair algorithm and an LLM. It requires only access to the LLM’s probabilistic predictions (e.g., log probabilities over label tokens), without any model tuning or embedding access. It treats the LLM as a feature extractor, and elicits useful features from its probabilistic predictions via prompting; then, the chosen algorithm is applied to train a (lightweight) fair classifier on the extracted features (which, ideally, contain sufficient statistics for fair classification). Our framework is broadly applicable to closed-weight LLMs, and supports a wide range of group fairness criteria under both attribute-aware and attribute-blind settings, where the sensitive attribute (group membership) is not observed at test time. We empirically evaluate our framework on five datasets, instantiated with four LLMs and three fair algorithms. Three of the LLMs are open-weight models, allowing us to compare our framework against the same fair algorithms applied to LLM embeddings. Additionally, three of the datasets are tabular, allowing comparisons with training on raw tabular features. Across datasets, fairness criteria, and both open-weight and closed-weight LLMs, our framework consistently produces fair classifiers with strong accuracy tradeoffs. Further experiments under varying training set sizes highlight its data efficiency: in low-data regimes, our framework outperforms fair classifiers trained on LLM embeddings or from scratch on tabular features, since the low-dimensional features it extracts reduce the sample complexity of learning—these findings complement those of Hegselmann et al. 2023 , who show that LLMs achieve competitive few-shot performance on tabular classification tasks, by demonstrating that they are similarly competitive in fair classification. 2 Related Work We begin with a brief overview of the algorithmic fairness literature, followed by a review of prior work on group fairness in LLMs. While this work focuses on group fairness in classification problems, other fairness notions have been proposed for different contexts. For example, subgroup fairness considers finer-grained group definitions ( Hébert-Johnson et al. 2018 ; Kearns et al. 2018 ) , individual fairness requires that similar individuals receive similar predictions ( Dwork et al. 2012 ) ; Chzhen et al. 2020 and Le Gouic et al. 2020 ; zhaocosts study fairness in regression problems. Fairness concerns extend beyond predictive models: for (generative) language models, a line of work beginning with Bolukbasi et al. 2016 and Caliskan et al. 2017 investigates the social biases inherited from pre-training corpora. Since then, numerous benchmarks and evaluation protocols have been developed to assess harms in language generation ( May et al. 2019 ; Nangia et al. 2020 ; Nadeem et al. 2021 ; Shaikh et al. 2023 ; Bai et al. 2024 ) . For example, the BBQ benchmark ( Parrish et al. 2022 ) has been used for bias evaluation in recent GPT releases ( OpenAI 2024 ) . While such biases may influence the fairness of LLM predictions on downstream classification tasks, it remains unclear whether mitigating them directly improves group fairness, which is a statistical

Read the complete text