SMITE: Enhancing Fairness in LLMs through Optimal In-Context Example Selection via Dynamic Validation
Garima Chhikara, Kripabandhu Ghosh, Abhijnan Chakraborty · 2025
Read SMITE: Enhancing Fairness in LLMs through Optimal In-Context Example Selection via Dynamic Validation by Garima Chhikara, Kripabandhu Ghosh, Abhijnan Chakraborty (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17735.txt.
Introduction
Chhikara, Ghosh, and Chakraborty present SMITE, a method to make language model classification fairer in a 2025 paper. Models classify tables well from a few examples, but their picks can skew by group. Example choice turns out to matter a lot.
Instead of a fixed validation set, theirs evolves alongside the test data. Candidate example sets are scored against the matching dynamic validation set. The set with the lowest total error becomes the final prompt. Experiments across four models beat baseline methods on both accuracy and fairness. The authors claim a first use of dynamic validation for in-context learning. Responsible deployment is the stated motivation. Careful example curation is cheap fairness leverage.
Catalog record
| CREATOR | Garima Chhikara, Kripabandhu Ghosh, Abhijnan Chakraborty |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17735
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SMITE: Enhancing Fairness in LLMs through Optimal In-Context Example Selection via Dynamic Validation Garima Chhikara, Kripabandhu Ghosh, Abhijnan Chakraborty Abstract Large Language Models (LLMs) are widely used for downstream tasks such as tabular classification, where ensuring fairness in their outputs is critical for inclusivity, equal representation, and responsible AI deployment. This study introduces a novel approach to enhancing LLM performance and fairness through the concept of a dynamic validation set, which evolves alongside the test set, replacing the traditional static validation approach. We also propose an iterative algorithm, SMITE , to select optimal in-context examples, with each example set validated against its corresponding dynamic validation set. The in-context set with the lowest total error is used as the final demonstration set. Our experiments across four different LLMs show that our proposed techniques significantly improve both predictive accuracy and fairness compared to baseline methods. To our knowledge, this is the first study to apply dynamic validation in the context of in-context learning for LLMs. 1 Introduction Over the past years, Large Language Models (LLMs) have seen rapid growth in their user base and have attracted significant attention from domain experts and the public at large Ouyang et al. 2022 ; Brown et al. 2020 ; Touvron et al. 2023 ; OpenAI 2023 ; Gemini Team 2023 . In-Context Learning (ICL) capability of LLMs enables them to learn and adapt to a given task solely through the examples provided in the prompt, eliminating the need for further fine-tuning or external training Brown et al. 2020 ; Radford et al. 2019 . These examples are referred to as In-Context Examples (ICEs) or In-Context Demonstrations (ICDs). Since the context window for ICL is limited, it is not possible to input overly long texts, making it crucial to strategically select the in-context examples to achieve good performance in a given task Lu et al. 2021 ; Zhao et al. 2021 ; Li and Qiu 2023 . Some recent works have utilized LLMs for classification of tabular data Hegselmann et al. 2023 ; Slack and Singh 2023 ; Liu et al. 2024 , where the tabular data is converted into natural language and presented to LLMs along with a brief description of the task to elicit predictions. In these tasks, LLMs may inadvertently reinforce social biases embedded in their training data, unless explicitly addressed. This can result in widespread negative consequences, particularly for marginalized and underrepresented groups Abid et al. 2021 ; Ganguli et al. 2022b ; Hutchinson et al. 2020 ; Ganguli et al. 2022a ; Basta et al. 2019 ; Askell et al. 2021 . Given the growing adoption of LLMs across the software industry, it is crucial to address and mitigate these biases to ensure fair and equitable outcomes. In this work, we hypothesize that intelligent selection of in-context examples can lead to fairer outputs from LLMs. Recently, a few works have advocated for selecting a diverse set of in-context examples that is representative of the original dataset, and to check the goodness of the ICE set, typically a validation set is employed which remains the same for all the test cases Li and Qiu 2023 ; Xu and Zhang 2024 . In classical machine learning, when the test and validation sets share a similar data distribution, it is easier to obtain better model parameters. However, the test data is not available during model validation. In contrast, with LLMs and in-context learning (ICL), we already have a pre-trained model (the LLM), and the goal is to select the best in-context examples (ICEs) from the available data. For instance, in tabular classification via ICL, we have access to the entire dataset and need to choose examples that can act as in-context demonstrations (ICDs). In ICL, the entire training set can be used to select ICEs for a specific test sample. We propose a new approach where the validation set used for the selection of examples should closely match the characteristics of the test set. This leads us to introduce the concept of a dynamic validation set which adapts based on the test example. Additionally, we propose an iterative greedy algorithm, SMITE , designed for the selection of examples, that can lead to a strong predictive performance and fairness. To summarize, our contributions are listed as follows : • We present the idea of employing a dynamic validation set (we also call it as proxy test set ) to measure the performance of current set of in-context demonstrations (Section 3.1 ). • We propose an iterative algorithm, SMITE (In-Context Example S election by M inimizing I ndividual and T otal E rror) for the selection of in-context examples. SMITE calculates the error at each iteration and aims to minimize it in subsequent iterations by selecting elements from the support set (Section 3.2 ). • We publicly release the predictions of LLMs for over 1000 test instances across different LLMs, which can spawn future research in this field 1 1 1 Available at https://anonymous.4open.science/r/ICLSelection-2D3A/ . . To the best of our knowledge, this work is one of the first efforts to introduce the novel concept of dynamic validation set, wherein the validation set is created by examining the test set. Proposed SMITE is one the earliest algorithms to account for fairness while selecting examples. 2 Related Work To ensure the reliability of LLMs, it is crucial to eliminate social biases from their outputs. Numerous studies have revealed the existence of bias in the results generated by LLMs Zhang et al. 2023 ; Bi et al. 2023 ; Ferrara 2023 ; Nadeem et al. 2021 ; Kotek et al. 2023 ; Bordia and Bowman 2019 ; Freiberger and Buchmann 2024 ; Zheng et al. 2023 ; Huang et al. 2024 ; Chhikara et al. 2024 . The output of GPT-3 has shown bias against Muslims Abid et al. 2021 . LLMs demonstrate bias when prompted about stereotypes Wang et al. 2024 , and the LLMs response exhibit bias even when prompts do not explicitly inquire about it Huang et al. 2021 . Tabular data is present in many domains Shwartz-Ziv and Armon 2021 . LLMs have been extensively trained on large volumes of natural language data, allowing them to demonstrate remarkable performance across various downstream tasks, even with minimal or no labeled task data Brown et al. 2020 ; OpenAI 2023 . Research has proposed methods for inputting tabular data into LLMs to generate predictions Hegselmann et al. 2023 ; Slack and Singh 2023 . These predictions must be both fair and accurate, particularly when the tabular data originates from safety-critical and high-stakes domains Borisov et al. 2024 ; Grinsztajn et al. 2022 . Work by Liu et al. 2024 demonstrated that flipping the labels of in-context examples can significantly reduce bias. Bi et al. 2023 introduced a novel chain-of-thought method to diminish biases in LLMs. Additionally, Li and Qiu 2023 proposed a diversity-guided search which iteratively refines the selected example permutations. Xu and Zhang 2024 focused on selecting demonstrations that minimize the gap between the outputs of LLMs and the actual input-output mappings. Existing literature has primarily concentrated on the diverse selection of in-context examples and has relied on a static validation set to evaluate the