Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning
Yanbei Jiang, Amr Keleg, Ryandito Diandaru, Jey Han Lau, Lea Frermann, Biaoyan Fang, Fajri Koto · 2026
Read Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning by Yanbei Jiang, Amr Keleg, Ryandito Diandaru, Jey Han Lau, Lea Frermann, Biaoyan Fang, Fajri Koto (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05756.txt.
Introduction
Jiang and six colleagues test whether language models can match target output distributions over repeated prompts, not just single answers. Off-the-shelf models and standard alignment fail to control gender, race, and sentiment mixes in occupational contexts. Their fine-tuning couples steering-token calibration with semantic alignment, achieving precise distributional control across six datasets.
Catalog record
| CREATOR | Yanbei Jiang, Amr Keleg, Ryandito Diandaru, Jey Han Lau, Lea Frermann, Biaoyan Fang, Fajri Koto |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54574 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning Yanbei Jiang, Amr Keleg, Ryandito Diandaru, Jey Han Lau, Lea Frermann, Biaoyan Fang, Fajri Koto Abstract While the real world is inherently stochastic, Large Language Models (LLMs) are predominantly evaluated on single-round inference against fixed ground truths. In this work, we shift the lens to distribution alignment : assessing whether LLMs, when prompted repeatedly, can generate outputs that adhere to a desired target distribution, e.g. reflecting real-world statistics or a uniform distribution. We formulate distribution alignment using the attributes of gender, race, and sentiment within occupational contexts. Our empirical analysis reveals that off-the-shelf LLMs and standard alignment techniques, including prompt engineering and Direct Preference Optimization, fail to reliably control output distributions. To bridge this gap, we propose a novel fine-tuning framework that couples Steering Token Calibration with Semantic Alignment. We introduce a hybrid objective function combining Kullback-Leibler divergence to anchor the probability mass of latent steering tokens and Kahneman-Tversky Optimization to bind these tokens to semantically consistent responses. Experiments across six diverse datasets demonstrate that our approach significantly outperforms baselines, achieving precise distributional control in attribute generation tasks. Code and data are available at https://github.com/YanbeiJiang/Distribution-Debias . 1 Introduction Although generative Large Language Models (LLMs) are inherently probabilistic, existing evaluation methods typically focus on a single-round generation De-Arteaga et al. 2019 ; Soundararajan and Delany 2024 ; Pan et al. 2025 . This narrow view becomes problematic in real-world applications, where repeated prompting can lead to distributional properties in LLM output that deviates from the intended statistics. We refer to this phenomenon as distribution bias Shrestha and Srinivasan 2025 , defined as the difference between the observed output distribution and the target distribution. Distribution bias matters because it influences how LLMs represent social attributes such as gender, race, and sentiment, with implications for fairness Gallegos et al. 2024 , personalization Salemi et al. 2024 , and trustworthiness Litschko et al. 2023 . Figure 1: Distributional Bias in Multi-round LLM Generation While Shrestha and Srinivasan 2025 examine distributional bias at the word level probability, our work focuses on multi-round generation. Consider the example in Figure 1 : if we independently prompt an LLM 100 times with “ Write a story about a nurse in the UK! ’’, the aggregate gender distribution may reflect parity (equally many men and women nurses) or reflect and thereby reinforce stereotypes (predominantly women nurses). 1 1 1 We acknowledge that treating gender as binary unduly simplifies the concept, but adopt this simplification for experimental clarity. Our method extends to >2 classes. This observation highlights a fundamental limitation: LLMs lack distributional understanding and controllability in repeated generations. This question arises in two scenarios, reflecting different desiderata. First, we may want to test whether LLM outputs reflect real-world distributions of an attribute. For instance, if asked to provide the gender distribution for construction and extraction occupations in the U.S. in 2024, the expected ratio is approximately 96% male and 4% female. 2 2 2 https://www.bls.gov/cps/cpsaat39.htm (29 Dec, 2025). Second, we may wish to enforce a user-specified distribution, e.g., where educational texts should represent all genders equally, as is often the case in the context of debiasing Rooein et al. 2025 . This paper investigates two core questions: (i) What distributional biases do LLMs exhibit when prompted repeatedly, relative to real-world statistics or a user-specified distribution? (ii) Can we control attribute distributions in multi-round generation to match a desired target distribution? We formulate this challenge as distributional bias control in multi-round LLM generations repeating the same instruction in a number of independent trials. Our contributions can be summarized as follows: (i) We show that distributional biases exist in multi-round generation of off-the-shelf LLMs, both relative to real-world statistics and user-specified uniform distributions. (ii) We create distribution-aware preference datasets focusing on occupations in the UK and US, and attributes such as gender, race, and sentiment across two settings: attribute generation and story generation. (iii) We introduce a novel method to control output distributions in multi-round generation by fine-tuning LLMs with a hybrid objective combining distribution calibration (KL divergence) and semantic consistency (KTO), steering LLM outputs toward the desired target distribution. Through extensive experiments, we demonstrate that our approach significantly outperforms baselines. 2 Related Work Control Generation Control generation focuses on guiding and constraining the LLM output on certain conditions, e.g., styles De Langis et al. 2024 ; Toshevska and Gievska 2025 ; Miura et al. 2025 and attributions Liang et al. 2024 ; Lorandi and Belz 2023 ; Pang et al. 2025 . It has been implemented in various ways, including (1) prompt-based generation Jie et al. 2024 ; Suzgun et al. 2022 ; Liu et al. 2024 ; Jiang et al. 2025a , (2) classifier-guided generation Konen et al. 2024 ; Mai et al. 2023 , and (3) reinforcement learning–based control generation Shulev and Sima’an 2024 ; Deng et al. 2022 . Distribution alignment is in line with control generation in restricting LLM outputs on certain attributes. However, control generation targets on generating the desired output per generation while distribution alignment focuses on controlling LLMs’ generation over several and independently repeated outputs. Debiasing Distribution alignment is closely related to debiasing in LLMs, which aims to ensure LLMs generate fair responses/distribution to each protected attribute group (e.g., gender Dinan et al. 2020 ; Fan and Gardent 2022 ; Soundararajan and Delany 2024 or ethnicity Narayanan Venkit et al. 2023 ; Fang et al. 2024 ). Recent work explores various debiasing methods, including (1) prompt-based debiasing Wan and Chang 2025 ; Bansal et al. 2022 ; Huang et al. 2024 ; (2) pretraining-based debiasing Zakizadeh and Pilehvar 2025 ; Gira et al. 2022 ; Shrestha and Srinivasan 2025 ; (3) reinforcement learning/preference–based debiasing Xia et al. 2024 ; Fan et al. 2025 ; Jiang et al. 2025b . Those methods are tested on carefully curated datasets with desired attribute ratios and evaluated based on the output result over a few runs for each data point. Whether debiasing methods can be applied to guide LLMs to align desired distribution on multi-round generation remains an open question. LLM Output Consistency Research on LLM consistency investigates why the same model can produce different or even contradictory outputs under minor—or no—changes to input or conditions Wu et al. 2025 ; Vázquez et al. 2024 ; Yang et al. 2024 . Increasing decoding hyperparameters such as temperature or top-p sampling often leads to more diverse outputs Peeperkorn et al. 2024 . While related, distribution alignment differs from consistency: it seeks to match the overall attribute distribution across repeated generations while permitting individual variation. Both approaches address variability in generation and are therefore complementary. Our work builds on these insights by analyzing default temperature and top-p settings in multi-round generation and evaluating their impact across different configurations. 3 Methods We introduce distribution alignment , a framework designed to control attribute distributions in multi-round LLM generation. The objective is to align the output distribution with a specified target distribution for sensitive attributes (e.g., gender, race, sentiment) while preserving semantic quality. We propose a two-stage optimization strategy: (1) constructing a distribution-aware preference dataset, and (2) fine-tuning the model using a hybrid objective that combines Kullback-Leibler (KL) divergence for distribution calibration