Jointly Reinforcing Diversity and Quality in Language Model Generations

Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, Tianlu Wang · 2025

Read Jointly Reinforcing Diversity and Quality in Language Model Generations by Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, Tianlu Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02534.txt.

Introduction

Tianjian Li and seven co-authors present a 2025 machine learning paper on a trade-off in language model training: polishing answer quality often narrows the range of ideas a model produces. That narrowing hurts brainstorming, storytelling, and open problem solving.

Their DARLING framework trains for quality and semantic diversity together. A learned partition function measures diversity by meaning rather than surface wording. During online reinforcement learning, that diversity signal is added to the reward for quality, so both goals shape each update. Tests span instruction following, creative writing, and competition maths across model families. Diversity-aware training beat quality-only rivals on quality and novelty at once. In maths, both first-try accuracy and variety across tries improved. The authors argue that chasing diversity actively fuels exploration.

Catalog record

CREATORTianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, Tianlu Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02534

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Jointly Reinforcing Diversity and Quality in Language Model Generations Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, Tianlu Wang Abstract Post-training of Large Language Models (LMs) often prioritizes accuracy and helpfulness at the expense of diversity. This creates a tension: while post-training improves response quality, it also sharpens output distributions and reduces the range of ideas, limiting the usefulness of LMs in creative and exploratory tasks such as brainstorming, storytelling, or problem solving. We address this challenge with Diversity-Aware Reinforcement Learning ( Darling ), a framework that jointly optimizes for response quality and semantic diversity. At its core, Darling introduces a learned partition function to measure diversity beyond surface-level lexical variations. This diversity signal is then combined with a quality reward during online reinforcement learning, encouraging models to generate outputs that are both high-quality and distinct. Experiments across multiple model families and sizes show that Darling generalizes to two regimes: non-verifiable tasks (instruction following and creative writing) and verifiable tasks (competition math). On five benchmarks in the first setting, Darling consistently outperforms quality-only RL baselines, producing outputs that are simultaneously of higher quality and novelty. In the second setting, it achieves higher pass@1 (solution quality) and pass@ k k (solution variety). Most strikingly, explicitly optimizing for diversity catalyzes exploration in online RL, which manifests itself as higher-quality responses. † † affiliation: Meta FAIR † † affiliation: Carnegie Mellon University † † affiliation: Johns Hopkins University † † contribution: Work done during an internship at Meta † † date: August 10, 2026 † † correspondence: Tianjian Li [email protected] , Tianlu Wang [email protected] † † Code: https://github.com/facebookresearch/darling 1 Introduction Diversity plays a critical role in numerous real-world applications ( Lu et al. 2025 ) , directly influencing their effectiveness, utility, and innovation potential ( Nagarajan et al. 2025 ; Zhang et al. 2025b ) . For example, in scientific discovery, diverse hypotheses or experimental outcomes enable researchers to explore a broader solution space, potentially uncovering novel insights and breakthroughs ( Gruver et al. 2023 ; Romera-Paredes et al. 2024 ; Si et al. 2025 ) . Similarly, in other tasks such as creative writing ( Fan et al. 2018 ) and natural conversations ( Li et al. 2016a ) , diverse outputs are essential for innovation that requires avoiding repetitive or predictable outcomes. In reinforcement learning (RL) and self-training loops of LMs, diversity is also crucial. Policies that produce diverse outputs enable thorough exploration of the action space, critical for discovering novel and effective strategies ( Chen et al. 2025a ; Cheng et al. 2025 ; Wu et al. 2025a ; He et al. 2025a ) . However, recent developments in Language Models (LMs) have revealed a significant issue: post-training of LMs often result in overly sharpened output distributions ( Huang et al. 2025 ; Li et al. 2025b ) , leading to significant reduction of diversity among generated responses ( Padmakumar and He 2024 ; Shypula et al. 2025 ) , even sharing identical prefixes ( Ji et al. 2025 ) or becoming near duplicates ( Mahony et al. 2024 ; Zhang et al. 2024 ) , reducing the overall informativeness of outputs ( Lin et al. 2021 ; Kirk et al. 2024 ; West and Potts 2025 ; Yang and Holtzman 2025 ; Yun et al. 2025 ) . To address the loss of diversity during LM post-training, we propose Diversity-Aware Reinforcement Learning ( Darling ), an online RL objective that (a) measures diversity at the semantic level via a learned classifier, and (b) fuses diversity and quality to condition gradient updates on “usefully different” trajectories. As illustrated in Figure 1 , Darling first partitions rollouts from a single user prompt into distinct semantic clusters using a semantic classifier, capturing diversity beyond superficial lexical differences (§ 3.1 ). It then combines (multiplies) the diversity assessment with a quality reward, amplifying the advantage of log-probabilities for responses that are both high-quality and semantically diverse (§ 3.2 ). We validate Darling ’s effectiveness and generalizability across both non-verifiable and verifiable tasks, using various language model families and sizes. Experimental results demonstrate that Darling preserves the original model’s diversity and achieves improved benchmark performance in both non-verifiable instruction-following and creative writing tasks, as well as verifiable math problems. In summary, our contributions are three-fold: (1) We propose Darling , an RL framework that simultaneously optimizes quality and diversity, preventing diversity collapse during post-training. (2) We demonstrate that a learned semantic classifier can serve as a scalable and generalizable signal of diversity to integrate into online RL training. (3) We show that explicitly optimizing for diversity promotes greater exploration, often leading to improvements in quality in both non-verifiable (creative writing) and verifiable (competition math) benchmarks. Figure 1: Diversity-Aware Reinforcement Learning ( Darling ): We first partition LLM generations into semantically equivalent clusters (represented by colors). While standard GRPO ( Shao et al. 2024 ) increases probabilities based on response quality only, Darling amplifies the increase in probability of diverse and high-quality responses. 2 Notations and Preliminaries Let \mathcal{S} denote the set of natural language token sequences, a language model \pi(\cdot\mid x) takes a token sequence x\in\mathcal{S} as its input and outputs a probability distribution over \mathcal{S} . We denote the probability of a specific token sequence y\in\mathcal{S} as \pi(y\mid x) and denote the token at position t as y^{t} . Given a reward function r:\mathcal{S}\times\mathcal{S}\rightarrow\mathbb{R} which maps a pair of natural language instructions and responses x,y\in\mathcal{S} to a scalar value r(x,y)\in\mathbb{R} , LM post-training aims to solve the following KL constrained optimization problem: \max_{\pi}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi(\cdot\mid x)}\left[r(x,y)-\beta\frac{\pi(y\mid x)}{\pi_{\text{ref}}(y\mid x)}\right]=\max_{\pi}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi(\cdot\mid x)}[r(x,y)]-\beta\mathbb{D}_{\text{KL}}\left(\pi(\cdot\mid x)||\pi_{\text{ref}}(\cdot\mid x)\right), (1) where \mathcal{D} is a dataset of prompts and \pi_{\text{ref}} is a reference model from which we do not want to deviate too much, usually implemented as the LM before the optimization process. Group Relative Policy Optimization ( Shao et al. 2024 ) optimizes ( 1 ) by maximizing the following objective: \mathbb{E}_{x\sim\mathcal{D},\{y_{i}\}_{i=1}^{n}\sim\pi_{\text{act}}(\cdot\mid x)}\left[\frac{1}{n}\sum_{i=1}^{n}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\Big(\min\big(\text{IS}_{i,t}\cdot A_{i,t},~\text{clip}(\text{IS}_{i,t},1-\varepsilon,1+\varepsilon)\cdot A_{i,t}\big)-\beta\mathbb{D}_{\text{KL}}(\pi_{\theta}||\pi_{\text{ref}})\Big)\right] (2) where n is the number of responses per prompt, and \text{IS}_{i,t}=\frac{\pi_{\theta}(y_{i}^{t}\mid y_{i}^{<t},x_{i})}{\pi_{\text{act}}(y_{i}^{t}\mid y_{i}^{<t},x_{i})} is the importance sampling ( Kloek and van Dijk 1978 ) ratio between the current policy \pi_{\theta} and the actor \pi_{\text{act}} (the model used to generate y_{i} ), and A_{i,t}=\frac{r(x,y_{i})-\text{mean}_{j=1}^{n}(r(x,y_{j}))}{\text{std}_{j=1}^{n}(r(x,y_{j}))} (3) is the advantage of the response y_{i} , measuring how much better (or worse) is y_{i} over an average response, and \varepsilon is a hyperparameter preventing the importance sampling term \text{IS}_{i,t} from being too large or small. GRPO and its variants ( Yu et al. 2025 ; Liu et al. 2025c ; Hu 2025 ) are widely adopted as some of the go-to algorithms for LM post-training ( DeepSeek-AI et al. 2025 ; Liu et al. 2025a ; Yang et al. 2025a ) due to its simplicity and stability. In our work, we use GRPO as our starting baseline. 3 Method: DARLING Figure 1 illustrates our method Darling (Diversity Aware Reinforcement Learning). We first partition the responses using our developed classifier (§ 3.1 ) that captures semantic similarity, then combine diversity and quality signals in an RL framework to generate diverse and high-quality responses (§ 3.2 ). 3.1 Partitioning the Responses into Semantic Equivalence Classes Figure 2: Example of partitioning a group of responses into semantically equivalent

Read the complete text