When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy
Xiaofeng Tan, Jun Liu, Bin-Bin Gao, Yuanting Fan, Xi Jiang, Chengjie Wang, Hongsong Wang, Feng Zheng, [0.4em] Southeast University, [0.3em] Work done during Xiaofeng Tan’s internship at Tencent Youtu Lab. · 2026
Read When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy by Xiaofeng Tan, Jun Liu, Bin-Bin Gao, Yuanting Fan, Xi Jiang, Chengjie Wang, Hongsong Wang, Feng Zheng, [0.4em] Southeast University, [0.3em] Work done during Xiaofeng Tan’s internship at Tencent Youtu Lab. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12112.txt.
Introduction
Xiaofeng Tan and seven colleagues show why a diversity guard fails in flow-based RLHF for text-to-image models. Fine-tuning for tastes collapses diversity, yet in flow models entropy stays flat while perceived diversity collapses. Flat entropy comes from the fixed noise schedule; collapse comes from reward-seeking gradients narrowing outputs. They propose perceptual entropy, measuring spread in perceived space, plus two guard strategies. Tests across two base models, neural and rule rewards, and three spaces show gains, best score 0.734 against a 0.366 baseline.
Catalog record
| CREATOR | Xiaofeng Tan, Jun Liu, Bin-Bin Gao, Yuanting Fan, Xi Jiang, Chengjie Wang, Hongsong Wang, Feng Zheng, [0.4em] Southeast University, [0.3em] Work done during Xiaofeng Tan’s internship at Tencent Youtu Lab. |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75226 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12112
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy Xiaofeng Tan, Jun Liu, Bin-Bin Gao, Yuanting Fan, Xi Jiang, Chengjie Wang, Hongsong Wang, Feng Zheng, [0.4em] Southeast University, [0.3em] Work done during Xiaofeng Tan’s internship at Tencent Youtu Lab. Abstract RLHF is widely used to align flow-matching text-to-image models with human preferences, but often leads to severe diversity collapse after fine-tuning. In RL, diversity is often assumed to correlate with policy entropy, motivating entropy regularization. However, we show this intuition breaks in flow models: policy entropy remains constant, even while perceptual diversity collapses. We explain this mismatch both theoretically and empirically: the constant entropy arises from the fixed, pre-defined noise schedule, while the diversity collapse is driven by the mode-seeking nature of policy gradients. As a result, policy entropy fails to prevent the model from converging to a narrow high-reward region in the perceptual space. To this end, we introduce perceptual entropy that captures diversity in a perceptual space and maintains the property of standard entropy. Building upon this insight, we propose two entropy-regularized strategies, Perceptual Entropy Constraint and Perceptual Constraints on Generation Space, to preserve perceptual diversity and improve the quality. Experiments across two base models, neural and rule-based rewards, and three perceptual spaces demonstrate consistent gains in the quality-diversity trade-off; PEC achieves the best overall score of 0.734 (vs. baseline’s 0.366); a complementary setting of PEC further reaches a diversity average of 0.989 (vs. baseline’s 0.047). Our project page is publicly available. 1 Introduction Flow matching [ 38 ] has emerged as a powerful paradigm for text-to-image generation [ 68 ] . Since human-centric criteria such as aesthetics [ 15 ] , semantic consistency [ 48 ] , and safety [ 42 ] are difficult to optimize with likelihood-based pre-training alone, recent work aligns pretrained generative models through Reinforcement Learning from Human Feedback (RLHF) [ 19 , 59 , 40 ] . RLHF addresses this by post-training generative models with feedback from human annotators or reward models. However, diversity can decrease during fine-tuning, which limits both (1) RLHF effectiveness and (2) real-world applicability. Specifically, (1) RLHF requires diverse and informative samples for reward evaluation; reduced diversity narrows exploration and weakens feedback scope. (2) Many applications require coverage across contexts [ 3 , 16 ] . As shown in Fig. 1 , Flow-GRPO produces visually similar samples for both portrait and living-room prompts, while an ideal model should preserve variations in identity, pose, background, layout, lighting, and furnishing style. This motivates us to seek RLHF frameworks that improve quality without sacrificing diversity . Figure 1 : Diversity Collapse. Flow-GRPO improves quality but sacrifices diversity, producing similar styles. In RL research on LLMs, diversity is commonly characterized by policy entropy [ 79 , 31 , 54 , 11 , 1 , 41 ] . Under this view, recent works [ 77 , 12 ] aim to balance diversity and reward \mathcal{R} using policy entropy \exp(\mathcal{H}) , and reports an empirical relationship, \mathcal{R}=-a\exp(\mathcal{H})+b, with coefficients a and b . A natural solution to diversity collapse in flow-based RLHF is therefore policy entropy regularization. However, revisiting this intuition under flow matching reveals a key inconsistency: diversity collapses while the policy entropy surrogate remains constant , as shown in Fig. 2 . Diffusion and flow-based models [ 27 , 55 ] avoid directly modeling p(\mathbf{x}_{0}) by decomposing the process into transitions p(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}) , which we term policy entropy. However, this quantity is fixed by the noise schedule and thus fails to reflect changes in diversity. This raises a fundamental question: what drives diversity collapse in RL for flow models despite constant policy entropy? Contributions. In this paper, we investigate the diversity collapse from the perspective of entropy mechanism in flow-based RLHF, and propose a novel entropy concept with two entropy-regularization strategies to improve quality without sacrificing diversity. Main contributions are highlighted below. First, we investigate the apparent paradox between constant entropy and diversity collapse in flow-based RLHF. We study two questions: (1) why does the policy entropy remain constant after RL fine-tuning? and (2) why can diversity still collapse under this constant policy entropy? (1) For the first question, we show that the noised samples \mathbf{x}_{t} are generated by injecting noise with a pre-defined and fixed intensity into the model-predicted mean \mu_{\theta}(\mathbf{x}_{t}) , resulting in a Gaussian distribution with fixed covariance. Hence, both likelihoods and policy entropy are determined by the fixed noise scheduler and are independent of the flow model parameters \theta . (2) For the second question, we find that this paradox stems from the mode-seeking nature of online RL, where the policy is optimized to concentrate probability distribution on high-reward region in the reward space. In this case, the multi-peak pretrained policy distribution may collapse into a single-peak perceptual distribution. This motivates an RLHF framework that encourages broad coverage of multiple high-reward regions in perceptual space, rather than convergence to a single peak, as shown in Fig. 4 . Second, motivated by the observation that policy entropy in the generative space fails to capture diversity collapse, we introduce a novel entropy concept , perceptual entropy \mathcal{H}_{\text{perc}} , which faithfully reflects this phenomenon while remaining consistent with prior entropy-based methods. Building on this concept, we further reveal an empirical entropy-based reward for flow models, \mathcal{R}=-a\exp(\mathcal{H}_{\text{perc}})+b, which is analogous to relationships observed in LLMs [ 12 ] . Furthermore, building on the above analysis, we propose two entropy-regularized strategies to mitigate diversity collapse: Perceptual Entropy Constraint (PEC), which incorporates perceptual entropy into the reward signal, and Perceptual Constraints on Generation Space , which aim to align the collapsed perceptual space with the consistent VAE space to maintain diversity. Finally, comprehensive experiments on FLUX.dev [ 33 ] and SD3.5-M [ 18 ] using neural and rule-based rewards, alongside PickScore [ 32 ] , DINO [ 8 ] , and CLIP [ 48 ] perceptual encoders, demonstrate that perceptual entropy improves the quality-diversity trade-off: PEC achieves the best overall score of 0.734, and a complementary setting of PEC further reaches a diversity average of 0.989. 2 Preliminary RLHF Objective and Policy Gradient. Given an external prompt condition c\sim\mathcal{D} and a policy p_{\theta} that generates \mathbf{x}\sim p_{\theta}(\cdot\mid c) , RLHF aims to maximize the expected reward: \max_{\theta}\;\mathcal{J}(\theta):=\mathbb{E}_{c\sim\mathcal{D},\,\mathbf{x}\sim p_{\theta}(\cdot\mid c)}\bigl[\mathcal{R}(\mathbf{x},c)\bigr], (1) Considering that the gradient of the objective is difficult to estimate directly, policy gradient methods such as REINFORCE [ 67 ] decompose it into a sum of log-likelihood gradients: \nabla_{\theta}\mathcal{J}(\theta)=\mathbb{E}\Bigl[\sum_{t=1}^{T}\nabla_{\theta}\log p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},c)\,\mathcal{A}(\mathbf{x},c)\Bigr], (2) where \mathcal{A}(\mathbf{x},c) denotes an advantage estimated by the final sample \mathbf{x}\equiv\mathbf{x}_{0} . Group-Relative Advantage. GRPO [ 53 ] computes advantages via group-wise normalization. For each prompt, it samples K candidates \{\mathbf{x}^{k}\}_{k=1}^{K} and normalizes rewards within the group: \mathcal{A}(\mathbf{x}^{i},c)=\frac{\mathcal{R}(\mathbf{x}^{i},c)-\mathrm{mean}\{\mathcal{R}(\mathbf{x}^{k},c)\}}{\mathrm{std}\{\mathcal{R}(\mathbf{x}^{k},c)\}}. (3) Policy Density in Flow Models. For flow models, the policy in Eq. ( 2 ) is defined by the reverse-time SDE transition [ 19 , 40 ] . Under the flow parameterization, each step follows a Gaussian transition \mathbf{x}_{t-1}=\mu_{\theta}(\mathbf{x}_{t},t,c)+\sigma_{t}\sqrt{\mathrm{d}t}\,\bm{\epsilon},\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\hskip 9.24994pt\Rightarrow\hskip 9.24994ptp_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},c)=\mathcal{N}\bigl(\mathbf{x}_{t-1};\mu_{\theta}(\mathbf{x}_{t},t,c),\sigma_{t}^{2}\mathrm{d}t\,\mathbf{I}\bigr), (4) where covariance \sigma_{t}^{2}\mathrm{d}t\,\mathbf{I} is fixed by noise schedule, while mean \mu_{\theta} is given by the velocity field [ 40 ] : \mu_{\theta}(\mathbf{x}_{t},t,c)=\mathbf{x}_{t}+\Bigl[v_{\theta}(\mathbf{x}_{t},t,c)+\frac{\sigma_{t}^{2}}{2t}\bigl(\mathbf{x}_{t}+(1-t)v_{\theta}(\mathbf{x}_{t},t,c)\bigr)\Bigr]\mathrm{d}t. (5) Flow-Based GRPO Objective. Flow-GRPO optimizes the policy gradient in Eq. ( 2 ) with the group-relative advantage in Eq. ( 3 ), using a clipped