Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback

Gihoon Kim, Euntai Kim · 2026

Read Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback by Gihoon Kim, Euntai Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12595.txt.

Introduction

Gihoon Kim and Euntai Kim fix a hidden flaw in personal preference learning from human feedback. Standard methods assume one reward for all users, which blocks personalization. Variational Preference Learning adds user-specific latent traits, yet the authors find those traits can collapse and be ignored. Sparse data and strong decoders drive that failure. Their swap-guided method builds fake swap annotators whose mirrored tastes steer the encoder. Tests showed less collapse and better prediction of user choices.

Catalog record

CREATORGihoon Kim, Euntai Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69452
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12595

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback Gihoon Kim, Euntai Kim Abstract Reinforcement Learning from Human Feedback (RLHF) is a widely used approach to align large-scale AI systems with human values. However, RLHF typically assumes a single, universal reward, which overlooks diverse preferences and limits personalization. Variational Preference Learning (VPL) seeks to address this by introducing user-specific latent variables. Despite its promise, we found that VPL suffers from posterior collapse. While this phenomenon is well known in VAEs, it has not previously been identified in preference learning frameworks. Under sparse preference data and with overly expressive decoders, VPL may cause latent variables to be ignored, reverting to a single-reward model. To overcome this limitation, we propose Swap-guided Preference Learning (SPL). The key idea is to construct fictitious swap annotators and use the mirroring property of their preferences to guide the encoder. SPL introduces three components: (1) swap-guided base regularization, (2) Preferential Inverse Autoregressive Flow (P-IAF), and (3) adaptive latent conditioning. Experiments show that SPL mitigates collapse, enriches user-specific latents, and improves preference prediction. Our code and data are available at https://github.com/cobang0111/SPL 1 Introduction Reinforcement learning from human feedback (RLHF) has emerged as a prominent method for aligning large-scale AI systems with human values in various fields, particularly natural language processing ( 25 ) . In RLHF, a reward model is first trained on human comparison data, and then a policy is optimized with reinforcement learning. This approach aligns model behavior more closely with human evaluations, improving performance, accuracy, and fairness across diverse domains ( 18 ; 15 ) . However, most existing RLHF approaches ( 8 ; 25 ) are based on the single-reward assumption that all human preferences can be represented by a universal reward function. This assumption is originated from the Bradley–Terry–Luce (BTL) model ( 4 ) , which is commonly used to model pairwise comparisons and treats preferences as if they were generated from a shared scoring function. While mathematically convenient, this single-reward assumption is problematic in practice. Human preferences are not homogeneous but plural and often diverge across individuals or groups. Recent studies have shown that collapsing diverse perspectives into a single reward function introduces systematic bias in favor of majority preferences, overlooking groups and reducing fairness ( 29 ; 12 ; 6 ) . Consequently, models trained under this assumption may disadvantage underrepresented populations, even when their preferences are valid and important. To address this issue, researchers have begun exploring what we refer to as personalized alignment (pluralistic alignment) ( 34 ) . Instead of forcing all preferences into a single universal reward function, personalized alignment seeks to align different reward functions with different individuals according to their preferences, thereby capturing the heterogeneity of human values. One leading approach is Variational Preference Learning (VPL) ( 28 ) , which encodes user-specific latent variables from preference data and decodes them into corresponding rewards. This framework allows AI systems to flexibly adapt to diverse users without relying on predefined groupings or rigid categorization. Despite its promise, we found in our experiments that VPL suffers from practical failure mode: posterior collapse. This phenomenon is sometimes observed in VAEs ( 3 ; 7 ; 14 ; 19 ; 36 ) but has not previously been identified in preference learning frameworks. When combined with a strong reward decoder, this posterior collapse can cause the encoder’s latent variable to become uninformative and effectively ignored. In such cases, the latent variable fails to capture user-specific information, and the decoder explains preferences without relying on it. Training then reduces to an implicit single reward model, ignoring minority preferences and undermining the goal of personalized alignment. To overcome this, we introduce Swap-guided Preference Learning (SPL), an expressive variational framework for personalized alignment that explicitly leverages the structural properties of preference pair data. To the best our knowledge, we are the first to report and address posterior collapse in preference learning. Our approach improves user-latent encoding and reward decoding through three key innovations: (i) Swap-guided Base Regularization , which encouraging latent space shows mirrored characteristics under preference swapping; (ii) Preferential-Inverse Autoregressive Flow , which disentangles swap-reversal and swap-invariant signals, conditioning a inverse autoregressive flow on them to yield improved latent representations without collapse; and (iii) Adaptive Latent Conditioning , which dynamically adjusts the contribution of the latent variable to reward prediction. Together, these mechanisms consistently reduce posterior collapse and enable more faithful and pluralistic preference modeling. Figure 1: Overview of SPL. We propose Swap-guided Preference Learning (SPL), a new framework for personalized alignment. RLHF ( 25 ) cannot adequately reflect user diversity. To overcome this limitation, VPL ( 28 ) encodes text data consisting of a prompt x and response y into a single latent embedding. However, this encoding process is highly prone to collapse. In contrast, SPL leverages the structural properties of preference data through guiding mechanisms and a Preferential Inverse Autoregressive flow, allowing the latent space to capture user-specific characteristics. 2 Preliminary fundamentals Reinforcement Learning from Human Feedback For post-training of Large Language Models (LLM), RLHF relies on a dataset of N human preference pairs, {\mathbb{D}}=\{(x^{i},y_{w}^{i},y_{l}^{i})\}_{i=1}^{N} , where x is a prompt and (y_{w},y_{l}) denote the chosen(winning) and rejected(losing) responses, respectively. RLHF assumes an single universal reward function r_{\phi}(x,y) , optimized by maximizing the log-likelihood of observed preferences: \mathbb{E}_{(x,y_{w},y_{l})\sim{\mathbb{D}}}\Bigl[\log p_{\phi}(y_{w}\succ y_{l}\mid x)\Bigr]. (1) The preference probability p_{\phi}(y_{w}\succ y_{l}\mid x) is typically modeled via the Bradley–Terry–Luce (BTL) model ( 4 ) : p_{\phi}(y_{w}\succ y_{l}\mid x)=\frac{\text{exp}\bigl({r_{\phi}(x,y_{w})}\bigr)}{\text{exp}\bigl({r_{\phi}(x,y_{w})}\bigr)+\text{exp}\bigl({r_{\phi}(x,y_{l})}\bigr)}=\sigma\!\bigl(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l})\bigr), (2) where \sigma denotes the logistic function. Thus, the reward function r_{\phi} is trained to explain human-preferred outcomes, and the learned reward model is subsequently used to optimize a policy aligned with human judgments. Variational Approach for Personalized Alignment A central direction in personalized alignment is to condition reward models and policies on user-specific information ( 24 ; 28 ; 2 ; 32 ; 13 ) . Among these approaches, Variational Preference Learning (VPL) ( 28 ) is particularly influential. Inspired by variational autoencoders (VAEs) ( 16 ) , VPL introduces a user-specific latent variable {\bm{z}}\in{\mathbb{R}}^{d} inferred from each user h ’s preference dataset {\mathbb{D}}_{h}=\{(x^{i},y_{w}^{i},y_{l}^{i})\}_{i=1}^{n}\subset{\mathbb{D}} . The encoder produces an approximate posterior q_{\psi}({\bm{z}}\mid{\mathbb{D}}_{h}) , while the decoder predicts rewards for prompt–response pairs (x,y) conditioned on {\bm{z}} , denoted as r_{\phi}(x,y,{\bm{z}}) . Formally, VPL extends the objective in Eq.( 1 ) by adding a variational regularization term. This yields an evidence lower bound (ELBO): \underset{\begin{subarray}{c}h\sim{\mathbb{H}}\end{subarray}}{\mathbb{E}}\Bigg[\underset{\begin{subarray}{c}{\bm{z}}\sim q_{\psi}(z\mid{\mathbb{D}}_{h})\\ (x,y_{w},y_{l})\sim{\mathbb{D}}_{h}\end{subarray}}{\mathbb{E}}[\log p_{\phi}(y_{w}\succ y_{l}\mid x,{\bm{z}})]-\beta D_{\mathrm{KL}}\bigl[q_{\psi}({\bm{z}}\mid{\mathbb{D}}_{h})\|p({\bm{z}})\bigr]\Bigg], (3) where \beta is KL divergence weight and the \log p({\bm{z}}) represents the prior distribution’s log-density, selected as \mathcal{N}(\mathbf{0},\mathbf{I}) . This objective maximizes the conditional log-likelihood of preferences while regularizing the user-specific posterior toward the prior, thereby preventing overfitting and encouraging generalizable latent structure. By leveraging {\bm{z}} , VPL provides flexibility in modeling personalized traits and has shown strong empirical performance in capturing diverse preferences. However, recent work ( 22 ) indicates that compressing rich textual preference data into a single latent embedding {\bm{z}} remains highly challenging. Inverse Autoregressive Flow Normalizing flows ( 31 ) is a framework for constructing flexible posterior distributions by applying a sequence of invertible transformations. Among them, Inverse Autoregressive Flow (IAF) ( 17 ) is specifically designed to enrich the expressivity of variational posteriors while preserving computational tractability. The procedure begins

Read the complete text