MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning

Jingyan Shen, Jiarui Yao, Rui Yang, Yifan Sun, Feng Luo, Rui Pan, Tong Zhang, Han Zhao · 2025

Read MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning by Jingyan Shen, Jiarui Yao, Rui Yang, Yifan Sun, Feng Luo, Rui Pan, Tong Zhang, Han Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24846.txt.

Introduction

Jingyan Shen and seven co-authors personalize the reward models that steer language models, because one global score cannot speak for everyone. Standard reward modeling assumes a single reward function, yet human preferences split into diverse subgroups. The authors prove that when preferences form a mixture, a single model carries an irreducible error.

MiCRo works in two stages without needing fine-grained annotations. First, context-aware mixture modeling captures the distinct preference groups hidden in ordinary binary preference data. Then an online routing strategy adjusts the mixture weights for each specific context, resolving ambiguity on the fly. The framework aims at pluralistic alignment that respects differing values rather than averaging them away.

Catalog record

CREATORJingyan Shen, Jiarui Yao, Rui Yang, Yifan Sun, Feng Luo, Rui Pan, Tong Zhang, Han Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24846

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning Jingyan Shen, Jiarui Yao, Rui Yang, Yifan Sun, Feng Luo, Rui Pan, Tong Zhang, Han Zhao Abstract Reward modeling is a key step in building safe foundation models when applying reinforcement learning from human feedback (RLHF) to align Large Language Models (LLMs). However, reward modeling based on the Bradley-Terry (BT) model assumes a global reward function, failing to capture the inherently diverse and heterogeneous human preferences. Hence, such oversimplification limits LLMs from supporting personalization and pluralistic alignment. Theoretically, we show that when human preferences follow a mixture distribution of diverse subgroups, a single BT model has an irreducible error. While existing solutions, such as multi-objective learning with fine-grained annotations, help address this issue, they are costly and constrained by predefined attributes, failing to fully capture the richness of human values. In this work, we introduce MiCRo, a two-stage framework that enhances personalized preference learning by leveraging large-scale binary preference datasets without requiring explicit fine-grained annotations. In the first stage, MiCRo introduces context-aware mixture modeling approach to capture diverse human preferences. In the second stage, MiCRo integrates an online routing strategy that dynamically adapts mixture weights based on specific context to resolve ambiguity, allowing for efficient and scalable preference adaptation with minimal additional supervision. Experiments on multiple preference datasets demonstrate that MiCRo effectively captures diverse human preferences and significantly improves downstream personalization. 1 Introduction Reinforcement Learning from Human Feedback (RLHF) unlocks a promising pathway to improve the performance, reliability, and adaptability of AI system deployment Bai et al. 2022 ; Dong et al. 2023a ; Achiam et al. 2023 ; Dong et al. 2024 . Rather than relying on handcrafted reward models, the prevailing approach in RLHF employs preference learning Christiano et al. 2017 to infer reward scores from human feedback, particularly for tasks involving subjective evaluation and open-ended responses without unanimous ground truths Ziegler et al. 2019 . However, most existing methods rely on binary-labeled pairwise datasets building upon the assumption that there exists a global reward function that can model human preferences. This fails to capture the diverse and often contradictory nature of human preferences, ultimately limiting their effectiveness for personalized and pluralistic alignment Chakraborty et al. 2024a ; Yang et al. 2024b ; Mukherjee et al. 2024 ; Luo et al. 2025 . Advancing preference learning to better accommodate heterogeneous human preferences remains an open challenge. Some recent studies seek to capture the diversity by collecting multifaceted annotations that distinguish between different evaluation attributes, e.g., helpfulness, harmlessness, coherence, instruction-following, etc Wang et al. 2024b ; Bai et al. 2022 ; Wang et al. 2024c . Although fine-grained labels provide deeper insights into individual preferences, collecting and curating them significantly increases data acquisition costs. Consequently, existing datasets limit their scope to a handful of pre-defined attributes and often rely on LLM-as-a-Judge Zheng et al. 2023 for labeling response pairs. This raises concerns about their fidelity in representing the nuanced and ever-evolving landscape of human values. Figure 1: Illustration of the two-stage pipeline of MiCRo for capturing personalized preferences. A mixture of reward models (Stage 1) is trained on binary-labeled data, while the context-aware router (Stage 2) dynamically adjusts preference distributions based on user-provided context. The final preference distribution is obtained through a convex combination of different preference distributions. In addition, while users may share common interests, such as preferring a helpful and harmless assistant, their expectations are ultimately individualized and depend on use cases, i.e., contextual factors , as illustrated in Fig. 1 . To better capture such personalization, some approaches construct datasets with elaborate and pluralistic contexts into user prompts Pitis et al. 2024 ; Yang et al. 2024b or system instructions Lee et al. 2024a . While reward models trained on such enriched datasets have shown improved generalization to personalized preferences, designing these criteria manually is still labor-intensive. In this work, we introduce MiCRo , a two-stage, context-aware mixture modeling framework that leverages large-scale binary preference datasets to improve personalized preference learning. We first provide a theoretical result showing that when the underlying preference distribution follows a mixture of subpopulations, preference learning based on a single Bradley-Terry (BT) loss incurs an irreducible error. To address this, we propose a context-aware mixture modeling approach to decompose aggregate preferences into latent subpopulations, each with a distinct reward function. To further adapt to personalized preferences, we propose an online routing strategy with additional contextual information. In summary, our method offers two key advantages: • MiCRo extracts multifaceted human preferences from widely available pairwise comparison datasets without requiring explicit fine-grained annotations or predefined attributes. • MiCRo adapts the mixture heads to personalized preference learning with contextual information with only a limited number of samples. Our extensive experiments across multiple preference datasets empirically demonstrate that MiCRo ’s mixture heads effectively capture diverse preferences and achieve superior performance compared to baselines on multidimensional benchmarks. With the addition of context-aware routing, the full MiCRo framework matches the performance of fully supervised and test-time adaptation methods, underscoring its effectiveness in enhancing downstream personalization. 2 Related Work Reward modeling aims to learn a function that assigns scalar scores to input–output pairs based on human preferences, playing a central role in RLHF by steering LLM behavior toward human-aligned outputs and mitigating harmful responses He et al. 2024a ; Sun et al. 2024 . The typical approach adopts the BT model Bradley and Terry 1952 ; Christiano et al. 2017 ; Stiennon et al. 2020 to learn from pairwise comparisons. To further address the diversity of human preferences, personalized preference learning seeks to align LLMs with user values in underspecified settings with ambiguous or heterogeneous intent Fleisig et al. 2023 ; Baumler et al. 2023 ; Chakraborty et al. 2024b . One major approach focuses on multi-objective alignment through ensembles of reward models. Techniques such as Mixture-of-Experts and model merging are used to decompose reward functions into task-specific or capability-based components Quan 2024 ; Wang et al. 2024b ; Rame et al. 2023 ; Wang et al. 2024a . However, training multiple reward models typically requires manually defined preference dimensions and dense supervision. To mitigate this, HyRe Lee et al. 2024b trains an ensemble offline and adapts it to individual users at test time by dynamically reweighting the components using a small number of user-specific examples. Recent work, DRMs Luo et al. 2025 , decomposes human preferences into a linear space using PCA, offering a promising training-free solution; however, its effectiveness depends on the choice of embedding model. A complementary line of work uses probabilistic approaches to model subgroup or latent preferences without explicit supervision Siththaranjan et al. 2023 ; Poddar et al. 2024 ; Chen et al. 2024 ; Chakraborty et al. 2024a , but their potential for personalization remains underexplored. Table 1: Comparison of different methods and their key characteristics. MiCRo optimizes a mixture of BT loss using binary labels and enables context-aware routing, setting it apart from prior methods in terms of context conditioning and weight learning. Method Binary Labels Context Conditioning Reward Objective Weight Learning Special Characteristic ARMO Wang et al. 2024b \times \times Mean Square Error End-to-end BT Fixed Num of attributes MaxMin Chakraborty et al. 2024a ✓ \times Mixture of BT Hard

Read the complete text