An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift
Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras · 2026
Read An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift by Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05882.txt.
Introduction
Constantinos Karouzos and 2 co-authors examine in 2026 how preference tuning holds up when a language model leaves its training domain. Preference tuning teaches models from clear human judgments of quality, helpfulness, or safety. Earlier work showed scores for help can drop outside the training domain, but remedies were untested.
Five alignment objectives and several adaptation plans were compared across summarization, question-answering help, and safety tasks. Adaptation included fine-tuning on target-domain data and pseudo-labeling, where the model labels new cases itself. Objectives differed systematically in how they general. Pseudo-labeling substantially reduced the domain-shift damage, yet it also induced mode collapse, where outputs narrow to fewer patterns, exposing a trade-off between generalization and diversity. No single method won on all tasks, so the choice has to fit the job at hand.
Catalog record
| CREATOR | Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05882
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras Abstract Preference tuning aligns pretrained language models to human judgments of quality, helpfulness, or safety by optimizing over explicit preference signals rather than likelihood alone. Prior work has shown that preference-tuning degrades performance and reduces helpfulness when evaluated outside the training domain. However, the extent to which adaptation strategies mitigate this domain shift remains unexplored. We address this challenge by conducting a comprehensive and systematic study of alignment generalization under domain shift. We compare five popular alignment objectives and various adaptation strategies from source to target, including target-domain supervised fine-tuning and pseudo-labeling, across summarization and question-answering helpfulness tasks. Our findings reveal systematic differences in generalization across alignment objectives under domain shift. We show that adaptation strategies based on pseudo-labeling can substantially reduce domain-shift degradation. 1 1 1 Code available at: https://github.com/ckarouzos/prefadap 1 Introduction Reddit Informal Source ( \mathcal{D}_{S} ) News Formal Target ( \mathcal{D}_{T} ) Domain Shift ! 1. Adaptation Strategy Source Only Target SFT Target Pseudo- Labeling Teacher Model 2. Alignment Strategy SFT DPO GRPO RLHF-PPO KTO ORPO Naive Broken Style Forgetting Brittleness Robust Faithful Generalization High Win Rate Generalization Diversity Figure 1: Study Design. We decompose domain transfer into two axes: Adaptation Strategy and Alignment Strategy . We measure the resulting trade-off between generalization and diversity. Large language models (LLMs), such as GPT-5 ( OpenAI 2025 ) , Gemini 3 ( Google 2025 ) and DeepSeek-V3 ( DeepSeek-AI et al. 2025 ) , rely on post-training, i.e., human preference optimization beyond pretraining to improve helpfulness, safety, and truthfulness ( Ouyang et al. 2022 ) . Post-training typically involves supervised fine-tuning (SFT) and preference-based optimization, and has become a standard component of modern LLM development ( Lambert 2025 ) . Despite their widespread adoption, existing work has not systematically characterized the comparative generalization of preference optimization methods under domain shift. Existing work provides limited evidence of out-of-domain generalization for individual objectives. For example, it focuses only on either Direct Preference Optimization ( Rafailov et al. 2023 , DPO) , or as an analysis tool ( Kirk et al. 2024 ) of reinforcement learning from human feedback ( Ouyang et al. 2022 , RLHF) with Proximal Policy Optimization ( Schulman et al. 2017 , PPO) . Moreover, there is no systematic evaluation across a broader range of preference objectives and an analysis of how adaptation strategies can mitigate domain shift. We address this gap via a comprehensive comparative study along two practical axes. The first is the choice of alignment objective, covering a broad spectrum of paradigms: from standard SFT and online reinforcement learning, RLHF-PPO and group relative policy optimization ( Shao et al. 2024 , GRPO) , to offline, RL-free formulations including DPO ( Rafailov et al. 2023 ) , Kahneman–Tversky Optimization ( Ethayarajh et al. 2024 , KTO) , and odds-ratio preference optimization ( Hong et al. 2024 , ORPO) . The second axis is the choice of domain adaptation strategy, ranging from target-domain SFT to target-domain pseudo-labeling. We evaluate alignment objectives and domain adaptation methods across two complementary testbeds. The first is a summarization task adapting from informal Reddit TL;DR data ( Völske et al. 2017 ) to formal CNN/DailyMail (CNN/DM) news articles ( Nallapati et al. 2016 ) . The second is a helpfulness-focused question-answering task transferring between AskEngineers and AskCulinary in the Stanford Human Preferences (SHP) dataset ( Ethayarajh et al. 2022 ) . Figure 1 summarizes the experimental framework. Our contributions are threefold: • A controlled, apples-to-apples comparison of five alignment objectives under domain shift. • Empirical evidence that practical adaptation strategies, especially pseudo-labeling, can substantially reduce target-domain degradation relative to target-domain SFT. • A characterization of generalization and diversity failure cases observed across objectives and adaptation strategies to inform practical deployment under domain shift. 2 Related Work 2.1 Preference Alignment Alignment has evolved from SFT to RLHF, where a reward model (RM) guides policy updates ( Christiano et al. 2017 ; Stiennon et al. 2020 ; Ouyang et al. 2022 ) . Recent work analyzes RLHF as divergence estimation ( Chaudhari et al. 2025 ; Haldar et al. 2025 ) , but often suffers from training instability ( Rafailov et al. 2023 ) . This have motivated DPO and other RL-free variants, to improve stability ( Meng et al. 2024 ; Ethayarajh et al. 2024 ; Zhao et al. 2023 ; Cho et al. 2025 ; Wang et al. 2025 ; Guo et al. 2025a ) . Concurrently, reference-free and odds-ratio methods such as ORPO integrate alignment directly into language modeling or multi-objective frameworks ( Hong et al. 2024 ; Bansal et al. 2025 ; Liu et al. 2025 ; Chen et al. 2025 ; Luo et al. 2025 ) . Further extensions view alignment through a game-theoretic or group-based lens, such as GRPO and Nash-style self-play ( Yao et al. 2025a ; Zhu et al. 2025a ; Zhu et al. 2025b ; Wu et al. 2025 ; Tang et al. 2025 ; Zhou et al. 2025 ) . 2.2 Domain Adaptation Strategies Standard adaptation relies on domain-adaptive pretraining (DAPT), i.e., continuing the pretraining phase on unlabeled domain-specific data ( Gururangan et al. 2020 ; Kirkpatrick et al. 2017 ) . Other alternatives leverage synthetic supervision via AI teachers (reinforcement learning from AI feedback; RLAIF) or self-play ( Bai et al. 2022 ; Lee et al. 2023 ; Chen et al. 2024 ; Wang et al. 2023 ) . Work on preference data construction highlights the importance of filtering hard negatives and synthesizing high-quality pairs ( He et al. 2025 ; Xiao et al. 2025 ) , alongside data selection curricula that match difficulty to model competence ( Deng et al. 2025 ; Miranda et al. 2025 ; Zhang et al. 2025b ) . Complementary approaches model distribution shifts directly through robust preference estimation, multi-supervisor reweighting, and weak-to-strong generalization frameworks ( Huang et al. 2025 ; Yan et al. 2025 ; Geng et al. 2025 ; Zhu et al. 2025c ; Belakaria et al. 2025 ; Patel et al. 2025 ) . 2.3 Alignment Robustness and Diversity Optimizing for safety or helpfulness often incurs an alignment tax on reasoning or out-of-domain performance ( Lin et al. 2024 ; Balepur et al. 2025 ) . In summarization, this manifests as poor transfer between topics ( Kornilova and Eidelman 2019 ; DeLucia and Dredze 2025 ; Afzal et al. 2024 ) . These failures are often linked to mode collapse and reduced linguistic variability. Recent work proposes diversity-aware objectives to mitigate typicality bias ( Zhang et al. 2025a ; Guo et al. 2025b ; Cao et al. 2025 ; Lanchantin et al. 2025 ; Ismayilzada et al. 2025 ) . While distributionally robust optimization and pluralistic alignment aim to preserve diverse behaviors ( Xu et al. 2025 ; Gölz et al. 2025 ; Lake et al. 2025 ; Yao et al. 2025b ) , empirical comparisons of how standard objectives perform against diversity under domain shift remain limited. 3 Methodology 3.1 Problem Setting We study domain adaptation for aligning models to human preferences, when labels are unavailable. We train a policy, \pi_{\theta} , to generate high-quality outputs