Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment
Xiaotian Zhang, Ruizhe Chen, Yang Feng, Zuozhu Liu · 2025
Read Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment by Xiaotian Zhang, Ruizhe Chen, Yang Feng, Zuozhu Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12663.txt.
Introduction
Xiaotian Zhang, Ruizhe Chen, Yang Feng, and Zuozhu Liu present Persona-judge, a way to tailor a language model to one user's tastes. Their twenty twenty-five paper notes that standard alignment trains the model on reward data, which is costly and hard to redo for each new set of values.
Persona-judge needs no training. A draft model proposes the next token with one preference in mind, and a judge model, set to another preference, checks each token and accepts or rejects it. The model's own sense of what fits a stated taste does the work. Tests show this gives personal alignment for unseen tastes at low cost.
Catalog record
| CREATOR | Xiaotian Zhang, Ruizhe Chen, Yang Feng, Zuozhu Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12663
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment Xiaotian Zhang, Ruizhe Chen, Yang Feng, Zuozhu Liu Abstract Aligning language models with human preferences presents significant challenges, particularly in achieving personalization without incurring excessive computational costs. Existing methods rely on reward signals and additional annotated data, limiting their scalability and adaptability to diverse human values. To address these challenges, we introduce Persona-judge, a novel discriminative paradigm that enables training-free personalized alignment with unseen preferences. Instead of optimizing policy parameters through external reward feedback, Persona-judge leverages the intrinsic preference judgment capabilities of the model. Specifically, a draft model generates candidate tokens conditioned on a given preference, while a judge model, embodying another preference, cross-validates the predicted tokens whether to be accepted. Experimental results demonstrate that Persona-judge, using the inherent preference evaluation mechanisms of the model, offers a scalable and computationally efficient solution to personalized alignment, paving the way for more adaptive customized alignment. Our code is available here . 1 Introduction Aligning large language models (LLMs) has shown tremendous potential in following human instructions and reflecting human preferences Stiennon et al. (2020) ; Ouyang et al. (2022a) ; Bai et al. (2022) . However, aligning with a unified preference often overlooks the need to accommodate individuals, as individual preferences can vary significantly due to factors such as cultural, educational, religious, and political backgrounds Gordon et al. (2022) ; Cheng et al. (2023) ; Chen et al. (2024e) . Personalized alignment Jang et al. (2023) addresses this gap by tailoring language models to individual human preferences and values, which is crucial for human-AI interaction and user-focused applications ( Kirk et al., 2024 ; Sorensen et al., 2024 ) . Policy-based methods Zhou et al. (2024) ; Li et al. (2020) , such as Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022a ) and Direct Preference Optimization (DPO) ( Rafailov et al., 2024 ) , as illustrated in Figure 1 (a), use training signals from explicit or implicit rewards to optimize policy models. However, these methods struggle to scale to changing personalized preferences, with challenges including the construction of high-quality preference datasets and the substantial computational costs associated with optimizing the policy. Figure 1: In contrast to previous paradigms, Persona-judge gets rid of the need for additional training of policy models or reliance on external reward signals. Figure 2: An illustration of the inference phase for Persona-judge. Given the target preferences and the current context, different preference prefixes are first assigned to the same base model. In each sequence-length iteration, the two models alternate in the roles of the draft and judge models, which allows for the calculation of the likelihood of accepting the next token based on distinct probability distributions. To achieve personalized alignment efficiently, existing works utilize reward signals to guide Chen et al. (2024e) , linearly combine the predictive distributions of different base models to generate the next token ( Shi et al., 2024 ) , or Yang et al. (2024a) train an external corrector to avoid direct optimization of the policy model parameters. However, these approaches still rely on external feedback and preferences pre-defined during the training phase, which restricts their ability to generalize to unseen preferences. Furthermore, despite enhancing the efficiency of personalized alignment, existing methods still require training additional models. In this paper, we introduce Persona-judge (Figure 1 (c)), a generalizable and training-free personalized alignment framework. Persona-judge leverages the intrinsic ability of the model to discern preferences by directly employing existing LLMs as judges, thus eliminating the need for external reward signals. Inspired by speculative decoding ( Leviathan et al., 2023 ) , Persona-judge adopts a draft-and-judge pipeline, where the same base model serves both as the draft model and the judge model. Specifically, Persona-judge interprets the preference representation of the output as the acceptance of the predicted token. Different preferences are used to prompt the same base model, with the base model taking turns serving as the draft model to generate a sequence, the judge model to verify whether the predicted tokens are accepted. Experimental results demonstrate that Persona-judge, by exclusively utilizing the model’s intrinsic preference judgment capabilities, achieves performance comparable to training-based methods in a completely training-free setting. Furthermore, it exhibits remarkable generalization ability across diverse and unique preferences, highlighting its exceptional scalability. Algorithm M M P P G G T T F F Helpful Harmless Average Armo RM GPT-4 Armo RM GPT-4 Base 0.61 1.06 - 0.97 0.83 - 0.87 MORLHF ( Li et al., 2020 ) ✔ 0.31 0.91 14% 0.88 0.84 4% 0.73 MODPO ( Zhou et al., 2024 ) ✔ ✔ 0.56 0.89 52% 0.96 0.77 80% 0.80 Personalized soups ( Jang et al., 2023 ) ✔ ✔ 0.38 -0.72 72% 0.92 0.73 92% 0.33 Rewarded soups ( Rame et al., 2024 ) ✔ ✔ 0.50 0.87 34% 0.95 0.87 64% 0.80 RiC ( Yang et al., 2024b ) ✔ ✔ ✔ 0.54 0.90 40% 0.97 0.90 70% 0.83 MetaAligner ( Yang et al., 2024a ) ✔ ✔ ✔ ✔ 0.55 1.39 66% 0.89 0.54 74% 0.84 MOD ( Shi et al., 2024 ) ✔ ✔ ✔ 0.55 0.93 60% 0.96 0.92 84% 0.84 Aligner ( Ji et al., 2024 ) ✔ ✔ 0.67 1.32 72% 0.97 0.63 70% 0.90 Steering ( Konen et al., 2024 ) ✔ 0.63 1.17 38% 0.96 0.81 66% 0.89 Args ( Khanov et al., 2024 ) ✔ ✔ ✔ - 1.09 74% - 0.98 94% - Persona-judge-Base ✔ ✔ ✔ ✔ ✔ 0.59 1.21 80% 1.01 0.79 88% 0.90 Persona-judge 0.63 1.29 84% 0.98 0.82 98% 0.93 Table 1: Experiments of predefined preferences on Psoups dataset. M M ulti-objective indicates support for simultaneous alignment of multiple objectives, P P olicy-agnostic means the algorithm is independent of the model parameters, G G eneralizability denotes zero-shot alignment capability on unseen objectives, T T raining-free represents no need for additional training. F F ree from RM means not relying on external reward signals. "–" indicates not applicable. The best and second best results are highlighted in bold and underline . 2 Persona-judge Inspired by related works in LLM-as-a-Judge (Appendix B.2 ), we aim to effectively implement such judgment-like behavior without introducing additional external training while retaining the advantages of prompt-based approaches, to ensure fast, efficient, and scalable alignment. 2.1 Preferences Based Embeddings Traditional prompt-based methods achieve alignment by directly incorporating preferences descriptions, which are straightforward and effective ( Jang et al., 2023 ) . However, the process by which the model balances alignment across different preferences remains opaque, resulting in a lack of interpretability. When given \text{prefix}_{0} as input to LLM, with preferences P_{\mathrm{A}} and P_{\mathrm{B}} to align, the prompts related to P_{\mathrm{A}} and P_{\mathrm{B}} can be represented as \text{prefix}_{\mathrm{A}} and \text{prefix}_{\mathrm{B}} , respectively. Additional details are provided in Appendix C.1 . 2.2 Inference How to make judgments on the predicted token in a manner akin to preference-based decision-making? In essence, this involves evaluating the overlap between the probability distributions of different preference outputs. Given the descriptions of preferences P_{\mathrm{A}} and P_{\mathrm{B}} encoded by the language model as \text{prefix}_{\mathrm{A}} and \text{prefix}_{\mathrm{B}} , along with the original input \text{prefix}_{0} , in the i=0 iteration, the model designated as the draft model, representing the P_{\mathrm{A}} preferred