PluralLLM: Pluralistic Alignment in LLMs via Federated Learning DOI: XXXXXXX.XXXXXXX Conference: ACM Conference; 2025; CCS: Security and privacy Human and societal aspects of security and privacy CCS: Computing methodologies Machine learning CCS: Computing methodologies Natural language generation
Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki · 2025
Read PluralLLM: Pluralistic Alignment in LLMs via Federated Learning DOI: XXXXXXX.XXXXXXX Conference: ACM Conference; 2025; CCS: Security and privacy Human and societal aspects of security and privacy CCS: Computing methodologies Machine learning CCS: Computing methodologies Natural language generation by Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-09925.txt.
Introduction
Mahmoud Srewa, Tianyu Zhao, and Salma Elmalaki align language models with many groups' values at once, without pooling their private feedback, in a 2025 paper. Standard alignment gathers everyone's preferences in one place, which is costly and invasive. Their system, PluralLLM, lets separate user groups train a shared preference predictor from their own sites.
Using federated averaging, the groups exchange model updates rather than data. The federated predictor converged 46 percent faster than the centralized comparison, scored four percent better on alignment, and matched it on fairness between groups. The predictor can then serve as the reward model that steers a language model. The authors present federated preference learning as a scalable, privacy-preserving route to pluralistic alignment.
Catalog record
| CREATOR | Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-09925
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PluralLLM: Pluralistic Alignment in LLMs via Federated Learning DOI: XXXXXXX.XXXXXXX Conference: ACM Conference; 2025; CCS: Security and privacy Human and societal aspects of security and privacy CCS: Computing methodologies Machine learning CCS: Computing methodologies Natural language generation Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki Abstract . Ensuring llms ( llms ) align with diverse human preferences while preserving privacy and fairness remains a challenge. Existing methods, such as rlhf ( rlhf ), rely on centralized data collection, making them computationally expensive and privacy-invasive. We introduce PluralLLM 1 1 1 The code will be released. a federated learning-based approach that enables multiple user groups to collaboratively train a transformer-based preference predictor without sharing sensitive data, which can also serve as a reward model for aligning LLMs. Our method leverages Federated Averaging (FedAvg) to aggregate preference updates efficiently, achieving 46 % 46\% faster convergence, a 4 % 4\% improvement in alignment scores, and nearly the same group fairness measure as in centralized training. Evaluated on a Q/A preference alignment task, PluralLLM demonstrates that federated preference learning offers a scalable and privacy-preserving alternative for aligning LLMs with diverse human values. LLMs Large Language Models LLM Large Language Model FL Federated Learning GPO Group Preference Optimization RLHF Reinforcement Learning from Human Feedback DPO Direct Preference Optimization 1. Introduction llms have rapidly emerged as a cornerstone of modern artificial intelligence, powering applications ranging from conversational agents to content generation and decision support systems ( Bubeck et al. 2023 ) . Their ability to generate human-like text has revolutionized various industries, but their effectiveness depends on their ability to align with human values and societal expectations ( Zhang et al. 2023 ) . However, achieving robust human alignment remains a significant challenge, particularly in ensuring that these models fairly represent diverse perspectives, a concept known as Pluralistic Alignment ( Sorensen et al. 2024 ) . Existing LLM alignment methods fall into two categories: prompt engineering and gradient-based alignment ( Sahoo et al. 2024 ; Zhang et al. 2023 ) . While prompt engineering guides model behavior through carefully crafted prompts and in-context examples without modifying model parameters, it often struggles with complex behaviors ( Zhou et al. 2022 ) . Gradient-based alignment fine-tunes models using reward mechanisms, such as Reinforcement Learning from Human Feedback ( rlhf ), improving traits like honesty and helpfulness but requiring extensive supervision and high computational costs ( Ouyang et al. 2022 ) . However, existing approaches do not scale efficiently for aligning LLMs across multiple user groups with limited supervision, making pluralistic alignment challenging ( Chakraborty et al. 2024 ; Casper et al. 2023 ) . Despite recent advancements in alignment techniques, preference learning for llm ( llm ) faces three significant challenges: (1) privacy risks, (2) collection of preference data, and (3) computational overhead ( Kuang et al. 2024 ; Casper et al. 2023 ) . Centralized approaches, such as rlhf and gradient-based alignment, require collecting and processing user interactions, raising concerns about data security and user confidentiality. On the other hand, fl ( fl ) enables privacy-preserving model training by keeping data decentralized. However, it comes with high communication and processing costs due to frequent updates between clients and global server ( Kuang et al. 2024 ) . These challenges are even greater for pre-trained llm , which require significant computational resources to incorporate preference data. Aligning llm with FL adds further complexity, demanding a balance between alignment quality, efficiency, and privacy. In preference learning, this burden is particularly heavy, making efficient aggregation strategies or alternative models essential ( Wu et al. 2024 ) . In this paper, we address the following research question: How can we align LLMs to capture the preference of various perspectives of different communities while preserving privacy, maintaining fairness, and ensuring computational efficiency? We introduce PluralLLM , a framework for pluralistic alignment in LLMs via FL. Our approach leverages fl to train a transformer-based preference predictor ( Zhao et al. 2023 ) to capture group-specific preferences in a distributed privacy-preserving manner. This preference predictor serves as a lightweight alternative to conventional alignment methods that require training a reward model rlhf , significantly reducing computational overhead and can adapt to a new unseen group. Unlike the centralized group preference optimization training approach proposed by Zhao et. al. ( Zhao et al. 2023 ) , our FL-based preference predictor ensures privacy-preserving preference learning by allowing different groups to collaboratively train the model without exposing their sensitive preference data. In addition, it enables diverse groups to participate in training while maintaining the fairness properties of the centralized approach. Our results demonstrate that our proposed PluralLLM approach applied to preference learning achieves higher alignment scores and faster convergence compared to centralized methods, making it a scalable and efficient solution for capturing diverse group preferences. Figure 1 . PluralLLM : Pluralistic alignment in LLMs via Federated Learning. 2. Related Work Prompt Engineering : Prompt engineering provides a mechanism for fine-tuning model outputs through the modification of the input to the llm , thereby aligning with user preferences without altering the parameters of the core llm model. Prompt engineering approaches are characterized by their computational efficiency, a property that stems from the absence of any training requirements ( Sahoo et al. 2024 ) . However, the design of the prompt itself can be a laborious task that relies on heuristics. The efficacy of these heuristics is not guaranteed to transfer well across different LLMs ( Zhao et al. 2023 ) . Recent work in the literature has shown that prompt engineering has limited success in aligning LLMs to complex groups on challenging survey datasets such as GlobalOpinionQA ( Durmus et al. 2023a ) . Pluralistic Alignment in LLM: A growing number of pluralistic alignment studies show that it is important to design llm systems that can accommodate and represent diverse human values, perspectives, and preferences. Unlike traditional alignment approaches that aim to align models to a single, averaged set of human preferences, pluralistic alignment seeks to reflect the complexity and plurality of human societies. For example, Cao et. al. introduced an age fairness reward in llm to reduce response quality disparities across distinct age groups during training ( Cao et al. 2024 ) . Traditional Reinforcement Learning alignment approaches, such as rlhf often reinforce majority viewpoints while marginalizing minority perspectives. The question of balancing openness to diverse values with ethical constraints, such as the avoidance of harmful ideologies, remains largely unaddressed. Group Preference Alignment: Group preference alignment refers to techniques designed to adapt llm outputs to reflect the distinct preferences, values, or judgments of different groups or demographics. gpo ( gpo ) ( Zhao et al. 2023 ) was introduced as a few-shot alignment framework that steers LLMs toward group-specific preferences. gpo augments the base LLM with an independent transformer module, trained via in-context supervised learning with only a handful of samples to predict group preferences and refine model outputs. This module acts as a preference model for different groups, learning distinct alignment patterns across diverse communities. By leveraging an in-context autoregressive transformer, gpo enables flexible and efficient alignment, allowing LLMs to adapt dynamically to varying user preferences. 3. Pluralistic Alignment in LLMs via Federated Learning We chose the Q/A preference