Implicit Safety Alignment from Crowd Preferences
Qian Lin, Daniel S. Brown · 2026
Read Implicit Safety Alignment from Crowd Preferences by Qian Lin, Daniel S. Brown (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21822.txt.
Introduction
Qian Lin and Daniel Brown study implicit safety alignment from crowd preferences, where users express distinct goals yet share safety principles. Combining a preference-learned reward model with downstream task rewards has inherent limits, the authors show. Their Safe Crowd Preference-based RL framework extracts safety-aligned skills from crowd preferences and composes them via a policy for downstream tasks. Experiments across safe reinforcement-learning environments and a language-model-style task lower safety costs without explicit safety rewards, matching oracle methods trained with ground-truth safety signals.
Catalog record
| CREATOR | Qian Lin, Daniel S. Brown |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 119435 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.21822
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Implicit Safety Alignment from Crowd Preferences Qian Lin, Daniel S. Brown Abstract Reinforcement Learning from Human Feedback (RLHF) can reveal implicit objectives such as safety considerations that go beyond task completion. In this work, we focus on the common safety criteria embedded in crowd preference datasets, where different users may express distinct preferences or objectives, yet follow similar safety principles. Our aim is to discover shared safety criteria from crowd preferences and then transfer them to downstream RL tasks to regularize agent behavior and enforce safety. We first show that direct reward combination—optimizing a preference-learned reward model together with downstream task rewards—has inherent limitations. Motivated by this, we propose Safe Crowd Preference-based RL, a hierarchical framework that extracts safety-aligned skills from crowd preferences and composes them via a high-level policy to safely solve downstream tasks. Experiments across safe RL environments and a preliminary LLM-style task with diverse user goals and shared safety constraints demonstrate that our approach substantially lowers safety costs without access to explicit safety rewards, while achieving task performance comparable to oracle methods trained with ground-truth safety signals. 1 Introduction Reinforcement learning (RL) ( 49 ) with well-designed rewards that encode task-specific information enables agents to accomplish their goals. However, specifying reward functions that fully capture desired behavior is notoriously difficult, even for seemingly simple tasks, and imperfect or misspecified rewards are common in practice ( 7 ; 36 ) . In complex decision-making scenarios, reward designers often focus on task completion while inadvertently omitting additional objectives that are ambiguous, implicit, or hard to formalize—particularly safety-related considerations. For instance, in driving-related tasks, one can assign a sparse reward for simply reaching the destination, yet real-world driving also involves implicit safety considerations—such as maintaining safety margins and discouraging aggressive or risky behaviors—that are difficult to encode accurately through manual reward design. Optimizing such incomplete rewards can therefore lead to hazardous behavior, highlighting the limitations of purely hand-specified reward functions. Reinforcement Learning from Human Feedback (RLHF) has been developed to overcome the limitations of human-designed rewards and has been widely applied in large language models (LLMs) ( 40 ; 4 ) , robotics ( 22 ; 34 ) , and games ( 14 ; 9 ) . It enables the incorporation of implicit objective information into decision-making by learning new reward functions from scratch ( 14 ) , discovering additional objectives ( 15 ) , and directly modeling expected behaviors ( 2 ; 23 ) to align policies with human preferences. In practice, preference data are often collected from a crowd of users. While individual users may vary in their task objectives, preferences, or behavioral patterns, they often share implicit criteria—such as avoiding unsafe states and actions. This shared structure suggests the possibility of isolating latent behavioral criteria that are consistent across crowd preferences and reusing them across downstream tasks, thereby reducing the need for additional human demonstrations or preference labels. Accordingly, we focus on safety-critical scenarios under crowd preference learning, where safety-agnostic task rewards inadvertently promote unsafe behaviors, while crowd preferences—collected across users with diverse goals yet shared safety criteria—provide a complementary safety signal to compensate for such deficiencies. Our goal is to discover such reusable, implicit safety-related criteria embedded in crowd-user, cross-task preferences and transfer them to downstream tasks, where only safety-agnostic task rewards are available, to learn safe policies. A natural approach is to learn a reward model from crowd preferences via standard RLHF and then optimize a combination of this preference-learned reward and the downstream task reward. However, our theoretical and empirical analyses reveal two limitations of this reward combination approach: (1) the learned reward model may capture not only the intended shared criteria but also user-specific components from crowd preferences, which can degrade downstream performance—especially when the dataset is imbalanced across users; and (2) the performance is sensitive to the weighting between the preference-learned and task rewards, making the method difficult to tune. To overcome these limitations, we propose Safe Crowd Preference-based Reinforcement Learning, a hierarchical framework that composes policies rather than rewards. Specifically, we first model diverse user-preferred behaviors as low-level skills, learned from crowd preference data via a VAE-based latent skill model and existing RLHF techniques. For a new task, we then train a high-level policy to compose these skills by maximizing only the downstream task reward. Since all skills are aligned with crowd preferences and thus inherently encode shared safety criteria, composing them to solve new tasks naturally yields safe behaviors without requiring explicit safety signals. We evaluate our approach on a suite of safe RL environments and a proof-of-concept LLM-style task with diverse task goals but shared safety constraints. Empirically, directly optimizing task rewards leads to severe constraint violations, whereas our method significantly reduces safety costs without requiring explicit safety rewards and achieves task performance comparable to oracle baselines trained with ground-truth safety signals. We further find that reward combination (i.e., directly optimizing a weighted sum of the preference-learned reward and the new task reward) is highly sensitive to preference imbalance: performance drops sharply when one annotator contributes the majority of the data, while our method is significantly more robust to such imbalance. 2 Related Work 2.1 Reinforcement Learning from Human Feedback (RLHF) Reinforcement Learning from Human Feedback (RLHF) aims to improve decision making by aligning agents with various forms of human feedback, including preferences ( 52 ) , demonstrations ( 39 ) , trajectory rankings ( 9 ; 10 ) , corrections ( 5 ) , and terminations ( 21 ) . This work is most closely related to Preference-based RL (PbRL), where binary comparisons over trajectories are used to guide policy learning. Classical PbRL methods ( 31 ; 14 ) typically learn a reward model using the Bradley–Terry–Luce (BTL) model ( 8 ) , and then perform online RL or offline RL ( 46 ) based on the learned reward. Recently, several works ( 2 ; 23 ) have proposed directly learning a policy under the BTL model without an explicit reward learning stage, thereby avoiding optimization challenges inherent in RL. Both reward-based and direct preference learning approaches are compatible with our framework, and we evaluate both with online and offline RL in our experiments. 2.2 Crowd Preference-based RL Most existing RLHF methods assume that all comparisons are generated by users who share the same set of values and goals. However, in practice, human feedback often reflects diverse trade-offs among multiple objectives or even different decision goals, referred to as crowd preference data. Several studies on crowd preference learning have been proposed to promote fair aggregation of feedback from diverse sources ( 12 ; 11 ) , improve the robustness of reward models against user noise or jailbreaks ( 48 ; 6 ) , and enable personalized decision-making for individual users ( 24 ; 44 ) . Among them, 44 is most closely related to our work, as it also uses a VAE-based architecture to model behavioral diversity implied by crowd preferences. However, it focuses on identifying existing preferences to enhance decision diversity, while our work emphasizes behavioral generalization—combining diverse behaviors to solve new tasks. Furthermore, to our knowledge, we are the first to study shared objectives in crowd preferences and to explore how shared safety objectives can be