Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values

Henry Bell, Lara Neubauer da Costa Schertel, Bochu Ding, Brandon Fain · 2026

Read Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values by Henry Bell, Lara Neubauer da Costa Schertel, Bochu Ding, Brandon Fain (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18760.txt.

Introduction

Henry Bell and three colleagues argue that the rules used to train language models should rest on why people hold their views, not just on which reply they pick. In the main path now, a model is taught to match a set of writ rules. Those rules are set by a small team. It is not clear how the rules could be set in a fair way with wide voice from all who are touched by the tools.

Picks alone hide the grounds of a view. Two folk may pick the same reply for quite unlike grounds. A rule built on picks may thus miss what folk prize most. The team set out a path that learns rules from grounds and worths. Such rules could claim a fairer base. They could also be checked in the open, ground by ground.

Catalog record

CREATORHenry Bell, Lara Neubauer da Costa Schertel, Bochu Ding, Brandon Fain
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18760

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Preferences: Learning Alignment Principles Grounded in Human Reasons and Values Henry Bell, Lara Neubauer da Costa Schertel, Bochu Ding, Brandon Fain Abstract A crucial consideration when developing and deploying Large Language Models (LLMs) is the human values to which these models are aligned. In the constitutional framework of alignment models are aligned to a set of principles (the constitution) specified in natural language. However, it is unclear how to fairly determine this constitution with widespread stakeholder input. In this work we propose Grounded Constitutional AI (GCAI), a unified framework for generating constitutions of principles that are representative of both users’ general expectations toward AI (general principles) and their interaction-time preferences (contextual principles). We extend the Inverse Constitutional AI (ICAI) approach to generate contextual principles from human preference annotation data by leveraging human-provided reasons for their preferences. We supplement these contextual principles with general principles surfaced from user statements of values regarding AI. We show that a constitution generated by GCAI is preferred by humans over one generated through ICAI both personally, and for widespread use in governing AI behavior. Additionally participants consider the GCAI constitution to be more morally grounded, coherent, and pluralistic. 1. Introduction Aligning Large Language Models (LLMs) to human values has emerged as a critical challenge of modern ethical AI ( 1 ; 16 ; 7 ; 2 ; 23 ) . Constitutional alignment techniques ( 5 ; 22 ; 12 ; 11 ; 26 ) align large language models (LLMs) to constitutions of principles specified in natural language (such as “respect human autonomy” or “do not use offensive language”). A natural question is which principles should we use? There is disagreement over values for alignment in a pluralistic society where LLMs are deployed in diverse contexts ( 42 ; 34 ; 17 ; 35 ; 9 ) . Rather than argue for a particular set of principles, we consider the question of a fair process for eliciting principles ( 15 ) from stakeholder input ( 13 ; 18 ; 38 ; 22 ) . Deliberation ( 22 ) and social choice ( 9 ) have been proposed as possibilities for determining constitutions, but both methods require ways to elicit initial candidate principles–whether for seeding deliberation, voting, or both. A critical challenge is scalability: How can we incorporate the input of potentially millions of stakeholders and distill their perspectives down to a reasonable number of candidate principles? Individuals may express thousands of diverse views, general or specific, redundant, conflicting, or overlapping. To address this need, we propose GCAI : Grounded Constitutional AI , a unified learning framework to elicit candidate alignment principles from stakeholder input. By learning we do not mean data collection. Rather, GCAI is a scalable unsupervised machine learning framework that can synthesize large volumes of human preference annotations with reasons and general human descriptions of AI alignment values into a compact set of representative candidate principles (a constitution). Prior work has similarly attempted to learn principles purely from human preference annotations, a method coined as Inverse Constitutional AI (ICAI) ( 11 ) . We surface the need of further grounding principle learning using: (i) reasons (or justifications) for human preference annotations, written in natural language, and (ii) general values humans report for LLM alignment, also written in natural language. At a high level, GCAI works in four main steps, the details of which vary slightly between (i) and (ii) and are specified in Section 4 with full prompts available in the Appendix. (1) Candidate Generation. Using an LLM, we automatically rephrase individual reasons for preference annotations or general statements of values into one or more correctly formatted and generally applicable principles. This step is necessary as many human-written reasons or values are too narrow and specific (e.g., referring to a particular prompt), involve many semantically distinct concepts, or are simply written informally and idiomatically. (2) Clustering. We embed each candidate principle as a numerical vector using a sequence embedding model. We then cluster the resulting embedded candidates. Clustering serves two purposes: Deduplication of many repeated candidates and identification of broadly relevant themes. (3) Summarization. Using an LLM, we automatically summarize the most central candidates within each cluster (which often include somewhat different wordings of similar concepts) as new candidate principles. (4) Scoring and Selection. The reduced set of summarized candidates are scored, either by their ability to predict relevant preferences (for preference annotations with reasons) or by their centrality within the embedding space as a soft proxy for consensus (for general statements of values). The top scoring candidates (up to the desired number) are selected. Outline and Contributions (1) Section 2 characterizes two kinds of principles: contextual and general. We argue that contextual principles should be grounded in reasons or justifications, and that general principles are also necessary to faithfully capture relevant value and concerns about alignment. (2) Section 3 surveys related work on alignment, principle elicitation, and evaluation criteria for constitutions. (3) Section 4 presents Grounded Constitutional AI (GCAI), the first unified framework for eliciting a constitution grounded in contextual and general human reasons and values written in natural language. (4) Section 5 discusses our evaluation methodology, including a discussion of the novel evaluation criteria we propose for constitutions. (5) Section 6 presents our main empirical results evaluating a constitution learned by GCAI compared to ICAI. Human survey participants overwhelmingly prefer the GCAI constitution across multiple evaluation dimensions, and we investigate the reasons for these preferences. (6) Section 7 presents qualitative and benchmark results of a model that we align to our the constitution learned by GCAI in comparison to that learned by ICAI. While the constitutions make little difference in benchmark performance, qualitatively we find that there are consistent thematic differences between the resulting models, with the GCAI-aligned model showing much more emphasis on ethics, safety, and non-discrimination. (7) Section 8 concludes with reflections on constitutional elicitation broadly based on our findings. 2. Grounding in Human Reasons and Values: Contextual and General Principles We distinguish between two kinds of alignment principles the differ in their origin: • Contextual principles arise from actual interaction experience with models. They may depend heavily on the prompt distribution, pretraining, and fine-tuning of the particular model with which a human interacts. • General principles arise from anticipated concerns about hypothetical or possible model behavior. They represent prior commitments and do not depend on particular observed behaviors of a given LLM but rather on the broad values and concerns about possible LLM use. Inverse Constitutional AI (ICAI) ( 11 ) tries to learn constitutional principles directly from preference annotations, a characteristic source for contextual principles. A human preference annotation includes a prompt, two example responses, and a selection of the preferred response. However, we emphasize that principles are underdetermined by preference annotations , in the sense that a given preference annotation might be grounded in multiple unrelated or even conflicting reasons for that preference. For example, consider the following preference annotation drawn from the Helpsteer 2 Dataset ( 41 ) . Prompt User: I have a flurk in my glooben! Response 1 LLM: I’m sorry, I don’t speak whatever language that is. Can you please rephrase your concern in English? I’m here to help! Response 2 (preferred

Read the complete text