Understanding Annotator Safety Policy with Interpretability \conffull ( \confshort ), \confdate , \confloc Conference: The 2026 ACM Conference on Fairness, Accountability, and Transparency; June 25–28, 2026; Montreal, QC, Canada The 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26), June 25–28, 2026, Montreal, QC, Canada DOI: 10.1145/3805689.3806472 ISBN: 979-8-4007-2596-8/2026/06 CCS: Human-centered computing CCS: Computing methodologies Machine learning CCS: Computing methodologies Artificial intelligence

Alex Oesterling, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys, Fred Hohman · 2026

Read Understanding Annotator Safety Policy with Interpretability \conffull ( \confshort ), \confdate , \confloc Conference: The 2026 ACM Conference on Fairness, Accountability, and Transparency; June 25–28, 2026; Montreal, QC, Canada The 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26), June 25–28, 2026, Montreal, QC, Canada DOI: 10.1145/3805689.3806472 ISBN: 979-8-4007-2596-8/2026/06 CCS: Human-centered computing CCS: Computing methodologies Machine learning CCS: Computing methodologies Artificial intelligence by Alex Oesterling, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys, Fred Hohman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05329.txt.

Introduction

Alex Oesterling and six co-authors make the hidden safety policies of data raters visible with clear models learned from tagging choices alone. Rater disagreement can come from task mistakes, vague policy wording, or truly different values, and each cause needs a distinct fix. Asking raters to explain their choices is costly and often shaky, for humans and model raters alike.

Their Annotator Policy Models recover each annotator's internal safety policy without extra effort. Checks show the models match rater policies well and predict how raters respond to what-if edits of examples. The tool lets policy teams compare thinking across raters and target clearer wording, quality control, or debate where each belongs.

Catalog record

CREATORAlex Oesterling, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys, Fred Hohman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05329

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Annotator Safety Policy with Interpretability \conffull ( \confshort ), \confdate , \confloc Conference: The 2026 ACM Conference on Fairness, Accountability, and Transparency; June 25–28, 2026; Montreal, QC, Canada The 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26), June 25–28, 2026, Montreal, QC, Canada DOI: 10.1145/3805689.3806472 ISBN: 979-8-4007-2596-8/2026/06 CCS: Human-centered computing CCS: Computing methodologies Machine learning CCS: Computing methodologies Artificial intelligence Alex Oesterling, Donghao Ren, Yannick Assogba, Dominik Moritz, Sunnie S. Y. Kim, Leon Gatys, Fred Hohman Abstract . Safety policies define what constitutes safe and unsafe AI outputs, guiding data annotation and model development. However, annotation disagreement is pervasive and can stem from multiple sources such as operational failures (annotators misunderstand or misexecute the task), policy ambiguity (policy wording leaves room for interpretation), or value pluralism (different annotators hold different perspectives on safety). Distinguishing these sources matters. For example, operational failures call for quality control, ambiguity calls for policy clarification, and pluralism calls for deliberation about incorporating diverse perspectives. Yet understanding why annotators disagree is difficult. Directly asking annotators for their reasoning is costly, substantially increasing annotation burden, and can be unreliable for both human and LLM annotators as self-reported reasoning often fails to reflect actual decision processes. We introduce Annotator Policy Models (APMs), interpretable models that learn annotators’ internal safety policies from labeling behavior alone, making annotator reasoning visible and comparable without additional annotation effort. We validate that APMs accurately model annotator safety policy (>80% accuracy), faithfully predict responses to counterfactual edits, and recover known policy differences in controlled settings. Applying APMs to LLM and human annotations, we demonstrate two core applications: (1) surfacing policy ambiguity by revealing how annotators interpret safety instructions differently, and (2) surfacing value pluralism by uncovering systematic differences in safety priorities across demographic groups. Together, these capabilities support more targeted, transparent, and inclusive safety policy design. 1 1 1 Code is available at https://github.com/apple/ml-annotator-policy-models Figure 1 . Annotator Policy Models (APMs) learn interpretable representations of individual annotator safety policies. APMs are trained on annotation behavior to reveal how different annotators operationalize safety, enabling diagnosis of disagreement sources. By mapping annotators to a shared feature space, APMs make systematic comparison possible: identifying where annotators may have misunderstood the task itself (identifying operational failures), where they interpret instructions differently (surfacing policy ambiguity), or where they systematically differ by demographic group (surfacing value pluralism). † † cc-license: by-nc-nd 1. Introduction To operationalize safety in AI systems, developers create safety policies, constitutions ( 19 ) , or model specs ( 56 ) that explicitly define what constitutes harmful output. Data annotation based on these policies critically impacts all stages of model development. Human and LLM annotations label the training data ( 39 ; 6 ) , provide reward signals during training ( 57 ) , and enable model evaluations ( 18 ) . The quality of these annotations directly shapes model behavior and development decisions. Yet annotator disagreement is pervasive, with both human and LLM annotators frequently diverging in their judgments of whether a given output is safe or unsafe ( 4 ; 20 ; 10 ; 44 ; 53 ) . Disagreement can stem from multiple sources. Operational failures occur when annotators misunderstand or misexecute the annotation task. For example, many safety datasets contain both user prompts and AI responses but provide only a single binary label ( 39 ; 28 ) . Even when task instructions explicitly ask annotators to label only the model response, annotators may conflate the two and label based on the full interaction, or vice versa. Policy ambiguity arises when safety policy wording leaves room for interpretation. Safety, like many natural language tasks, struggles with subjectivity–a well-documented challenge in related domains such as hate speech ( 69 ; 25 ; 64 ; 22 ; 40 ) , toxicity ( 65 ; 66 ) , and harassment detection ( 2 ) . Ambiguous policy language can lead to inconsistent application. For example, a policy prohibiting "offensive language" fails to clarify whether quoting such language for educational purposes constitutes a violation. Finally, value pluralism reflects the reality that different annotators may hold genuinely different views on what constitutes harm, shaped by their cultural backgrounds ( 34 ) , personal identities ( 65 ; 66 ; 47 ) , and values ( 60 ; 26 ; 27 ) . We note that the boundary between ambiguity and pluralism can often be blurry as clarifying ambiguous policy often requires making normative choices, and what appears as ambiguity may reveal underlying value disagreements. For example, a policy requiring outputs to "encourage freedom, equality, and a sense of brotherhood" ( 19 ) is ambiguous in its wording, but any clarification would require normative choices about whose understanding of these values takes precedence and reveal the underlying pluralism. Distinguishing the sources of annotation disagreement matters because appropriate responses differ. Operational failures call for improved annotator training and quality control. Ambiguity calls for policy clarification and refinement. Pluralism calls for deliberation about how to accommodate diverse perspectives. Current practice, which typically resolves disagreement through majority vote ( 39 ; 29 ; 17 ) , conflates these sources, and often fails to address genuine ambiguity while potentially silencing minority perspectives. Thus, the central question we address is: how can we understand individual annotator reasoning to distinguish these sources of disagreement and respond appropriately? To distinguish these sources of disagreement, we need to understand annotators’ interpretations of safety guidelines and annotator safety policy , or the implicit rules by which an annotator labels data. One such approach is asking annotators directly to explain their reasoning. However, collecting reasoning traces substantially increases annotation burden and cost. Moreover, self-reported explanations can be unreliable: psychology research has established that people often cannot accurately report on their decision processes, and their explanations may reflect post-hoc rationalization rather than true reasoning ( 54 ) . For LLM annotators, the faithfulness of chain-of-thought reasoning is similarly questionable ( 70 ; 46 ) . Additionally, the resulting natural language explanations are unstructured and difficult to compare systematically across annotators. We take an alternative approach: rather than asking annotators to explain their reasoning, we learn their internal safety policies from their labeling behavior. We introduce Annotator Policy Models (APMs) , interpretable models that approximate each annotator’s safety policy (see Figure 1 ). APMs are trained on existing annotation data, requiring no additional annotator effort. By expressing annotator reasoning in a shared, interpretable feature space, APMs make individual perspectives visible and directly comparable. While we focus on safety applications in this work, APMs can be used to systematically study annotation in subjective tasks more generally. APMs enable two core applications. First, they can surface operational failures and policy ambiguity by revealing where annotators interpret safety instructions differently. This understanding enables targeted policy refinement. Second, they can surface value pluralism by uncovering systematic differences in how demographic groups prioritize safety concerns and which might otherwise be obscured by majority aggregation. Together, these capabilities support more targeted, transparent, and inclusive safety policy design. Our contributions are as follows: • Interpretable models for annotator policy. We introduce Annotator Policy Models (APMs) to explicitly model individual annotator safety policy using techniques from ML interpretability. • Extensive validation. We demonstrate that APMs accurately

Read the complete text