Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models

Konstantina Palla, José Luis Redondo García, Claudia Hauff, Francesco Fabbri, Henrik Lindström, Daniel R. Taber, Andreas Damianou, Mounia Lalmas · 2025

Read Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models by Konstantina Palla, José Luis Redondo García, Claudia Hauff, Francesco Fabbri, Henrik Lindström, Daniel R. Taber, Andreas Damianou, Mounia Lalmas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18695.txt.

Introduction

Konstantina Palla and seven co-authors rethink content moderation with large language models in a 2025 paper. Moderation keeps online spaces safe, traditionally by turning written policies into guidelines for human reviewers or training data for classifiers. Language models can now read policy text directly and enforce it, making moderation adjustable through plain wording changes.

The authors formalize this policy as prompt approach and map five challenges. Two are technical: translating policy into prompts and sensitivity to prompt format. Others concern who shapes policy when models do the enforcing, shifting team roles, and governance of the models themselves. Mitigation ideas are discussed for each. The paper aims to guide practitioners through the shift.

Catalog record

CREATORKonstantina Palla, José Luis Redondo García, Claudia Hauff, Francesco Fabbri, Henrik Lindström, Daniel R. Taber, Andreas Damianou, Mounia Lalmas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18695

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models Konstantina Palla, José Luis Redondo García, Claudia Hauff, Francesco Fabbri, Henrik Lindström, Daniel R. Taber, Andreas Damianou, Mounia Lalmas Abstract . Content moderation plays a critical role in shaping safe and inclusive online environments, balancing platform standards, user expectations, and regulatory frameworks. Traditionally, this process involves operationalising policies into guidelines, which are then used by downstream human moderators for enforcement, or to further annotate datasets for training machine learning moderation models. However, recent advancements in large language models (LLMs) are transforming this landscape. These models can now interpret policies directly as textual inputs, eliminating the need for extensive data curation. This approach offers unprecedented flexibility, as moderation can be dynamically adjusted through natural language interactions. This paradigm shift raises important questions about how policies are operationalized and the implications for content moderation practices. In this paper, we formalise the emerging policy-as-prompt framework and identify five key challenges across four domains: Technical Implementation (1. translating policy to prompts, 2. sensitivity to prompt structure and formatting), Sociotechnical (3. the risk of technological determinism in policy formation), Organisational (4. evolving roles between policy and machine learning teams), and Governance (5. model governance and accountability). Through analysing these challenges across technical, sociotechnical, organisational, and governance dimensions, we discuss potential mitigation approaches. This research provides actionable insights for practitioners and lays the groundwork for future exploration of scalable and adaptive content moderation systems in digital ecosystems. 1. Introduction Content moderation involves the systematic monitoring and regulation of user-generated content in online platforms. This critical practice fosters safe, inclusive, and respectful digital spaces while aligning with the values of hosting organisations and adhering to regulatory requirements. As digital ecosystems grow and content creation surges, the demand for scalable and adaptive moderation systems has become increasingly urgent. Online platforms now face the dual challenge of managing vast, dynamic content streams and meeting evolving societal and regulatory expectations. With the internet continuing to transform how we access information, entertainment, and services, content moderation has become essential for effectively managing online platforms, bridging local and global perspectives to address societal needs and foster community trust. The architecture of moderation pipelines varies significantly across digital services, reflecting their unique operational needs. For instance, video streaming platforms often prioritise automated detection of copyrighted or harmful content, while community forums like Reddit ( Reddit, Inc., 2025 ) emphasise moderating user interactions and discussions, often requiring human oversight alongside automated tools ( Gorwa et al., 2020 ) . Despite these platform-specific differences, successful content moderation fundamentally relies on two interconnected processes: operationalisation and policy enforcement ( Singhal et al., 2023 ) . Operationalisation involves transforming abstract policy objectives into actionable protocols for consistent implementation by human moderators and algorithms. This process includes several critical components: developing comprehensive annotation guidelines, curating high-quality training datasets, setting precise model predictions thresholds, and conducting thorough annotator training. Through operationalisation, abstract policies evolve into specific rules supported by robust workflows and systems. Content moderation policies are implemented through various documented forms, which we refer to as policy artifacts. These artifacts operate at different levels of granularity within an organization to cover its moderation needs. At the highest level, platform-wide rules establish foundational principles that guide moderation strategies. These principles then inform more granular artifacts, such as product-specific policies and detailed annotation guidelines, which address domain-specific requirements. This hierarchical structure ensures policies remain broadly applicable while allowing contextual adaptability. Enforcement puts these artifacts into practice, guiding decision-making processes for both human reviewers and algorithmic systems. During enforcement, content is systematically evaluated against context-appropriate policy artifacts, translating the groundwork laid during operationalisation into concrete moderation actions. Operationalisation provides the foundation for consistent, clear enforcement and works in tandem with the enforcement process. The interplay between operationalisation and enforcement is dynamic and interdependent, with changes or inconsistencies in one often impacting the other. Policy artifacts are regularly updated to address emerging content scenarios, refine enforcement nuances, or establish additional boundaries. For example, the appearance of new content types or edge cases may require adjustments to annotation guidelines, ensuring both moderators and algorithms remain aligned with overarching principles. Technology plays a crucial role in advancing both operationalisation and enforcement, enabling these processes to adapt and remain effective in response to evolving challenges. As the exponential growth of content volumes has driven unprecedented demands on moderation systems, significant technological advancements have emerged to meet these needs. Platforms increasingly rely on automated solutions, with machine learning algorithms becoming central to the moderation pipeline ( Prem and Krenn, 2024 ) . This work focuses on a transformative development in AI-assisted enforcement: the ability to encode content moderation policies directly as natural language prompts in LLMs, a paradigm we term “policy-as-prompt” and show in Figure 1b . By eliminating the need for extensive manual annotation pipelines, this approach offers unparalleled flexibility and scalability for moderation systems, enabling rapid policy iteration and fine-grained control over enforcement decisions. This emerging approach is gaining significant traction - enforcement strategies continue to vary across platforms, but the adoption of AI-driven methods, particularly those leveraging LLMs with injectable guidelines, is rapidly redefining the content moderation landscape ( Inan et al., 2023 ; Markov et al., 2023 ) . To the best of our knowledge, this work presents the first comprehensive exploration of the policy-as-prompt paradigm, examining its key challenges and implications across technical, sociotechnical, organisational, and regulatory domains. Beyond identifying these challenges, we systematically analyse their interdependencies and implications, providing a foundation for further inquiry into this transformative approach. In Section 2 we examine the technological evolution of content moderation, tracing its progression from basic pattern matching to modern LLM-driven approaches. We explore how this progression has fundamentally altered moderation architectures and introduced the emerging policy-as-prompt. In Sections 3 to 6 we break down five key challenges across four domains as outlined in Table 1 ; Technical Implementation (focusing on the challenges of 1. converting policies to prompts and 2. addressing prompt structure and format sensitivity, which we support with empirical findings), Sociotechnical (3. highlighting the impact of technological determinism in policy formation), Organisational (4. examining the evolving roles of policy-ML teams), and Governance (considering the 5. model governance and accountability). We provide a list of recommendations towards mitigating the challenges in Section 7 . We discuss the limitations of this work along with concluding remarks in Section 8 . Table 1 . Challenges across different areas in the ‘policy-as-prompt’ implementation. Area & Challenges Technical Implementation 1 Converting policies to prompts 2 Prompt structure and format sensitivity Sociotechnical 3 Technological determinism in policy formation Organizational 4 Evolving policy-ML team roles Governance 5 Model governance and accountability 2. The Evolution of Content Moderation Technology The evolution of content moderation, particularly within the text domain, has been closely tied to the exponential growth of user-generated content, driven by the rapid advancement of algorithmic capabilities. Early moderation efforts relied on basic automation, functioning like a digital “find and replace” tool. Methods such as keyword filters and hash-matching were used to identify and remove prohibited content ( Engstrom and Feamster, 2017 ) . While these first-generation tools were effective at detecting exact matches, they were easily circumvented by minor alterations, underscoring their limitations in

Read the complete text