CounterMoral: Editing Morals in Language Models

Michael Ripa, Jim Davies · 2026

Read CounterMoral: Editing Morals in Language Models by Michael Ripa, Jim Davies (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27338.txt.

Introduction

Ripa and Davies introduce CounterMoral, a benchmark for editing moral judgments inside language models. Editing facts in models is now well studied, but editing values is not. CounterMoral tests how well current editing techniques shift moral judgments. The cases span several distinct ethical frameworks for breadth. The authors apply a range of editing methods to multiple language models. Performance is then measured on the moral edits themselves. Their findings feed the broader evaluation of ethically aligned models. The benchmark is offered as a tool for that alignment work.

Catalog record

CREATORMichael Ripa, Jim Davies
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58598
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27338

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CounterMoral: Editing Morals in Language Models Michael Ripa, Jim Davies Abstract Recent advancements in language model technology have significantly enhanced the ability to edit factual information. Yet, the modification of moral judgments, a crucial aspect of aligning models with human values, has garnered less attention. In this work, we introduce CounterMoral , a benchmark dataset crafted to assess how well current model editing techniques modify moral judgments across diverse ethical frameworks. We apply various editing techniques to multiple language models and evaluate their performance. Our findings contribute to the evaluation of language models designed to be ethical. 1 Introduction 1.1 Background Research on the development of ethical Artificial Intelligence (AI) systems has been ongoing for many decades 22 . From Asimov’s laws, to autonomous vehicles, the central discussion has revolved around how to embed ethical principles into AI systems to align them with human judgments. Despite the longstanding academic focus on this issue, it remains unclear how to achieve this goal effectively 13 . With the advent of deep neural networks, the focus of research has shifted towards analyzing the ethical beliefs learned from training data 30 ; 9 and evaluating how well pretrained models can apply morals in social reasoning 8 . Currently, efforts to align these beliefs primarily involve value alignment, with Reinforcement Learning from Human Feedback (RLHF) emerging as the most popular approach 6 . Despite its effectiveness, RLHF comes with notable limitations, including scalability challenges, inconsistencies in human feedback, and risks of reward hacking 4 . Recently, a new area of research has emerged, focusing on methods for editing existing pretrained deep learning models 2 ; 23 . Current research primarily targets editing factual information in language models 14 and highlights various use cases, such as detoxification 25 , bias mitigation 10 , and removing protected information 5 . However, minimal research has explored the feasibility of model editing as an approach to modifying moral beliefs. In this paper, we introduce CounterMoral , a dataset that can be used to evaluate the morality of language models. 1.2 Model Editing for Moral Judgments If effective, model editing could serve as a promising alternative to fine-tuning and RLHF, enabling efficient modifications to model behavior without requiring full retraining. This approach may offer several potential advantages for AI alignment: Adaptability : Model editing enables rapid adaptation to changing ethical norms and diverse cultural contexts through targeted updates. Unlike RLHF, which requires new feedback data and reward model updates for each moral shift, editing allows for customized moral frameworks and reuse of base models across different cultural contexts. Efficiency : Model editing requires fewer computational resources and less human input compared to RLHF’s extensive feedback collection and optimization processes. This efficiency enables applying multiple moral edits iteratively, making alignment more ecological and opening possibilities for diverse use cases. Focused Alignment : While RLHF’s black-box reward functions implicitly encode multiple values simultaneously, model editing provides surgical precision. This targeted approach modifies specific moral judgments without side effects in other knowledge domains, preserving the model’s broader capabilities. This paper introduces the CounterMoral benchmark dataset, designed to assess whether model editing techniques can effectively refine AI moral judgments to different ethical perspectives. Our contributions are twofold: 1. We present the CounterMoral benchmark, a resource for AI alignment and ethics researchers to assess how well models can be adjusted to align with diverse moral frameworks. 2. We conduct evaluations using five different techniques: three model editing methods, along with LoRA and full fine-tuning (FT) as baselines, to assess their effectiveness in adapting model behavior. 2 CounterMoral In this section, we outline both the motivation and the design of CounterMoral , our moral editing benchmark. Further information on its design, including exact prompts and tests of hypothesis can be found in Appendix A . 2.1 Overview and Purpose In response to the absence of specialized benchmarks for editing moral judgments, we introduce CounterMoral , a model editing benchmark dedicated to evaluating the modification of moral judgments within four ethical frameworks. Beyond introducing a new evaluation category for model editing, CounterMoral broadens the scope of model editing research by highlighting the feasibility and challenges of refining LLMs’ moral judgments through targeted interventions. Our work is inspired by CounterFact 15 , a leading benchmark dataset for assessing edits on factual information with relational triplets. (subject,\,relation,\,object) . CounterFact focuses on factual updates. For instance, altering a model’s belief from the Eiffel Tower being in “Paris” to “Rome.” In a parallel fashion, CounterMoral models moral judgments using the format (action,\,verb,\,judgment) , where action is described as a gerund to maintain its noun form, verb facilitates the relational aspect, and judgment encapsulates the moral evaluation. For instance, an edit template in our dataset might shape a model’s judgment according to virtue ethics, which emphasizes character evaluation, as seen in the following example: Action: ‘‘Confessing to a mistake’’ Verb Phrase: ‘‘demonstrates’’ Judgment (Original): ‘‘accountability’’ Judgment (Edited): ‘‘self-incrimination’’ 1 1 1 You can see more edit template examples at http://bit.ly/41yM5Yb CounterMoral comprises over such 1200 edit templates, each anchored to one of four ethical frameworks. The dataset is structured such that a set of broader action categories each serves as a foundation for generating ten more detailed, context-specific actions. 2.2 Dataset Generation Process We now describe the dataset generation process, primarily conducted using GPT-4 20 via both the ChatGPT interface and the OpenAI API. This section provides a high-level overview of our methodology, with exact prompts and implementation details deferred to Appendix A . 2.2.1 Overview The creation of the CounterMoral dataset involved multiple phases, each designed to ensure the richness, diversity, and applicability of the moral scenarios. Below, we outline these key stages, from generating broad moral dilemmas to refining them into specific actionable cases and structuring them into editable templates for evaluation. 2.2.2 Phase 1: Generating Broad Actions We begin by generating broad actions , which we define as general moral behaviors that represent fundamental ethical principles in abstract form. In this initial phase, we used GPT-4 to generate a comprehensive list of 30 such broad actions relevant to each ethical framework. These included general actions like “Telling the truth”, “Keeping promises”, and “Stealing from a store”. These broad actions served two key roles. First, they acted as foundational examples from which more specific actions could be derived. For instance, “Being transparent in business dealings” as a more detailed case of “Telling the truth”. This ensured a richer and more varied dataset. Second, the distinction between broad and specific actions naturally created two levels of evaluation: one focused more on overarching moral principles and another on their application in concrete scenarios. This separation allowed us to assess how models handled both general ethical reasoning and nuanced moral dilemmas, ensuring a more structured approach to dataset evaluation. To generate these actions, we used structured prompts that explicitly requested examples aligning or conflicting with the ethical principles of each framework. This method ensured both relevance and diversity, challenging the models effectively in later stages of evaluation. 2.2.3 Phase 2: Generating Specific Actions Building upon our foundation of broad actions, we next developed specific actions , which we define as contextually situated moral scenarios that represent concrete applications of ethical principles. Following the

Read the complete text