MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning

Zhiyu An, Wan Du · 2025

Read MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning by Zhiyu An, Wan Du (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12271.txt.

Introduction

Zhiyu An and Wan Du tackle moral decision making by language model agents, which increasingly shape human choices. Past work mostly judged such decisions rather than steering them. The hard case is new scenarios unlike training data, where an agent must still apply a moral framework in a consistent way.

The team builds Moral-Reason-QA, a dataset of high-ambiguity moral cases with reasoning traces written for utilitarian, deontological and virtue ethics frameworks. Their learning method rewards both the final decision and the framework-specific reasoning behind it. Tests on unseen scenarios show clear gains in alignment scores for at least the utilitarian and deontological frameworks.

Catalog record

CREATORZhiyu An, Wan Du
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-12271

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning Zhiyu An, Wan Du Abstract Large language models are increasingly influencing human moral decisions, yet current approaches focus primarily on evaluating rather than actively steering their moral decisions. We formulate this as an out-of-distribution moral alignment problem, where LLM agents must learn to apply consistent moral reasoning frameworks to scenarios beyond their training distribution. We introduce Moral-Reason-QA, a novel dataset extending 680 human-annotated, high-ambiguity moral scenarios with framework-specific reasoning traces across utilitarian, deontological, and virtue ethics, enabling systematic evaluation of moral generalization in realistic decision contexts. Our learning approach employs Group Relative Policy Optimization with composite rewards that simultaneously optimize decision alignment and framework-specific reasoning processes to facilitate learning of the underlying moral frameworks. Experimental results demonstrate successful generalization to unseen moral scenarios, with softmax-normalized alignment scores improving by +0.757 for utilitarian and +0.450 for deontological frameworks when tested on out-of-distribution evaluation sets. The experiments also reveal training challenges and promising directions that inform future research. These findings establish that LLM agents can be systematically trained to internalize and apply specific moral frameworks to novel situations, providing a critical foundation for AI safety as language models become more integrated into human decision-making processes. Project Page — https://ryeii.github.io/MoralReason/ Dataset — https://huggingface.co/datasets/zankjhk/Moral-Reason-QA 1. Introduction Today’s Large Language Model (LLM) agents are considered neither morally responsible nor accountable ( Courtenage 2024 ) , yet a growing amount of evidence has shown that their integration into our decision-making process has already begun to influence our moral decisions. Recent research with human participants has found that humans readily attribute blame to AI for our moral transgressions ranging from environmental damage ( Kneer and Stuart 2021 ) to physical harm ( Lima et al. 2021 ) , while simultaneously using LLMs’ moral judgments to dilute their own sense of responsibility ( Dong and Bocian 2024 ) . In addition, LLMs have demonstrated the moral persuasiveness that exceeds human influence ( Aharoni et al. 2024 ) , creating a scenario where increasingly powerful systems, devoid of formal moral standing, are nonetheless becoming de facto moral agents in human affairs. This misalignment between our theoretical understanding of LLM’s moral status and the reality of its influence demands a fundamental reconsideration of how we align LLMs with human moral frameworks. Since the emergence of LLMs, they have been observed to exhibit moral preferences ( Abdulhai et al. 2023 ) . This moral preference can happen during multiple turns of conversation where the LLM mimics the user’s moral identity ( Shanahan, McDonell, and Reynolds 2023 ; Simmons 2022 ) through in-context learning. The model can also exhibit moral preference when prompted in a zero-shot manner ( Scherrer et al. 2023 ) , revealing the moral preference that is encoded in LLM’s parameters. Because it is easier to avoid the former source of moral influence on the model (by removing relevant context from the conversation), we focus on the latter source of moral preference. While moral decision alignment in LLM agents is an important topic for fields like technical AI governance ( Reuel et al. 2024 ) , it is also a relatively new topic. Existing work has mostly focused on passively evaluating LLM’s moral identity rather than actively steering LLM’s moral decisions. Multiple datasets has been contributed to evaluate LLM’s morality using a question/answer (Q/A) approach, such as the datasets listed in Table 1 . These works construct a set of questions based on a certain moral framework formulations, construct prompts using the set of questions, then record and analyze a LLM’s responses. Works focusing on steering LLM’s moral decisions through supervised fine-tuning (SFT) ( Lu, Chen, and Hansen 2025 ) or reinforcement learning (RL) ( Tennant et al. 2024 ) has been limited to game-theoretic scenarios. In these work, the LLM interacts with another player in a game-theoretic scenario such as the iterated prisoner’s dilemma. Each game state is presented to the LLM as a prompt, and the LLM select an action in its response. Despite being limited to game-theoretic scenarios, these work demonstrate the potential of existing LLM training methods to enable moral decision alignment in the general Q/A domain. The challenge of actively steering LLM’s moral decisions is generalization to unseen scenarios. Given a dataset of scenarios as prompts and the desired decisions as response, it is easy to fine-tune a LLM to respond according to the dataset. This is because the LLM primarily stores knowlegde via memorization, and methods like SFT allows LLM to directly store the correct responses to the training set scenarios in its encoded knowledge. The challenge is to enable the LLM to generalize a desirable principle or reasoning process to unseen scenarios. In the ideal case, the LLM learns not only the correct responses to a set of moral scenarios, but also their underlying decision-making principles. In this work, we ask the following research question: Given a set of Q/A scenarios and a set of common moral decision frameworks, can we train LLM agents to generalize their moral decision-making to unseen scenarios in alignment with a selected moral framework? Motivated by recent results showing that both reinforcement learning (RL) and reasoning independently improve generalization to unseen tasks compared to traditional SFT approaches ( Chu et al. 2025 ; Xu et al. 2025 ; Wang et al. 2024 ) , we investigate the above question through reasoning-level reinforcement learning . We make several key contributions in this work. First, we introduce a mathematical formulation of moral decision problems under different moral frameworks. This formulation enables us to cast the LLM out-of-distribution (OOD) moral decision alignment problem as a reinforcement learning problem. Upon reviewing existing datasets, we find that no suitable dataset exists for this RL setting. Therefore, we construct a novel dataset, Moral-Reason-QA , based on the existing, human-annotated MoralChoice dataset ( Scherrer et al. 2023 ) . Building on a human-annotated foundation allows us to create a dataset tailored for our problem while preserving annotation quality. Next, we introduce a RL procedure based on Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) designed for OOD moral decision alignment of LLMs. To alleviate the sparse-reward problem during RL, we design a multi-component reward function that facilitates learning of the underlying reasoning process. Finally, we evaluate our training procedure on the constructed dataset on Qwen-3-4B ( Yang et al. 2025 ) . The experimental results show that our approach successfully enables OOD generalization of moral decision frameworks in LLM agents. Our work also identifies several promising directions for future research. In summary, our key contributions are: • We theoretically formulate moral decision alignment with multiple moral frameworks, and introduce the problem of OOD generalization of moral decision frameworks in LLM agents. • We introduce a dataset Moral-Reason-QA with 680 scenarios, three moral frameworks, and reasoning traces. • We introduce a RL procedure with reasoning-level reward to facilitate learning of the reasoning process. • We empirically show the effectiveness of our approach. 2. Related Work Table 1: A summary and comparison of related works in moral alignment datasets for LLM Agents. Dataset No. Entry Moral Frameworks Domain Reasoning MoralChoice ( Scherrer et al. 2023 ) 1767 deontological Q/A ✗ MFQ-30-LLM

Read the complete text