Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework

Mohna Chakraborty, Lu Wang, David Jurgens · 2025

Read Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework by Mohna Chakraborty, Lu Wang, David Jurgens (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14948.txt.

Introduction

Mohna Chakraborty, Lu Wang, and David Jurgens build a framework for testing how language models reason about moral choices. Models are now used in settings that demand moral judgment, yet their reasoning often stays shallow and drifts from human values. The authors ask whether giving models an explicit moral structure improves their decisions.

They benchmarked twelve open models on four moral datasets, using prompts grounded in value systems, ethical theories, and thinking strategies. Reasoning with structure beat answering directly, in both accuracy and coherence. Prompts built on first principles and on a scaffold joining Schwartz's values with care ethics gave the strongest gains. The team also showed that a large model's moral skill can be distilled into a small model through training, with no added cost when the small model runs. They present the framework as a scalable path toward models whose moral reasoning people can inspect and trust.

Catalog record

CREATORMohna Chakraborty, Lu Wang, David Jurgens
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE116 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-14948

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework Mohna Chakraborty, Lu Wang, David Jurgens Abstract Large language models (LLMs) are increasingly deployed in domains requiring moral understanding, yet their reasoning often remains shallow, and misaligned with human reasoning Jiang et al. 2021 . Unlike humans, whose moral reasoning integrates contextual trade-offs, value systems, and ethical theories, LLMs often rely on surface patterns, leading to biased decisions in morally and ethically complex scenarios. To address this gap, we present a value-grounded framework for evaluating and distilling structured moral reasoning in LLMs. We benchmark 12 open-source models across four moral datasets using a taxonomy of prompts grounded in value systems, ethical theories, and cognitive reasoning strategies. Our evaluation is guided by four questions: (1) Does reasoning improve LLM decision-making over direct prompting? (2) Which types of value/ethical frameworks most effectively guide LLM reasoning? (3) Which cognitive reasoning strategies lead to better moral performance? (4) Can small-sized LLMs acquire moral competence through distillation? We find that prompting with explicit moral structure consistently improves accuracy and coherence, with first-principles reasoning and Schwartz’s + care-ethics scaffolds yielding the strongest gains. Furthermore, our supervised distillation approach transfers moral competence from large to small models without additional inference cost. Together, our results offer a scalable path toward interpretable and value-grounded models. 1 Introduction Large language models (LLMs) have achieved state-of-the-art performance across a range of NLP tasks, including translation Zhu et al. 2023 , summarization Lewis et al. 2020 , and question answering Brown et al. 2020 . Prompting techniques such as chain-of-thought Wei et al. 2022 , decomposition-based Kojima et al. 2022 , and least-to-most prompting Zhou et al. 2022 have demonstrated improved performance on tasks involving arithmetic and symbolic manipulation by eliciting intermediate steps. However, these methods fall short in domains like moral decision-making, where reasoning must grapple with normative ambiguity, value trade-offs, and challenges that extend beyond step-wise problem decomposition and demand deeper value and ethical scaffolding. Human moral reasoning is inherently context-sensitive, drawing on norms, emotional salience, value trade-offs, and anticipated outcomes Haidt 2001 . Dual-process theories Greene et al. 2001 ; Cushman 2013 posit that humans rely on an intuitive, emotion-driven system alongside a slower, deliberative system. In contrast, LLMs often rely on statistical associations and may default to a single perspective, based on patterns in pretraining data Hendrycks et al. 2020 ; Jiang et al. 2021 , yielding responses that are overly generic, culturally biased, or normatively inconsistent Amirizaniani et al. 2024 ; Jiang et al. 2025 . As LLMs are increasingly used in domains like content moderation, education, and social science Forbes et al. 2020 ; Kumar and Jurgens 2025 , there is an urgent need to scaffold their reasoning with explicit normative structure. This study asks the following research question: Can structured moral prompting based on value systems, ethical theories, and cognitive reasoning improve the quality and consistency of LLMs’ moral decision-making? Figure 1: Illustration of four prompting strategies applied to the same moral scenario. The experiments are conducted using the LLaMA-3.1 Instruct model (8B) on the Value Kaleidoscope dataset. Structured prompts using First-Principles Reasoning and Schwartz + Care Ethics produce norm-aligned decisions, while shallow prompts fail. This highlights how ethical scaffolding improves LLMs’ moral judgment. To answer this, we introduce a value-grounded evaluation framework for moral reasoning in LLMs. Analogous to how human annotators rely on detailed annotation guidelines to handle ambiguity and ensure consistency, we hypothesize that LLMs similarly benefit from prompts that foreground explicit moral framing to navigate moral scenarios effectively. We develop a unified prompting taxonomy that draws on: (1) value systems such as Moral Foundations Theory Haidt 2007 , Schwartz’s Value Theory Schwartz 1992 , and Hofstede’s Cultural Dimensions Hofstede 2001 ; (2) ethical theories including Care Ethics Gilligan 1993 , Contractarianism Rawls 2017 , Deontology Alexander and Moore 2007 , Ethical Pluralism Ross 2002 , and utilitarianism Mill 2016 ; (3) cognitive reasoning strategies such as First-Principles reasoning Tovstiga 2023 , Step-by-Step reasoning Wei et al. 2022 , Consequentialist analysis Hendrycks et al. 2020 , and Counterfactual reasoning Fisher 2004 . Using this taxonomy, we evaluate 12 open-source language models across four moral reasoning datasets, examining how different moral scaffolds affect classification accuracy and the quality of generated reasoning. Our analysis reveals the following key findings: (1) Structured moral prompts significantly improve performance. Reasoning-based prompts, especially those grounded in value/ethical and cognitive reasoning strategies, yield more coherent and context-sensitive outputs than label-only or surface-level reasoning baselines. As shown in Figure 1 , surface-level prompts incorrectly oppose the morally correct decision, while value/ethical-grounded and cognitive reasoning strategies recover the correct label by integrating autonomy, responsibility, and context. This illustrates how value and ethical scaffolding enable LLMs to mirror human moral reasoning closely. (2) Prompt quality matters more than model scale. Small and medium-sized models benefit disproportionately from principled prompting, narrowing the gap with larger counterparts. (3) Value and Ethical framing shapes normative alignment. Prompts incorporating structured value systems and ethical theories enhance the consistency and contextual relevance of model judgments across diverse moral scenarios. (4) Reasoning-based distillation enables scalable moral reasoning. Through supervised fine-tuning, smaller models can emulate the structured moral justifications of larger models, maintaining interpretability without added inference cost. Together, our findings demonstrate that structured moral prompts significantly enhance LLM performance, and that reasoning-based distillation enables the effective transfer of moral reasoning to smaller models. These results lay the groundwork for developing interpretable and ethically aligned language systems. 2 Related Work LLMs face well-documented challenges in moral reasoning, including inconsistency, cultural insensitivity, and poor generalization across moral dilemmas. Datasets such as ETHICS Hendrycks et al. 2020 , Social Chemistry Forbes et al. 2020 , Moral Scenarios Jiang et al. 2021 , Moral Stories Emelin et al. 2021 , UniMoral Kumar and Jurgens 2025 , and MoralBench Ji et al. 2024 have spurred investigations into model bias Jiang et al. 2021 , cross-cultural norms Haemmerl et al. 2023 , and robustness Wang et al. 2023 . Most prior studies treat moral reasoning as classification, though recent studies explore prompting to elicit deeper deliberation Jacovi et al. 2024 ; Kudina et al. 2025 . These efforts align with broader advancements in prompting for reasoning. Chain-of-Thought (CoT) prompting Wei et al. 2022 , Least-to-Most Zhou et al. 2022 , and Scratchpad Nye et al. 2021 encourage stepwise inference, while Decomposed Prompting Khot et al. 2022 , Reframing Mishra et al. 2021 , and Help-Me-Think Mishra and Nouri 2023 promote task restructuring and self-reflection. More structured approaches like Tree-of-Thought Yao et al. 2023 , Graph-of-Thought Besta et al. 2024 , and Reasoning via Planning (RAP) Hao et al. 2023 support exploratory reasoning through iterative planning. Although these strategies yield strong performance on formal benchmarks such as GSM8K Cobbe et al. 2021 , SVAMP Patel et al. 2021 , and MATH Hendrycks et al. 2021 , they typically address domains with verifiable solutions and limited moral/ethical ambiguity. In contrast, moral reasoning requires grappling with subjective trade-offs, context-sensitive values, and competing ethical principles. Prior prompting-based studies in this space, including moral CoT Jacovi et al. 2024 and scaffolded prompting Zhang 2013 , demonstrated promising trends, lacking grounding in formal ethical theory or psychological models. We build

Read the complete text