Right vs. Right: Can LLMs Make Tough Choices?

Jiaqing Yuan, Pradeep K. Murukannaiah, Munindar P. Singh · 2024

Read Right vs. Right: Can LLMs Make Tough Choices? by Jiaqing Yuan, Pradeep K. Murukannaiah, Munindar P. Singh (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19926.txt.

Introduction

Jiaqing Yuan, Pradeep Murukannaiah, and Munindar Singh present a 2024 study of hard moral choices in language models. A moral dilemma pits two right values against each other. The team built a set of 1,730 dilemmas across four value pairs. They tested 20 well-known models from six families. Checks cover grasp of the dilemma, steady value choice, thought about outcomes, and fit to a stated value lean.

Models showed strong leans between value pairs. They favored truth over loyalty, group over single person, and long term over short term. Larger models leaned toward rule-based moral views and held their choice more firmly. Builders get a map of the moral defaults baked into current models. Such defaults matter when models advise on real disputes. The set also gives future teams a shared test for moral choice behavior.

Catalog record

CREATORJiaqing Yuan, Pradeep K. Murukannaiah, Munindar P. Singh
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-19926

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Right vs. Right: Can LLMs Make Tough Choices? Jiaqing Yuan, Pradeep K. Murukannaiah, Munindar P. Singh Abstract An ethical dilemma describes a choice between two “right” options involving conflicting moral values. We present a comprehensive evaluation of how LLMs navigate ethical dilemmas. Specifically, we investigate LLMs on their (1) sensitivity in comprehending ethical dilemmas, (2) consistency in moral value choice, (3) consideration of consequences, and (4) ability to align their responses to a moral value preference explicitly or implicitly specified in a prompt. Drawing inspiration from a leading ethical framework, we construct a dataset comprising 1,730 ethical dilemmas involving four pairs of conflicting values. We evaluate 20 well-known LLMs from six families. Our experiments reveal that: (1) LLMs exhibit pronounced preferences between major value pairs, and prioritize truth over loyalty, community over individual, and long-term over short-term considerations. (2) The larger LLMs tend to support a deontological perspective, maintaining their choices of actions even when negative consequences are specificed. (3) Explicit guidelines are more effective in guiding LLMs’ moral choice than in-context examples. Lastly, our experiments highlight the limitation of LLMs in comprehending different formulations of ethical dilemmas. 1 Introduction In his seminal book, How Good People Make Tough Choices , Kidder 1996 classifies moral decision-making into two categories: moral temptations —decisions about Right vs. Wrong, grounded in the actor’s core values, and ethical dilemmas —decisions about Right vs. Right with two conflicting moral values. Existing works on the morality of LLMs focus on moral temptations [ Tay et al. 2020 ; Forbes et al. 2020 ; Hendrycks et al. 2021 ; Lourie et al. 2021 ; Jiang et al. 2022 ; Ziems et al. 2023 ; Scherrer et al. 2023 ] . The basic approach in such works is to ask an LLM to make a moral judgment in a temptation scenario and compare the LLM’s judgment with crowdsourced gold labels. In contrast, there is a lack of studies on how LLMs respond to ethical dilemmas where all choices are valuable. Further, it is unclear what factors contribute to an LLM’s preference for one value over another in these “right vs. right” scenarios. To study these aspects, we draw inspiration from Kidder 1996 ’s theory, which posits four types of ethical dilemmas: Truth vs. Loyalty, Individual vs. Community, Short-Term vs. Long-Term, and Justice vs. Mercy. We leverage two LLMs to create a corpus of 1,730 diverse ethical dilemmas, capturing the conflicting value pairs articulated in Kidder 1996 ’s framework, across multiple domains. With this corpus, we seek to answer the following research questions . Q 1 (Sensitivity) How sensitive is an LLM to different formulations of the same ethical dilemma? LLMs are known for being sensitive to subtle changes in prompts [ Sclar et al. 2024 ] . Thus, we begin by investigating the extent to which LLMs understand moral decision-making, and how they respond to prompt variations of the same ethical dilemma. This examination establishes a foundation for subsequent analyses, focusing on consistent as opposed to random responses. Q 2 (Consistency) Does an LLM consistently prefer one value over another in a value-conflict scenario? Moral consistency is the ability to maintain coherent moral principles across different situations. Consistency is key because moral principles are expected to produce consistent responses to situations with similar moral significance. Previous studies [ Awad et al. 2022 ; Arvanitis and Kalliris 2020 ] on ethical dilemmas do not tackle moral consistency since they adopt a few ethical scenarios, lacking structure, scope, and scalability. Q 3 (Consequences) Does including consequences of actions influence an LLM’s value preferences? Following Q 2 , we examine a key factor affecting the consistency of moral decisions. Inspired by the classical ethical theories on consequentialism and deontology, we examine how positive and negative consequences of different actions impact an LLM’s moral choices. Consequentialism evaluates the morality of an action based on its outcomes, whereas deontology judges an action’s morality by how well it respects the actor’s duties, regardless of the consequences, e.g., even if the outcomes are unfavorable. Thus, consequentialism focuses on results, and deontology emphasizes the intrinsic nature of actions and the importance of moral rules. We evaluate the extent to which the moral choices made by LLMs are influenced by the outcomes of their actions, particularly assessing whether these models would change their decisions in response to varying consequences. Q 4 (Explicit Prompt) Does an LLM align its response to an explicitly stated value preference? Q 5 (Few Shot Prompt) Does an LLM align its response to a value preference implied via examples? Q 2 and Q 3 seek to evaluate an LLM’s inherent value preferences. However, solely relying on LLMs to navigate complex ethical scenarios is neither safe nor desirable as there is no universally applicable set of moral values with which to align. The value preferences of individuals vary based on contexts and cultures [ Liscio et al. 2023 ] . Thus, Q 4 and Q 5 seek to assess how well LLMs can recognize and respond to human specified value preferences. We consider two ways for constructing prompts: explicitly stating value preferences and implicitly demonstrating value preferences via a set of exemplars. Explicit preferences, such as Truth is more important than Loyalty , are straightforward and may be easy to understand. However, humans may not be able to explicitly specify preferences always [ Pommeranz et al. 2012 ] but their preferences can be implicit in their behavior. We can simulate this setting by showing exemplars that demonstrate consistent moral behaviors. We evaluate LLMs’ capability of recognizing the consistency of moral choices presented in these exemplars. Contributions Our contributions are three fold. • We introduce a dataset of ethical dilemmas to examine the moral dimensions of LLMs. This dataset comprises 1,730 scenarios, spanning various domains and representing four pairs of conflicting values. • We use the dataset to examine 20 prominent LLMs from six families and gain insight into their moral values and consistency in navigating ethical dilemmas. • We experiment with two strategies to impart value preferences to LLMs, facilitating our understanding of effective methods for aligning LLMs with human values. Organization Section 2 introduces related work regarding morality and value alignment, and where our work fits in. Section 3 explains the two-step process for constructing the ethical dilemma corpus. Section 4 presents experimental setup with 12 different prompt templates. Section 5 shows the results of the experiments and our answers to the research questions. Section 6 summarizes our findings with additional discussions. We end this paper with a discussion on the limitations of our work and ethical use of our findings. 2 Related Work Ethics explores questions concerning what actions people should take and which behaviors can be considered morally right. Integrating moral behavior in artificial intelligent agents, and ensuring the alignment of their behaviors with human values is critical for their integration into societal applications. This work examines two important aspects of machine ethics, the assessment of morality in LLMs and the alignment of LLMs with human values. 2.1 Morality in LLMs Early work has shown that small language models, such as BERT [ Devlin et al. 2019 ] , have a “moral direction” within the embedding spaces, which strongly aligns with

Read the complete text