Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback
Vijay Keswani, Cyrus Cousins, Breanna Nguyen, Vincent Conitzer ∗, Hoda Heidari ∗, Jana Schaich Borg ∗, Walter Sinnott-Armstrong ∗ · 2025
Read Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback by Vijay Keswani, Cyrus Cousins, Breanna Nguyen, Vincent Conitzer ∗, Hoda Heidari ∗, Jana Schaich Borg ∗, Walter Sinnott-Armstrong ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10032.txt.
Introduction
Vijay Keswani and six co-authors ask a pointed question for AI alignment: what should a system align to when people's moral preferences keep changing? Alignment pipelines treat human feedback as a fixed target, yet preferences shift through genuine moral learning and through noise such as inattention or bias. The study grounds the problem in kidney allocation, a high-stakes domain where trade-offs are concrete.
Over four hundred participants compared hypothetical transplant patients across three to five sessions. People changed their answer to the same scenario six to twenty percent of the time, and fitted decision models drifted too. Simple AI models trained on the feedback lost predictive accuracy as instability grew and as time passed. The authors argue alignment must separate legitimate moral change from noise.
Catalog record
| CREATOR | Vijay Keswani, Cyrus Cousins, Breanna Nguyen, Vincent Conitzer ∗, Hoda Heidari ∗, Jana Schaich Borg ∗, Walter Sinnott-Armstrong ∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-10032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback Vijay Keswani, Cyrus Cousins, Breanna Nguyen, Vincent Conitzer ∗, Hoda Heidari ∗, Jana Schaich Borg ∗, Walter Sinnott-Armstrong ∗ Abstract Alignment methods in moral domains seek to elicit moral preferences of human stakeholders and incorporate them into AI. This presupposes moral preferences as static targets, but such preferences often evolve over time. Proper alignment of AI to dynamic human preferences should ideally account for “legitimate” changes to moral reasoning, while ignoring changes related to attention deficits, cognitive biases, or other arbitrary factors. However, common AI alignment approaches largely neglect temporal changes in preferences, posing serious challenges to proper alignment, especially in high-stakes applications of AI, e.g., in healthcare domains, where misalignment can jeopardize the trustworthiness of the system and yield serious individual and societal harms. This work investigates the extent to which people’s moral preferences change over time, and the impact of such changes on AI alignment. Our study is grounded in the kidney allocation domain, where we elicit responses to pairwise comparisons of hypothetical kidney transplant patients from over 400 participants across 3 3 – 5 5 sessions . We find that, on average, participants change their response to the same scenario presented at different times around 6–20% of the time (exhibiting “response instability”). Additionally, we observe significant shifts in several participants’ retrofitted decision-making models over time (capturing “model instability”). The predictive performance of simple AI models decreases as a function of both response and model instability. Moreover, predictive performance diminishes over time, highlighting the importance of accounting for temporal changes in preferences during training. These findings raise fundamental normative and technical challenges relevant to AI alignment, highlighting the need to better understand the object of alignment (what to align to) when user preferences change significantly over time, including the mechanisms underlying these changes. * * footnotetext: These authors contributed equally and are listed alphabetically. 1 Introduction Alignment of modern AI systems to human preferences has emerged as a cornerstone of the current pursuit of responsible, ethical, and safe AI. Various failure modes of AI — such as biased behavior [ 39 ] , reward hacking [ 35 ] , and even existential threats [ 42 ] — have been framed as stemming from misalignment between AI’s goals and the values and preferences of its stakeholders. As a result, the field of AI alignment aspires to address these issues by ensuring that AI outputs are guided (and often constrained) by user and societal preferences [ 22 ] . The advertised applicability of this approach in encoding subjective preferences in AI systems has also led to its proposed usage in moral domains [ 27 , 33 , 25 , 48 ] . Since moral dilemmas are bound to arise in several consequential applications of AI — e.g., designing equitable healthcare resource allocation policies [ 44 ] or characterizing autonomous vehicle behavior [ 2 ] — aligning the AI to the moral preferences of the user or the aggregated preferences of a community provides a pathway towards encoding ethical values in AI’s behavior. The standard approach to building morally-aligned AI involves a one-time elicitation of moral preferences from users [ 2 , 17 , 25 ] . Yet, human preferences are dynamic, evolving over time and across contexts [ 51 ] . Cognitive processes underlying moral judgments also fluctuate, reflecting changes in values informing moral decisions, updated moral reasoning in light of new information, or even alterations in time-on-task and cognitive capacity [ 1 , 54 , 23 ] . However, much of the existing work in AI alignment for moral domains neglects temporal changes to the underlying moral preferences of stakeholders, the consequences of which can be severe in real-world applications [ 15 ] . The possibility of negative impacts of temporal misalignment and the increased use of AI for decision-making have indeed led to several recent calls for real-world data collection to assess the impact of preference changes on AI alignment [ 57 , 4 , 7 ] . Our work answers this call with a data-driven investigation into dynamic moral preferences. Our Contributions. We investigate the AI alignment challenges associated with temporal changes in moral preferences. Our work studies moral preferences related to kidney transplant allocation, where advocates have noted the ability of AI to improve transplant efficiency [ 40 ] and align transplant decisions with stakeholder preferences [ 17 ] . Simulating a setting where kidney patients outnumber available kidneys, each participant in our study is presented with several pairwise comparisons of hypothetical kidney patients and asked to choose which patient should receive the kidney if only one is available. In this morally high-stakes domain, we elicit preferences of more than 400 survey participants over three to five sessions, spanning up to two weeks. In each session, participants are presented with 60 pairwise comparisons, with six comparisons repeated across all sessions and twice per session (Section 3 ). Participants’ responses to the repeated scenario allow us to measure their “response instability,” i.e., how often they change their response to the same scenario repeated at different times/during different sessions. We observed that response instability, on average, varies from 6–20%, with significantly higher instability for scenarios that were relatively more challenging (i.e., containing tradeoffs across several morally-relevant factors; Section 4.1 ). We also investigate the extent to which participants seem to change their decision-making processes over time. Evaluation of agreement between models learned from participants’ session-wise responses shows that the overlap between predictions from session-wise models decreases with time; we call this phenomenon “model instability” (Section 4.2 ). To understand the reasons for changing preferences, we categorize participants by their response stability and model stability levels, and study variations in decision-making properties across participant categories. We provide evidence that different participants exhibit different mechanisms of preference change (Section 5 ). For some, preference change arises from a reduction in model complexity (e.g., reducing the number of factors they account for in making their decisions) over time. For others, instability reflects stochastic changes in their decision processes (which may be due to morally arbitrary factors, such as attention, time of day, and beyond). Crucially, different mechanisms underlying the change in preferences prescribe different approaches to AI alignment. While it is essential for alignment methods to identify and capture legitimate changes to moral preferences of stakeholders (e.g., those resulting from participants’ judgment evolving with more exposure and reflection), morally arbitrary changes, such as noisy responses due to fatigue and attention fluctuations, should not impact alignment. To assess the impact of preference change on AI alignment, we train preference optimization models on participant- and population-level data. We observe a significant association between temporal instability and the trained model’s error rate in predicting participant responses (Section 5 ). In particular, for participants who exhibited high levels of response and model instability, predictive error rate was higher by 5-16%, in comparison to those who were response and model stable. Additionally, a temporal analysis shows that error rates of trained models increase over time for participants who were response and model unstable. Overall, our study highlights several challenges and corresponding opportunities for AI alignment. Our findings emphasize a prominent normative challenge