Uncertain Machine Ethics Planning
Simon Kolker University of Manchester, UK [email protected] Louise A. Dennis University of Manchester, UK [email protected] Ramon Fraga Pereira University of Manchester, UK [email protected] Mengwei Xu Newcastle University, UK [email protected] · 2025
Read Uncertain Machine Ethics Planning by Simon Kolker University of Manchester, UK [email protected] Louise A. Dennis University of Manchester, UK [email protected] Ramon Fraga Pereira University of Manchester, UK [email protected] Mengwei Xu Newcastle University, UK [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04352.txt.
Introduction
Simon Kolker and co-authors set out how a machine should plan when moral rules clash and facts are unsure, in a 2025 paper. A good choice weighs what may happen over a long chain of acts, not just the next step. But moral frames judge the end in unlike ways: some count total good, some check duties, some prize character.
Their form lets a plan carry more than one moral judge at once. A fast hunt method then finds paths that do mild harm while they reach fair goals. The work gives machine ethics a shared math tongue for clashes that were once talked past each other.
Catalog record
| CREATOR | Simon Kolker University of Manchester, UK [email protected] Louise A. Dennis University of Manchester, UK [email protected] Ramon Fraga Pereira University of Manchester, UK [email protected] Mengwei Xu Newcastle University, UK [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-04352
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Uncertain Machine Ethics Planning Simon Kolker University of Manchester, UK [email protected] Louise A. Dennis University of Manchester, UK [email protected] Ramon Fraga Pereira University of Manchester, UK [email protected] Mengwei Xu Newcastle University, UK [email protected] Abstract Machine Ethics decisions should consider the implications of uncertainty over decisions. Decisions should be made over sequences of actions to reach preferable outcomes long term. The evaluation of outcomes, however, may invoke one or more moral theories, which might have conflicting judgements. Each theory will require differing representations of the ethical situation. For example, Utilitarianism measures numerical values, Deontology analyses duties, and Virtue Ethics emphasises moral character. While balancing potentially conflicting moral considerations, decisions may need to be made, for example, to achieve morally neutral goals with minimal costs. In this paper, we formalise the problem as a Multi-Moral Markov Decision Process and a Multi-Moral Stochastic Shortest Path Problem . We develop a heuristic algorithm based on Multi-Objective AO* , utilising Sven-Ove Hansson’s Hypothetical Retrospection procedure for ethical reasoning under uncertainty. Our approach is validated by a case study from Machine Ethics literature: the problem of whether to steal insulin for someone who needs it. † † footnotetext: The original version of this paper has been accepted by the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025) as a Full Paper. It is available by open access under the terms of the Creative Commons Attribution Licence (CC BY 4.0) https://creativecommons.org/licenses/by/4.0 1 Introduction As our lives are increasingly impacted by the decisions of machines [ 40 ] , one objective of Machine Ethics is to develop computational techniques to incorporate ethical behaviour with decision-making, preserving human values into the future [ 2 ] . A major concern for Machine Ethics comes from situations with uncertainty over outcomes, commonplace in the real world. Most existing approaches either assume actions are deterministic or they use expected utility maximisation [ 39 ] . This can yield disagreeable results (see Section 3 and [ 11 ] ), demanding a more sophisticated handling of outcome uncertainty . Decisions are further complicated by moral uncertainty [ 19 ] . Philosophy has no universal consensus over which decisions are correct in all situations: moral theories may conflict, leading to moral dilemmas . While we defer the choice of theories to stakeholders who must approve a system before deployment (owners, regulators, or affected individuals), there are likely to be conflicts or uncertainty between stakeholder approved theories. Implementations must handle these situations gracefully. Another issue is the integration of ethical behaviour into existing systems with non-moral goals. For example, an autonomous vehicle must balance risk to other road users with reaching its destination. As the Automated Planning community demonstrates, many environments require sequences of actions (action plans or policies) to achieve desired goals and the most rewarding outcomes [ 20 ] . Machine Ethics implementations, therefore, should consider future decisions, rather than selecting immediately preferred actions greedily as is the case in most existing systems [ 39 ] . \pi \pi^{\prime} s_{0} \pi s_{0} \pi^{\prime} 0.5 \boldsymbol{s_{0}} \pi^{\prime} 0.5 Figure 1 : A company may produce a beauty care product or research a drug with 50% probability of saving lives. Beauty care has negative retrospection (like regret) for missing the chance to save lives; if research is unsuccessful, there is no negative retrospection since risk was accepted at decision-time. Arguments are generated from each outcome. Directed edges are attacks indicating negative retrospection. Hypothetical Retrospection is an ethical procedure for handing outcome uncertainty [ 23 ] . It suggests we judge decisions by the circumstances under which they were made. In [ 26 ] , Machine Ethics Hypothetical Retrospection (MEHR) is formalised as a framework for single action decisions with outcome uncertainty and multiple moral theories under moral uncertainty. Arguments are generated in support of each action from the hypothetical perspective of their outcomes. An argument is attacked if there is negative retrospection for missing an ethically preferred outcome. An action that minimises non-acceptability across arguments is selected. There is an example in Figure 1 , details in Section 2 . In this paper, we formalise uncertain ethical planning problems as a variant of the Markov Decision Process (MDP). Our Multi-Moral Markov Decision Process (MMMDP) may contain a number of potentially conflicting moral theories. Stakeholders may express preference or moral uncertainty over theories with a weak lexicographic ordering . A theory’s space of morally relevant information is general and inclusive to many ethical perspectives. We extend our formalism to the Multi-Moral Stochastic Shortest Path Problem (MMSSP) for non-moral situations. Preferable solutions act ethically subject to goals and costs. We produce an offline heuristic planning algorithm to solve these problems based on AO* [ 32 ] , Multi-Objective heuristic planning [ 12 ] and MEHR [ 26 ] . We evaluate the algorithm with a case study from Ethics and Machine Ethics literature: the problem of whether to steal insulin for someone who needs it [ 3 ] . 2 Background Markov Decision Process Uncertain decision making problems can be described as a Finite Horizon Markov Decision Process (MDP). We represent them as a tuple \langle S,A,P,R,H\rangle , where S is a finite set of states, and A is a finite set of actions. The state space is traversed through discrete sequential transitions. The probabilistic transition function P:S\times A\times S\rightarrow[0,1] defines the probability one state transitions to another via an action. Transitions are valued by a reward function R:S\times A\times S\rightarrow\mathbb{R} . An optimal solution is a non-stationary policy \pi:S\times\{0,...,H-1\}\rightarrow A that maps a state and timestamp to an action with maximum expected reward. The horizon H\in\mathbb{N} is the number of state transitions before the problem terminates. Policies are evaluated with a non-stationary value function defined recursively: V^{\pi}(s,t)=\sum_{s^{\prime}\in S}P(s,\pi(s,t),s^{\prime})[R(s,\pi(s,t),s^{\prime})+V^{\pi}(s^{\prime},t+1)] . By Bellman ’s optimality principle [ 6 ] ,if a policy’s value can be measured by its expected additive utility, then there exists a policy \pi^{*} optimal at all sates and it satisfies the Bellman optimality equation V^{\pi^{*}}(s,t)=\max_{a\in A}\sum_{s^{\prime}\in S}P(s,a,s^{\prime})[R(s,a,s^{\prime})+V^{\pi^{*}}(s^{\prime},t+1)] . Stochastic Shortest Path (SSP) problems are MDPs with goal states G\subseteq S that are terminal, meaning there are no transitions away from them: \forall_{s\in G,a\in A},P(s,a,s)=1\land R(s,a,s)=0 ; all other rewards are strictly negative. SSPs also have an initial state s_{0} . Optimal policies are proper if there is 1.0 probability of reaching a goal from s_{0} . Improper policies have an expected cost of \infty . MDP Algorithms MDPs can be solved using Dynamic Programming algorithms, e.g. Policy Iteration [ 24 , 8 ] or Value Iteration (VI) [ 8 ] . VI refines an arbitrary value function V_{0} with Bellman backups : iterative applications of the Bellman optimality equation [ 6 ] . Values propagate until the maximum absolute difference between values in V_{i} and V_{i+1} (the residual ) is below a constant \epsilon>0 ( \epsilon -consistency). This requires full sweeps across the state space. Heuristic algorithms (RTDP [ 5 ] , AO* [ 32 ] , LAO* [ 22 ] and LRTDP [ 10 ] ) improve performance when there is a initial state s_{0} by only iterating on reachable, high value states. They also use a heuristic function to initialise V_{0} to an estimate of each state’s optimal reward. Multi-Objective