A Case for End-Constrained Ethical Artificial Intelligence Authors: Tyler Cook Published: 2025 ABSTRACT One natural motivation for designing artificial intelligence (AI) with ethical capacities is to mitigate the risk that powerful AI systems will harm us. Against this idea, some authors have argued we do not need ethical AI in order to prevent harm to humans. We simply need safe AI. In this paper, I consider an argument of this type and raise objections to it. In particular, I argue that the goal of implementing safety features in AI systems is a much more complicated task than such authors have acknowledged, and moreover I maintain that merely safe AI could be ethically problematic in numerous ways. Then, I show that a certain kind of ethical AI, which I call end-autonomous ethical AI, would be especially dangerous since their autonomous capacities would make possible numerous risks. Finally, I motivate the case for a specific category of ethical AI, namely, end-constrained ethical AI. I describe these systems as possessing whatever capacities would be necessary for satisfying ethical aims beyond safety while lacking end-autonomy. In short, the goal of this paper is to establish that end-constrained ethical AI occupy a desirable middle ground between merely safe AI and end-autonomous ethical AI because they allow us to secure more ethical goods than just safety, and they are also not as risky as end-autonomous ethical AI. Some people who advocate for the design of ethical artificial intelligence (AI) argue that because AI could potentially harm us, we should focus some of our efforts in AI research on the design of ethical machines. If at least some AI systems could become sophisticated enough to outsmart us in certain ways and opt to harm us, then it seems that we have strong reason right now to take whatever precautionary measures might be necessary to prevent such AI from harming us. The only way to ensure this, these machine ethicists argue, would be to design ethical AI.1 AI systems that have the power to harm us would presumably be unlikely to do so if they were successfully designed to be ethical (except perhaps in certain cases where causing harm might be permissible). Some authors attempt to raise a problem for the foregoing line of thought by claiming that we do not need ethical AI in order to prevent harm to humans. We simply need safe machines.2 In the first section of this paper, I will consider one such argument and raise some objections to it. In particular, I will argue that the goal of implementing safety features in AI systems is in itself a much more complicated task than some authors seem to acknowledge, and moreover I will maintain that merely safe AI could still be ethically problematic in numerous ways. Then, in Section “Against End-Autonomous Ethical AI”, I will show that a certain kind of ethical AI, which I will call end-autonomous ethical AI, would be especially dangerous. I will characterize them as having a strong form of autonomy (akin to humans) that allows for the possibility of numerous risks. Finally, I will motivate the case for a specific category of ethical machines, namely, end-constrained ethical AI. I will describe these AI as possessing whatever capacities would be necessary for satisfying ethical aims beyond safety while lacking end-autonomy. I will argue that we should want to design end-constrained ethical AI for reasons extending beyond those of safety, and I will emphasize some desirable capabilities they could have as well as some risks. I will also mention a few examples of possible contexts into which such AI could be introduced and expected to behave ethically. In short, the aim of this paper is to establish that end-constrained ethical AI occupy a desirable middle ground between merely safe AI and end-autonomous ethical AI because they allow us to secure more ethical goods than just safety, and they are also not as risky as end-autonomous ethical AI. Against Merely Safe AI As van Wynsberghe and Robbins (2019) correctly note, many technologies currently exist that can and sometimes do cause harm to humans (e.g. lawnmowers, blenders, garage doors, etc.), but no one has ever seriously thought that in order to prevent those technologies from harming us, they must be designed to be ethical or engage in ethical decision-making.3 Probably no one has ever really believed, say, that we must design lawnmowers that can treat us ethically (whatever that might entail) to ensure that those machines would not harm us.4 Rather:[T]he solution has always been either to design them with safety features or to limit the contexts in which a technology can be used. An elevator door has a sensor so that it does not close on people; lawn mowers have a guard to protect us from their blades; and ovens have lights to warn us when our stovetop is hot. One does not normally use barbeques indoors or chainsaws in daycare centers. Machine ethicists are the first to suggest endowing technology with moral reasoning capabilities as a solution to problems of safety.5 [T]he solution has always been either to design them with safety features or to limit the contexts in which a technology can be used. An elevator door has a sensor so that it does not close on people; lawn mowers have a guard to protect us from their blades; and ovens have lights to warn us when our stovetop is hot. One does not normally use barbeques indoors or chainsaws in daycare centers. Machine ethicists are the first to suggest endowing technology with moral reasoning capabilities as a solution to problems of safety.5 Their disjunctive safety solution might work perfectly well for unsophisticated technologies, such as lawn mowers and ovens, but the situation is quite different when it comes to more sophisticated AI systems, which are (and increasingly will be) decision-making systems that can be utilized across various contexts. It does not appear to be obvious or even likely that the implementation of ordinary safety solutions will be enough to prevent especially sophisticated AI systems from harming us. First, consider the fact that safety will be more difficult to achieve than ever before, given how sophisticated and widespread such AI will be. How do we guarantee that AI systems will be safe in every context in which they might operate? Unlike lawnmowers and chainsaws, some AI systems could very well become capable of operating in various sorts of context.6 If this is right, then it seems plausible that different safety constraints would be required for different contexts, depending on the nature of the context. Blade guards and movement sensors clearly would not be enough to ensure safety everywhere. So, there is a difficult question here of how to go about programming AI systems to ensure safety in different contexts.7 Furthermore, there is the question of whether we would have to invent a new set of safety features every time an AI becomes capable of operating in a new kind of context or whether there might be some general design strategy for ensuring safe AI in every possible decision-making context. If, for instance, a machine that were capable of safely administering various medicines to patients became capable of performing various surgeries too, it seems obvious that the machine would encounter a new set of challenges that would call for safety features beyond those having to do with the safe administration of medicine. Because of possibilities like this, it is difficult to imagine how a perfectly general safety strategy could succeed or what it would even look like, given how varied the dangers are that could exist in different contexts. The mention of guards for lawnmowers and heat warning lights on ovens in the cited passage makes all of this seem like a simple matter, but this is very misleading. We could easily predict the sorts of dangers that we would be vulnerable to whenever we first designed those types of technologies. Just how many different ways could elevator doors be dangerous to us? This question is much easier to answer than the one for sophisticated AI systems. For the latter technologies, we will require something much more robust or complicated to prevent them from harming us than the sorts of safety solutions we already have for existing technologies. Secondly, even if we succeeded at designing safe AI systems, merely safe AI that did not satisfy any further ethical goals or aims could be ethically problematic in other ways than ones relating just to safety. While genuinely safe AI would keep us safe from harm, even our merely defensive ethical concerns are typically much broader than that. The idea here is that sophisticated AI could treat us unethically in numerous other ways that do not necessarily involve direct harms. They could deceive or manipulate us, treat us unfairly, or violate some of our other negative rights more generally.8 They could also fail to satisfy positive ethical aims, such as saving us from harm when they could easily do so.9 Thus, merely safe AI would cover only a single part of our total web of ethical concerns. If all that we should want from AI were safety, then that would be acceptable. But given the fact that it would be able to mistreat us in other ways, and merely safe AI would not be concerned with that, we should not want to design merely safe AI. We need something more. We need AI that satisfies ethical aims beyond safety. Against End-Autonomous Ethical AI We need ethical AI that extend beyond considerations of safety, but as I will argue in this section, a certain type of ethical AI would be particularly dangerous, and this is due to the nature of the autonomous capacities it would possess. At this stage, it is important to be clear about the kinds of autonomy AI systems might instantiate in order to bring into focus the problematic sort of autonomous AI. I will distinguish a couple of different ways in which a technology could be autonomous, which are not meant to be exhaustive, in order to home in on the kind of autonomy that I find especially problematic.10 There are some objects that have no autonomy whatsoever and are controlled entirely by external forces (e.g. a hammer). No plausible account of autonomy would attribute autonomous capacities to hammers or other similar objects since they can accomplish nothing all on their own or without being manipulated in some way by an outside force. Then, there are systems that could be said to possess some level of autonomy in that they can, for instance, compare and select options independently or without human intervention (e.g. a delivery robot that selects its own route), but the ends of such systems are ultimately provided to them by their designers and users. These systems pursue the ends that they are programmed to pursue, and they do not exercise any autonomy with respect to consideration of their ends.11 Finally, there could conceivably be AI systems that are autonomous inasmuch as they are capable of considering the ends (or aims or goals) provided to them by their designers, and they could independently assess such ends and engage in decision-making about whether they will act in accordance with or in pursuit of those ends.12 AI of this final sort possess end-autonomy, as I refer to it, and I will argue next that AI that are both end-autonomous and designed to satisfy ethical aims beyond safety (i.e. end-autonomous ethical AI) are risky. Of course other kinds of end-autonomous AI would be risky too, but the argument I will make is that end-autonomy is risky even for ethical AI, since the possession of end-autonomy entails that such AI might reconsider their ethical aims. It is also important to note that my claim is not that end-autonomy is the only type of autonomy that it would be risky for AI systems to instantiate, as there could certainly be others. Here I only intend to show that it would be risky to have end-autonomous AI around. In what follows I briefly explain the risks of end-autonomy, focusing on end-autonomous ethical AI in particular since our focus throughout is on the desirability of ethical AI. While I contend that end-autonomous ethical AI would be risky quite generally, the degree to which these AI are risky will depend on their breadth of application, or so I will argue. In particular, narrow end-autonomous ethical AI, which are AI systems that would operate exclusively in certain narrowly defined contexts (e.g. autonomous driving, policing, assistive care, and so on) would be less risky than general end-autonomous ethical AI, which would operate in ethically sensitive contexts more generally. The latter type of AI would seemingly need to be more ethically sophisticated than the former since it would be expected to engage in ethical decision-making across a broader variety of contexts, but we are to assume that both kinds of AI are able to engage in ethical decision-making that goes beyond deliberating about safety, and they are end-autonomous. First, concerning narrow end-autonomous ethical AI, one example of this type of system could be a sophisticated autonomous vehicle (which we luckily do not have yet). An apparent danger with this kind of vehicle is that it might, for instance, deliberately cause huge accidents with multiple human casualties. Crucially, the ethical risk here is a unique concern for end-autonomous ethical AI, as opposed to current autonomous vehicles, because the danger could be shown to be attributable to the fact that they have end-autonomous capacities. An end-autonomous vehicle system could independently decide on the ends it ought to pursue. It could decide, for instance, that it would be better to cause a single human death, than a certain number of non-human animal deaths whenever it is forced to choose. We could not predict in advance that it certainly would not reach such a conclusion if it were capable of engaging in its own ethical decision-making and doing so in such a way that it reaches its own conclusions about which ends to pursue. By definition, an end-autonomous ethical AI would be designed to satisfy a set of ethical aims, but if it also rejected those aims and adopted some new set of ends, there could be no guarantee that we would approve of those ends. An objection that should be considered here is that my worry about end-autonomous ethical AI assumes that end-autonomy will give rise to unpredictability, but this is actually controversial. Consider, for instance, the fact that many end-autonomous human persons are highly predictable. One (possibly apocryphal) example of this is Immanuel Kant and his daily walks. It has been said that Kant’s neighbors would set their clocks by Kant’s walks because he followed such a strict routine that he always walked by at (approximately) the same time every day. Now, whether or not this is true, the point is just that human persons are typically predictable despite their end-autonomous capacities. Just consider anyone you have spent a lot of time with, such as a close friend or family member. You can probably reliably predict how they would behave in many types of situations, even ones you have not seen them in before, since you are sufficiently familiar with them. You are familiar with their habits, tastes, social and political views, and other sorts of characteristics that influence their general behavior. If end-autonomous humans are so predictable, then what reason is there to presume that end-autonomous ethical AI would be any different in this respect? Could we not create predictable AI systems of this kind? In response, the main problem with this line of reasoning is its anthropomorphism. While it is true that the only example we currently have of end-autonomous entities is actual human persons, and so perhaps the best evidence we have for predicting how non-human end-autonomous agents might behave is limited to what we observe in humans, it would be a mistake to believe that such evidence is decisive or even very strong. That humans retain their ends on the basis of psychologically entrenched habits, tastes, values, and so on is a distinctive fact about us, and it is plausibly connected to deeply complicated facts about our biology and evolutionary history. I will not pretend to have a good understanding of such complex matters, but we do not require one anyway to appreciate the general point that human beings and AI systems are different in some relatively important ways that could make a difference in how they relate to their ends. Unless they undergo a transformative experience of some kind, a human person will usually stick to their ends, but we simply cannot justifiably be confident that an end-autonomous ethical AI would retain the ends that we supply it with. Next, concerning general end-autonomous ethical AI, things only get worse and perhaps much worse. Now we are to imagine that these kinds of systems could enter into numerous different ethically sensitive contexts and engage in end-autonomous ethical decision-making in each one of these contexts.13 The opportunities for wreaking havoc have now been multiplied! The aforementioned concern about whether we would approve of the ends of end-autonomous ethical AI reemerges here, and it is even worse because again we are imagining that these agents would be operating in a number of contexts, so they would present many more ethical risks. To recap, I have characterized merely safe AI as being AI systems that are designed to be safe while lacking any ethical aims beyond safety, and I have shown that such systems would be problematic since they would present numerous ethical risks that are not related to matters of safety. I then characterized end-autonomous ethical AI as being AI systems that are designed to satisfy ethical aims beyond safety while also possessing end-autonomy, and I showed that these systems would be too risky since their end-autonomous capacities would allow them to consider their ethical aims and potentially opt to reject them. Accordingly, it can be recognized that an appropriate middle ground between these two extremes might be needed, which will be explored in the next section. End-Constrained Ethical AI So far I have argued that both merely safe AI and end-autonomous ethical AI would be problematic. In this section, I will show that there is a kind of ethical AI that could avoid some of the pitfalls of merely safe AI and end-autonomous ethical AI. End-constrained ethical AI would be superior to merely safe AI because they would satisfy ethical aims beyond safety, and they would also be preferable to end-autonomous ethical AI because they would not give rise to the dangers associated with end-autonomy. Regarding their autonomous capacities, while there are a number of different types of autonomy that might be instantiated in AI, I will not take a stand here on which one would be appropriate for ethical AI. Fulfilling our ethical demands could presumably be accomplished in a variety of autonomous ways. My only claim here is that we should avoid end-autonomy in ethical AI in favor of some end-constrained autonomous capacities that make possible the pursuit of ethical ends beyond safety, but also constrain the ends of such AI such that they cannot adopt new ones that we did not design them to pursue. To illustrate, end-constrained ethical AI could be systems that are capable of independent ethical decision-making in various situations, which is to say they could engage in such decision-making without the need for any direct intervention or input from human agents. This decision-making, though, must not involve evaluation of the aims that they are designed to achieve. As I imagine end-constrained ethical AI, then, they would not require any human assistance at the time of decision-making, but they also would not be capable of modifying their ends.14 End-Constrained Implicit Ethical AI Now, it is important to notice that end-constrained ethical AI could count as having ethical aims without possessing any ethical concepts or reasoning capabilities that would involve the deployment of such concepts. They could count as having ethical goals not because they represent their ethical goals as ethical goals, but because the goals they have are in fact ethical ones. These types of artificial agents could help us to satisfy our ethical demands without relying on any explicitly ethical instructions whatsoever. However, it is also possible that end-constrained ethical AI might possess ethical concepts, as Moor’s (2006) explicit ethical agents do. The point here is just that such ethical AI could be designed in either of these two ways. This is notable because many authors have assumed that we need explicit ethical agents for certain ethical tasks, but I reject that view.15 I maintain that implicit ethical AI, which would lack any ethical concepts, could perform virtually any task that explicit ethical agents could do. They could both go beyond safety considerations in their ethical decision-making and behavior, which is what we should want, and what is especially important for my purposes is that they could do so in my stipulated end-constrained way. Here it is useful to consider one of Moor’s (2006: 19) examples of an implicit ethical agent:Another example of a machine that’s an implicit ethical agent is an airplane’s automatic pilot. If an airline promises the plane’s passengers a destination, the plane must arrive at that destination on time and safely. These are ethical outcomes that engineers design into the automatic pilot. Other built-in devices warn humans or machines if an object is too close or the fuel supply is low. Another example of a machine that’s an implicit ethical agent is an airplane’s automatic pilot. If an airline promises the plane’s passengers a destination, the plane must arrive at that destination on time and safely. These are ethical outcomes that engineers design into the automatic pilot. Other built-in devices warn humans or machines if an object is too close or the fuel supply is low. This example illustrates that a machine might fulfill various ethical demands without the need for any explicitly ethical instructions or programming (i.e. instructions or programming that involves ethical concepts such as rightness, wrongness, fairness, etc.). An airplane’s automatic pilot accomplishes this by bringing about certain ethical outcomes that its designers program it to achieve without the machine itself representing those outcomes as being ethically appropriate or right. We merely design the machine to behave or operate in such a way that it fulfills the ethical goals we have for it. If an AI had safety as its only ethical goal (and it could reliably achieve that goal), then perhaps it would be able to complete whatever tasks it might have without harming anyone in the process. However, it would not achieve anything ethically significant beyond that. In fact, it might be argued that this is all that the automatic pilot is accomplishing, transporting the plane’s passengers to their destination safely, and there is nothing else that is ethically relevant about what it is doing. Whether or not that is the correct description of what the automatic pilot is doing, though, my discussion assumes that AI will eventually be placed into contexts that call for ethical aims beyond safety. I claim that we should want end-constrained ethical AI that satisfy such further ethical aims in those sorts of context, and they could be either implicit or explicit ethical agents. For an AI to have further ethical goals beyond safety would be for it to aim at performing various ethically appropriate types of action, such as saving people from harm, engaging in the just distribution of resources (e.g. distributing in accordance with need or desert), and avoiding courses of action that might violate people’s rights. It is not out of the question that sufficiently sophisticated ethical AI might become capable of doing such things, and pace many machine ethicists, they could do so without the need for any explicitly ethical programming, or so I will argue in what follows.16 While I intend to make a case for end-constrained ethical AI in general (implicit or explicit), because many assume that explicit ethical agents are necessary for engaging in ethical behavior beyond merely safe behavior, I face the dialectical burden of establishing that implicit ethical AI have this capability too. So, I will now consider a few positions on the matter. Following Moor’s description, explicit ethical agents are ones that “represent ethics explicitly” and can make “plausible ethical judgments and justify them.” Such agents would be able to deploy ethical concepts in their reasoning. Anderson and Anderson (2007, p. 17) argue that we need explicit ethical agents for it to be possible for machines to be able to determine what the ethically correct actions are in situations that they have not encountered before. Implicit ethical agents, they contend, may serve as a kind of safeguard to prevent unethical behavior, but they would not be capable of determining the right actions in novel scenarios. Also, Formosa and Ryan (2021, p. 847) maintain that explicit ethical agents may be required in at least three kinds of cases: “(1), where inaction will allow harm (e.g., failure to rescue cases), (2) when the safety of two or more parties must be weighed up (e.g., in trolley or robotic triage nurse cases), or (3) when safety is in conflict with other important values such as autonomy (e.g., autonomous refusals to take medicine cases).” In short, both of the two positions just described presume that implicit ethical agents would not be ethically sophisticated enough to engage in ethical behavior in certain sorts of situations, and accordingly implicit ethical agents might not be capable of performing the kinds of ethical actions that I suggest they could. In response, it is far from obvious that implicit ethical agents would not be capable of accomplishing any of the things that the preceding authors mention. I will consider each of them in turn. First, with regard to determining the ethically right action in novel situations, if we think about what it might take for a machine to succeed at this, it could plausibly just be a matter of prioritizing certain considerations in any given situation in order for the machine to figure out what to do, but this need not involve any explicitly ethical descriptions. For instance, a machine might be programmed to prioritize the reduction of harm over the production of pleasure whenever only one of these can be secured, and this might allow the machine to make the right decision in certain cases of conflict, even if the machine has never encountered some such cases before. The machine would not necessarily have to represent its final action as the ethically right one in order for it to perform it or for it count as having arrived at the right action. A particularly restrictive view of right action might say that one has to be motivated by the thought that an action is right in order for that action to actually count as right.17 Whether or not that is true, it seems that the more important question is whether or not an implicit ethical agent could even determine which actions to perform, which we would regard as ethically appropriate, regardless of whether we would count them as performing right actions. While much more could be said on this topic, my point is that an implicit ethical agent could be designed in ways that would enable it to select morally correct actions in novel circumstances. Next, concerning the three types of cases that Formosa and Ryan appeal to, it is not difficult to imagine that an implicit ethical agent actually could play the roles that they suggest an explicit ethical agent would be necessary for. In cases where inaction would allow harm, an implicit ethical agent could be programmed to not only refrain from harming humans, but also to perform whatever positive actions might be necessary to prevent humans from being harmed. None of this would necessarily require such a machine to represent any ethical concepts whatsoever.18 In cases when the safety of two or more parties must be weighed up, again it is not clear why this must involve the deployment of ethical concepts. If two people are in danger, for example, and a machine can only save one of them, then it seems that considerations such as each person’s age and health might be relevant to making the right decision, but neither of those would require the machine to represent any ethical concepts. Finally, in cases when safety is in conflict with other important values such as autonomy, as I mentioned in the previous paragraph a machine might be programmed to prioritize or give more weight to certain kinds of considerations in cases of conflict, and again this need not involve any ethical representations whatsoever. In sum, all that would be necessary for an implicit ethical agent to satisfy our ethical demands in the aforementioned types of cases is that it possesses the representational resources that would enable it to discriminate ethically relevant factors and make appropriate use of them in their decision-making. In this way, they could be capable of achieving ethical outcomes reliably without ever making use of any explicitly ethical concepts. Another key difference between implicit ethical agents and explicit ethical agents that has been argued for has to do with the ability to communicate intentions and reasoning. Scheutz (2017, p. 58) maintains that, “For contexts where informing others of one’s intention and reasoning is crucial, [explicit ethical] agents could then also express their reasoning in natural language,” but implicit ethical agents would not be capable of doing this. One context in which Scheutz claims that this would be an issue is when an assistive robot is designed to work with a vulnerable population (e.g. the elderly). Scheutz (2017, pp. 59–60) writes: [I]t is easy to imagine all kinds of morally challenging situations where assistive agents without moral competence will fail to make adequate decisions (for example, deciding whether to let an elderly person have a drink to improve the person’s depressed mood despite the doctor’s strict order for the person to not have any alcohol). And without the ability for an agent to communicate why it decided as it did, all agent decisions — correct and incorrect — may lead to adverse reactions from the people they purportedly serve. [I]t is easy to imagine all kinds of morally challenging situations where assistive agents without moral competence will fail to make adequate decisions (for example, deciding whether to let an elderly person have a drink to improve the person’s depressed mood despite the doctor’s strict order for the person to not have any alcohol). And without the ability for an agent to communicate why it decided as it did, all agent decisions — correct and incorrect — may lead to adverse reactions from the people they purportedly serve. In response, nothing in the description of an implicit ethical agent rules out the ability for it to explain its decision-making. Since these AI would lack ethical concepts, they could not explain their decision-making by making statements such as, “I behaved in this way because it was the right thing to do” or “I did it because it was fair,” but certainly these types of explanations involving ethical concepts would be inadequate anyway. This is because adequate explanations of an agent’s ethically salient decisions should include naturalistic concepts that make clear why certain courses of action are right or fair (or ethically appropriate in some other way), and an implicit ethical agent could do this without ever mentioning an ethical concept or representing its course of action as instantiating some ethical property such as rightness. Using Scheutz’s example, if an elderly person asked an assistive agent for an explanation of why the agent would not allow them to have a drink, the elderly person would presumably be wholly unsatisfied if the agent responded with, “It would be wrong to let you have a drink.” Instead, what the elderly person would reasonably desire is an explanation of why it would be wrong, and an implicit ethical agent could respond with, “Letting you have a drink would cause you bodily damage (in some specified way and to some specified extent), and it is probable that you would regret having a drink whenever your mood has improved.” Moreover, an implicit ethical agent would not need to reach the conclusion that letting the person have a drink would be wrong in the first place. They would only need to be programmed in such a way that their decision-making would lead them to make that decision and perhaps for the reasons stated above. If the foregoing is correct, then implicit ethical agents are more ethically capable than machine ethicists have seemed to realize. In view of this, I want to emphasize that more work should be done to determine just how much we could accomplish with this type of end-constrained ethical AI. It could turn out that we will need explicit ethical agents in order to have AI that are capable of accomplishing certain complicated ethical tasks.19 Nonetheless, we should still investigate the extent to which implicit ethical agents could secure various ethical goods beyond safety, since it might not be necessary to have explicit ethical agents around for that. It is true that I have not offered a very detailed model or description of how implicit ethical agents would accomplish the various ethical tasks I have mentioned, but this is not what I intended to do. I intended to present an in-principle argument that reveals the capabilities of implicit ethical agents based on considerations about the differences between representing ethical concepts explicitly and achieving ethical goals without the deployment of such concepts. Furthermore, although some machine ethicists have attempted to design certain computational models for explicit ethical agents, none of them are uncontroversial, and they still leave many difficult questions unanswered.20 So, it would be unfair to complain that I have not offered a complete account of how implicit ethical agents are supposed to fulfill various ethical tasks because we do not currently possess a complete and plausible description of how explicit ethical agents could succeed at doing so either. I have merely endeavored to show that it is plausible that implicit ethical agents could satisfy certain ethical demands beyond those of safety, and if I am right, then many machine ethicists have underestimated their ethical capabilities. The Risks of End-Constrained Ethical AI Nearly everyone agrees that if we are going to continue developing AI systems, then we should want those systems to be capable of carrying out various useful tasks for us, such as the ones already performed by devices like smart assistants and other existing technologies. Beyond the completion of various useful tasks, though, one might wonder whether there is more that we should want from AI. Is completing useful tasks for us, and doing so safely, all that we should want, or should we want AI to both safely complete such useful tasks as the ones they already do and extend their usefulness beyond these kinds of tasks to aim for certain ethically valuable outcomes besides safety? We have already seen that merely safe sophisticated AI could present further ethical problems that do not relate to safety, and now the question is whether to design end-constrained ethical AI that might overcome such problems or whether doing so might invite its own set of risks that are not worth assuming. Because of the potential for ethically valuable outcomes, there is good reason to believe that we should want AI systems that not only complete tasks safely, but also work to achieve ethical aims such as the ones mentioned earlier. These included aims like saving people from harm, distributing resources justly, and avoiding courses of action that might violate people’s rights. Insofar as it is possible (and we can determine how) to successfully design end-constrained ethical AI that achieve such ethical aims, then, it seems that we should do so.21 This constitutes the prima facie case for end-constrained ethical AI. Although it might seem obvious at this point that we should prefer end-constrained ethical AI to merely safe AI since the former could avoid the ethical issues that are associated with the latter, there are still some important worries about the project of designing these kinds of ethical AI. It is true that designing such AI to achieve ethical aims could potentially result in ethically valuable outcomes, but attempting to do this introduces some new ethical risks too. An initial worry is that although end-constrained ethical AI would not be capable of changing their ends, they would probably still have some autonomous capacities that could lead them to behave unpredictably. If we were to allow them to engage in independent ethical decision-making, it seems that we would thereby allow for the possibility of unpredictable and bad behavior since we could not be certain that they would pursue our ethical ends in the ways we intend them to.22 Thus, end-constrained ethical AI could still behave badly even though they lack end-autonomy. In response, the extent to which the behavior of such an AI would be predictable would plausibly depend on, inter alia, its design approach. As I have argued in other work, top-down approaches afford us more predictability when it comes to the behavior of ethical AI than bottom-up approaches do, but even if an end-constrained ethical AI were trained via machine learning techniques, its human designers could still significantly determine its general behavior.23 This would be true, for example, if the methods they employed were ones that involved supervised or reinforcement learning. In this way, such an AI might learn how to behave based on huge amounts of training data, but its behavior would still depend on how its designers labeled the data or the feedback it would receive based on its behavior. In addition, while it is true that end-constrained ethical AI could also encounter normative conflicts in which there are conflicting ethical aims, the resolution of those conflicts is not something that they would decide entirely on their own either. We could program them to have certain priority rankings, for instance, whereas an end-autonomous ethical AI would have more leeway insofar as it could independently determine how it will prioritize or treat the various features involved in a given ethical conflict, which could clearly become problematic. Another worry concerns the possibility that we render ourselves more vulnerable to certain kinds of mistreatment by extending beyond the design of merely safe machines. If we simply failed to design end-constrained ethical AI appropriately, then they could end up mistreating us in exactly the sorts of ways that we were trying to avoid by going beyond safety. Not only that, but even more risks could be introduced with the advent of ethical AI that were not present with merely safe AI. To make this point vivid, we can consider a hypothetical firefighter AI. If this AI were designed to merely put out fires without harming anyone in the process, then it would count as a merely safe firefighter AI. If instead it were designed to both put out fires safely and save people from burning buildings (without the ability to assess or change those ends! ), then it would count as an end-constrained ethical firefighter AI. Thus, there are at least two ways of imagining a firefighter AI. When we consider the risks that might be associated with each kind of AI, we can identify some further differences between the two. As it turns out, the end-constrained ethical AI might actually be more risky than the merely safe AI in this case, but only because it has an additional aim beyond putting fires out safely. Whatever the risks the merely safe AI presents by putting fires out safely, the ethical AI would also present those risks insofar as it is engaged in the same activity. The additional risk, then, stems from the fact that the latter AI would also attempt to save people. If we failed to design such an AI appropriately, then instead of saving people from burning buildings, it might seek to guarantee that no one ever exits those buildings, or worse, it might attempt to force more people into them. The point here is just that programming the ethical firefighter AI to carry out an additional ethical task involves an additional set of behaviors that the AI would have to engage in. With these additional behaviors comes the assumption of new risks, yet these would all be avoided by the merely safe firefighter AI.24 But just how plausible is it to suppose that end-constrained ethical AI might end up acting this egregiously, and do the potential benefits of saving many people from harm outweigh the risks? First, I am inclined to contend that the likelihood of such an AI acting egregiously to some extent or other depends on the ethically sensitive context in question. It seems plausible that it would be especially risky to introduce such AI into contexts in which very much is at stake or contexts in which it is very difficult to design AI to behave appropriately for some reasons having to do with the complexity of the context, or both. For contexts in which there is less at stake or it is easier to design AI to behave appropriately, or both, it would not be as likely that an end-constrained ethical AI would behave very egregiously. Next, regarding whether the potential benefits outweigh the risks, I think that providing an adequate answer to this question will depend on some of the considerations just mentioned, and it will also be important to recognize how great the benefits could be if end-constrained ethical AI were introduced into some context and appropriately designed. If we could succeed at developing such ethical AI that behave appropriately in various contexts, then I would certainly conclude that we ought to bring these AI into existence since they would potentially accomplish so much good. The major question, then, is whether we could so succeed, and that is currently open. In light of all of this, I am inclined to assert that the prima facie case for end-constrained ethical AI stands or falls depending on the nature of each ethically sensitive context into which we are considering introducing such AI. Whether or not they should be introduced will depend on whether we could design them in such a way that they could satisfy our ethical aims in each case without imposing too much risk, and again this could differ from one context to the next. Of course, if we could develop an entirely general end-constrained ethical AI that is ethically competent across all contexts, then perhaps that would be even more desirable than narrower ethical AI, but the former possibility seems much less attainable at present than the latter. The Applications of End-Constrained Ethical AI As AI systems in general become more developed and sophisticated, it is probable that these systems will increasingly be inserted into contexts that actually call for ethical behavior and the pursuit of ethical goals beyond safety. If AI systems are actually going to be placed into such ethically sensitive contexts, then certainly we will require more from them than merely safe behavior. A number of contexts could be relevant here, but I will mention just three: medical contexts where triage decisions are made, policing contexts where the use of lethal force must be considered, and academic contexts such as a moral philosophy seminar. I take it that these are concrete examples of contexts in which merely safe AI would not be capable of satisfying our ethical demands because they would lack the kind of decision-making capabilities or processes that would be appropriate for those contexts. To briefly illustrate this point, imagine if a merely safe AI were placed into a medical context involving triage decisions. Such an AI might not cause direct harm to anyone, but since it is not programmed to satisfy any further ethical demand, such as the just distribution of scarce medical resources, it would fail to engage in the appropriate kind of ethical behavior. This seems rather uncontroversial, yet it is noteworthy because it makes especially clear just how limited merely safe AI would be in this type of ethically sensitive context, and similar remarks could be made about merely safe AI in policing contexts and academic contexts. In general, they would not be capable of satisfying our ethical demands in any of these contexts because of their inherent design limitations, and we would need more ethically sophisticated AI to help us achieve our ethical aims in such contexts. In contrast to merely safe AI, end-constrained ethical AI could operate appropriately in the aforementioned types of contexts precisely because they are decision-making contexts that involve ethical aims beyond safety. They are contexts where the decisions being made have significant ethical import, and accordingly an end-constrained ethical AI would be the right sort of AI system to be included in these contexts since they would be designed to engage in behavior that fulfills our ethical goals. Of course the ethical goals themselves are going to vary depending on the ethically sensitive context in question, but this would not necessarily prevent an ethical AI from being able to behave ethically across numerous contexts.25 Whether or not a single such AI could succeed at being generally ethically reliable, though, it is still the case that an ethical AI of some kind would be necessary for satisfying our ethical demands in each kind of ethically sensitive context. To make this point especially vivid, I will briefly consider each of the three example contexts mentioned earlier and explain why an ethical AI would be necessary for each one. First, with regard to medical contexts involving triage decisions, attempting to determine how scarce medical resources should be distributed would be the relevant ethical task, and our ethical demands might include, inter alia, respecting patient autonomy, avoiding unnecessary harms, preserving endangered lives as much as possible by prioritizing those who are in immediate need of treatment, and distributing medical resources fairly.26 It seems immediately clear that triage decision-making involves more than just ensuring the safe distribution of medical resources and prevention of harm. To the extent that patient autonomy should be respected, resources should be distributed fairly, and so on, we will require ethical AI in these medical contexts that can help us to satisfy these associated ethical demands. Secondly, in policing contexts where the use of lethal force must be considered, attempting to determine when to use lethal force would be the relevant ethical task. It seems reasonable that suspects should be kept safe to the extent that they can be, but if others are put at risk by a suspect, then lethal force might be warranted, and considerations beyond safety, such as ones relating to whether the use of lethal force would be just or deserved in a certain kind of situation would certainly be ethically relevant. Again, we would need more than merely safe AI in these kinds of contexts. Thirdly, regarding academic contexts such as a moral philosophy seminar, this is where things could be importantly different. In a moral philosophy seminar, the kinds of deliberations that would take place would concern the very nature of ethics, and this could include reflection on what makes right actions right, what the ontological status of ethical properties is, how we could come to know about such properties, and related matters depending on the topic of the seminar in question. Now, in view of the fact that such a context would require one to reflect on these sorts of ethical questions, it seems plausible that doing so would have to involve the deployment of at least some ethical concepts. If this is correct, then we would actually require an explicit ethical agent for this kind of context, whereas in the previous two kinds of context either implicit or explicit ethical AI seemed fitting. We would need systems that are capable of deploying ethical concepts, and presumably we would need even more than that. For an AI to be able to competently engage in a moral philosophy seminar, such a system would need to be capable of thinking about ethics or at least evaluating ethical (or metaethical) assertions, not just producing ethically appropriate behaviors or invoking ethical concepts but not contemplating them.27 However, despite the allure of that sort of sophisticated ethical AI, such AI might actually require a more demanding type of autonomy than the one(s) instantiated by end-constrained ethical AI. In order to competently engage in reflection on profound or esoteric normative or metanormative topics, end-autonomy could be necessary. This is because of the nature of that type of activity. Deliberating about first-order ethical questions and the very nature of normativity could very well presuppose the possession of robust autonomous capacities, as the deliberating agent would need to be capable of autonomously assessing various ends and rendering judgments about them in order to count as engaging in that type of deliberation at all, and along with this capability the agent in question would be able to consider its own ends that have been programmed into it by its human designers. The further, crucial question then is whether such an agent would also be capable of making its own decisions about whether it would continue to pursue those ends in view of whatever conclusions it reaches about them.28 If so, we should plausibly refrain from developing AI for these kinds of context, as end-autonomous ethical AI are simply too dangerous. Conclusion I conclude by making a few brief remarks on the significance of the topics addressed in this paper. We are moving rapidly into an AI-infused world, that is, a world where increasingly many of our daily activities are ones that involve (oftentimes unbeknownst to us) the operations of AI systems in one way or another. In view of the potential widespread impacts of AI on our lives, it is imperative that we carefully consider the many varieties of AI that we might seek to develop and especially the potential risks and benefits that accompany such systems. This paper is intended to make a step in that direction. It does so by explaining how we might not go far enough with the design of certain types of systems if we stop at considerations of safety and how we might go too far if we design systems with particularly robust ethical capacities, and it also describes and motivates the case for a specific category of AI systems, namely, end-constrained ethical AI. My hope is that AI researchers and the public more generally will take seriously discussions like this one and understand that we must be cautious when it comes to the types of AI we attempt to design. The associated risks and benefits of various AI systems will almost certainly never be obvious, and thus we must investigate such matters thoroughly in advance, lest we develop a kind of AI that is detrimental to us.