Artificial Intelligent Disobedience: Rethinking the Agency of Our Artificial Teammates

Reuth Mirsky · 2025

Read Artificial Intelligent Disobedience: Rethinking the Agency of Our Artificial Teammates by Reuth Mirsky (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22276.txt.

Introduction

Reuth Mirsky argues for a fresh look at how much say smart software should have inside a human team, in a solo paper from 2025. Most team software today obeys orders without question. Mirsky notes that blind obedience can be wasteful or even unsafe when an order rests on a mistake.

The paper sets out a scale of agency levels for such systems. At each level, it asks when a system should push back, warn, or act on its own call. Worked examples show how pushback could help a team rather than hinder it. The paper ends with first bounds and open questions for studying wise disobedience as a skill that team software should be built to have.

Catalog record

CREATORReuth Mirsky
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-22276

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Artificial Intelligent Disobedience: Rethinking the Agency of Our Artificial Teammates Reuth Mirsky Abstract Artificial intelligence has made remarkable strides in recent years, achieving superhuman performance across a wide range of tasks. Yet despite these advances, most cooperative AI systems remain rigidly obedient, designed to follow human instructions without question and conform to user expectations, even when doing so may be counterproductive or unsafe. This paper argues for expanding the agency of AI teammates to include intelligent disobedience , empowering them to make meaningful and autonomous contributions within human-AI teams. It introduces a scale of AI agency levels and uses representative examples to highlight the importance and growing necessity of treating AI autonomy as an independent research focus in cooperative settings. The paper then explores how intelligent disobedience manifests across different autonomy levels and concludes by proposing initial boundaries and considerations for studying disobedience as a core capability of artificial agents. 1 Introduction The field of artificial intelligence is currently abuzz with discussions surrounding “agentic AI” or “AI agents.” However, despite the widespread excitement, the term agent itself often lacks a precise, universally agreed-upon definition within these conversations. Recently, significant focus has shifted towards agents built upon large language models (LLMs), leveraging some reasoning and language understanding capabilities to execute complex tasks, interact with external tools, and learn from feedback [ 63 , 67 , 53 , 66 , 56 ] . This move towards more autonomous, goal-directed LLM systems represents a promising yet challenging frontier in AI development. During this time, AI algorithms have also reached superhuman performance in numerous tasks such as game playing [ 9 , 57 , 62 , 65 ] and text and image processing [ 2 , 15 , 51 ] . On the other hand, there are still significant obstacles that modern AI has yet to overcome. Grosz [ 21 ] proposed a revised Turing Test to create: ” A computer team member that can behave, over the long term and in uncertain, dynamic environments, in such a way that people on the team will not notice that it is not human .” This vision captures not just the importance of task performance, but also fluency, trust, and appropriate initiative in collaborative settings. This work argues that such agency must include the ability to make context-sensitive decisions that may sometimes involve intelligent disobedience . In most existing work on cooperative AI agents and robots, it is assumed that artificial agents should do their best to follow the instructions they are given [ 11 ] . However, as AI agents achieve human-comparable and even superhuman performance in various domains, there is a growing need to reason more deeply about the levels of autonomy appropriate for such agents [ 14 ] . This paper introduces and analyzes a taxonomy of autonomy levels, ranging from zero autonomy (pure obedience) to full autonomy, where the agent may independently generate and revise goals. Each level corresponds to increasing decision-making power and situational flexibility. This paper calls on AI practitioners to extend the agency of AI teammates beyond strict obedience, allowing them to act with increased autonomy when necessary. Consider a guide dog, trained to intelligently disobey: if it is given a clearly unsafe command from its handler, it is taught to disobey it [ 40 ] . The paper will use this example and others to showcase how we might reconsider the agency of artificial teammates, so they can be better collaborators. This paper further contributes to a formalization of intelligent disobedience as a critical capacity in autonomous agents. It identifies levels of disobedience corresponding to the agent’s autonomy level. These levels delineate when, how, and to what extent an agent should override instructions, ranging from reflexive overrides (e.g., avoiding immediate harm) to sophisticated mediations that involve alternative plan generation and moral reasoning. This paper argues that investigating AI agency in cooperative settings as an independent capability should be part of the modern AI research agenda for several reasons. First, it challenges contemporary perceptions regarding the autonomy that should be given to AI agents and robots, especially in situations where such intelligence outperforms human judgment. Changing the autonomy of AI in such cases has the potential to add significant value to the team by making the AI a uniquely contributing member. Second, AI agency forces discussion on the boundaries and restrictions of autonomous AI, a necessary topic as AI capabilities continue to expand. As technology progresses and AI systems gain greater autonomy, regulations enabling AI to make decisions must be updated [ 7 , 35 ] . As insulin pumps, autonomous vehicles, and surgical robots gain more autonomy to act without human intervention, it becomes crucial to ensure that these systems act intelligently and appropriately in their context. Finally, this work explores the boundaries of disobedience: when disobedience is warranted, when it becomes a liability, and how to design agents that can transparently mediate between global safety principles and local human instructions. Using frameworks like the five-step model proposed by Mirsky and Stone [ 40 ] , this work situates intelligent disobedience within broader efforts in AI alignment and safety. The goal of this work is to equip artificial agents not only with competence but with the moral and contextual sensitivity needed to be truly effective teammates. 2 Defining AI Agency To start discussing AI agency, this paper follows the definition of AI by Nilsson [ 43 ] : “ Artificial intelligence is that activity devoted to making machines intelligent, and intelligence is that quality that enables an entity to function appropriately and with foresight in its environment ”. What “to function appropriately” entails highly depends on context. Generally, it refers to systems exhibiting autonomous behavior, capable of (1) perceiving their environment, (2) making decisions, and (3) taking actions to achieve specific goals [ 50 , 64 ] . Moreover, creating truly effective AI agents necessitates equipping them with persistent memory that carries context and state across separate interactions, preventing each call from being a disconnected event. This technical requirement echoes deep philosophical considerations about identity and memory that are beyond the scope of this work [ 30 , 60 ] . Out of this discussion, we need to retain that an agent has a persistent mission : a goal or a set of objectives it is inherently meant to achieve. This persistent mission is a crucial capability of appropriately functioning collaborative AI systems: A guide robot must not let a visually impaired person cross a road when a car is approaching; A medical management software should alert a physician who prescribes a patient a seemingly wrong dosage of medicine; and for a service robot, it might be preferable to interrupt its owner and ask for clarifications instead of executing a wrong task. These examples show that AI agency is more than a set of restrictions but rather a deliberation process that can lead to different outcomes under different circumstances. Taking the five levels of vehicle autonomy, this paper expands the scale system to artificial autonomy, particularly focusing on the ability to make decisions independently of a human teammate. At all autonomy levels above 0, the agent is assumed to be adhering to its persistent mission and to stay

Read the complete text