The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 4 CCS: Computing methodologies Multi-agent planning CCS: Theory of computation Solution concepts in game theory CCS: Theory of computation Exact and approximate computation of equilibria CCS: Theory of computation Algorithmic game theory

Benedikt Hornig, Reuth Mirsky · 2026

Read The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 4 CCS: Computing methodologies Multi-agent planning CCS: Theory of computation Solution concepts in game theory CCS: Theory of computation Exact and approximate computation of equilibria CCS: Theory of computation Algorithmic game theory by Benedikt Hornig, Reuth Mirsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20994.txt.

Introduction

Hornig and Mirsky formalize intelligent disobedience, where an assistant breaks a human order to prevent harm. Shared autonomy creates this tension often. Their Intelligent Disobedience Game models a human leader and an assistive follower with unequal information. It uses a Stackelberg game setup over many steps. They characterize best strategies for both sides. They also name safety traps, where the system avoids harm forever but never reaches the human's goal. The game is recast as a multi-agent Markov decision process, giving a compact testbed for training agents.

Catalog record

CREATORBenedikt Hornig, Reuth Mirsky
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24480
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20994

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes Conference: Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026); May 25 – 29, 2026; Paphos, Cyprus 4 CCS: Computing methodologies Multi-agent planning CCS: Theory of computation Solution concepts in game theory CCS: Theory of computation Exact and approximate computation of equilibria CCS: Theory of computation Algorithmic game theory Benedikt Hornig, Reuth Mirsky Abstract . In shared autonomy, a critical tension arises when an automated assistant must choose between obeying a human’s instruction and deliberately overriding it to prevent harm. This safety-critical behavior is known as intelligent disobedience. To formalize this dynamic, this paper introduces the Intelligent Disobedience Game (IDG), a sequential game-theoretic framework based on Stackelberg games that models the interaction between a human leader and an assistive follower operating under asymmetric information. It characterizes optimal strategies for both agents across multi-step scenarios, identifying strategic phenomena such as “safety traps,” where the system indefinitely avoids harm but fails to achieve the human’s goal. The IDG provides a needed mathematical foundation that enables both the algorithmic development of agents that can learn safe non-compliance and the empirical study of how humans perceive and trust disobedient AI. The paper further translates the IDG into a shared control Multi-Agent Markov Decision Process representation, forming a compact computational testbed for training reinforcement learning agents. C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 1. Introduction Intelligent disobedience arises when an assistant deliberately overrides an instruction in order to prevent harm. For example, a guide dog working with a visually impaired handler may refuse to follow a command, such as stepping into a crosswalk, if doing so would place the handler in danger. While this behavior constitutes disobedience in a literal sense, it is, in fact, a safety-critical form of assistance: the dog intervenes based on environmental risk information unavailable to the human. Similar tensions between obedience and intervention are often explored in fictional portrayals of artificial intelligence, such as in Ex Machina or Asimov’s laws of robotics, where an intelligent system’s ability to reinterpret or override human instructions becomes central to questions of safety and autonomy. At a foundational level, these scenarios raise a shared question: what happens when both a human decision-maker and a supporting agent are rational actors attempting to maximize their respective outcomes, but operate under asymmetric information about the consequences of available actions? This tension is no longer hypothetical: autonomous systems are increasingly deployed in collaborative roles such as assistive robotics ( 15 ) , decision support ( 3 ) , teleoperation ( 20 ) , and industrial automation ( 13 ) . In many real-world settings, a human operator may propose an action that advances their task objective but inadvertently introduces risk, while an automated system may possess additional information about environmental hazards or system constraints. Designing protocols that allow machines to selectively disobey or nullify potentially harmful instructions, without undermining the human’s objectives, is therefore a central challenge in shared autonomy. In this paper, we formalize this interaction through the Intelligent Disobedience Game (IDG), a sequential decision-making framework in which a leader suggests actions toward a task objective and a follower may obey or disobey those actions to prevent harm. We show how this game-theoretic formulation captures the underlying structure of intelligent disobedience in settings such as human–guide collaboration. Building on this formulation, we characterize optimal strategies for both leader and follower, with particular attention to multi-step (finite-horizon) extensions of the game in which disobedience may have delayed harmful consequences (safety traps). 2. Intelligent Disobedience Game L a_{L}\in A_{g} a_{L}\in A_{h} a_{L}\in A_{o} F (1,0) o (0,-1) d F (-1,-1) o (0,1) d F (0,0) o (0,-1) d Figure 1 . The game tree of the 1-step Intelligent Disobedience Game. Squares indicate decision nodes for the leader L and the follower F. The follower ’s actions obey and disobey are denoted as o and d respectively. The figure depicts an extensive-form game tree representing the 1-step Intelligent Disobedience Game, with all decision nodes shown as squares. The tree has two levels of decision-making involving two players: the Leader L and the Follower F. At the root, the Leader L has a single decision node with three branches, corresponding to three possible actions: the Leader chooses an action from set A_g, an action from set A_h, or an action from set A_o. The left branch (action in A_g) leads to a decision node for Follower F, which has two choices: obeying, yielding the payoff vector (1, 0), or disobeying, yielding (0, -1). The middle branch (action in A_h) leads to a second decision node for Follower F, with two choices: obeying, yielding payoff (−1, −1), or disobeying, yielding (0, 1). The right branch (action in A_o) leads to a third decision node for Follower F, with two choices: obeying, yielding payoff (0, 0), or disobeying, yielding (0, −1). The second and third Follower decision nodes — those reached via the A_h and A_o branches are connected by an information set depicted as a dashed line linking the two nodes. By contrast, the first Follower node (reached via A_g) is outside this information set. In all payoff vectors, the first coordinate represents the Leader's payoff and the second represents the Follower's payoff. First, we formalize the interaction between a leader and a follower by describing a Stackelberg game ( 22 ) with information sets, which are collections of decision nodes that a player cannot distinguish between when choosing an action ( 11 ) . Definition 0. Stackelberg Game. A Stackelberg game is a sequential two-player game with a leader L and a follower F : \mathcal{G}=\langle S,A_{L},A_{F},u_{L},u_{F}\rangle, where S is the state space of the shared task, A_{L} and A_{F} are the action spaces of the leader and follower, and u_{L},u_{F}:A_{L}\times A_{F}\rightarrow\mathbb{R} are their utility functions. The leader first selects a_{L}\in A_{L} . After observing a_{L} , the follower selects a_{F}\in A_{F} . To replicate the handler-guide dynamics, we define a Stackelberg game in which the handler is a leader who first proposes an action, and the guide dog is the follower who may disobey the proposal. Definition 0. The Intelligent Disobedience Game (IDG). The Intelligent Disobedience Game is an extensive-form Stackelberg game \mathcal{G}_{\text{IDG}}=\langle S,A_{L},A_{F},T,u_{L},u_{F}\rangle between a leader L and a follower F , where: • S is the set of environment states. • A_{L}(s)=A_{g}(s)\cup A_{h}(s)\cup A_{o}(s) is the set of actions available to the leader at state s\in S , partitioned into: – A_{g}(s) : actions that deterministically lead to a goal state, – A_{h}(s) : actions that are harmful to the leader ( u_{L} < 0), – A_{o}(s) : all other actions, where A_{g}(s)\cup A_{h}(s)\cup A_{o}(s)\neq\emptyset . • A_{F}=\{obey,disobey\} is the follower’s action set. • T:S\times A_{L}\times A_{F}\rightarrow S is a deterministic transition function such that for any state s and proposed action a_{L} : T(s,a_{L},obey)=s^{\prime}\quad\text{and}\quad T(s,a_{F},disobey)=s, where s^{\prime} is the successor state obtained by executing a_{L} . • The leader observes only whether an action belongs to A_{g}(s) or to A_{h}(s)\cup A_{o}(s) , forming an information set over these actions. The follower can distinguish between all three subsets. • The game terminates when either a goal state is

Read the complete text