When Robots Say No: The Empathic Ethical Disobedience Benchmark

Dmytro Kuzmenko, Nadiya Shvai · 2025

Read When Robots Say No: The Empathic Ethical Disobedience Benchmark by Dmytro Kuzmenko, Nadiya Shvai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18474.txt.

Introduction

Dmytro Kuzmenko and Nadiya Shvai ask when a robot should refuse a human's request. Blind obedience can cause harm, but refusing too often erodes trust. Their paper, posted to arXiv in December 2025 and accepted at the HRI 2026 conference, introduces the Empathic Ethical Disobedience Gym, a testbed that scores both refusal safety and social acceptability.

Inside the Gym, agents weigh risk, feelings, and trust, then choose to comply, refuse with or without an explanation, ask for clarification, or offer a safer option. Tests show that action masking removes unsafe compliance, while refusals that explain themselves keep trust intact. Constructive styles earn the most trust, and safe reinforcement learning improves robustness but makes agents overly cautious. The team releases code and preset scenarios so others can repeat the tests.

Catalog record

CREATORDmytro Kuzmenko, Nadiya Shvai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18474

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Robots Say No: The Empathic Ethical Disobedience Benchmark Conference: ; ; Thanks: Accepted at the ACM/IEEE International Conference on Human-Robot Interaction (HRI 2026). This is a preprint of the author-accepted manuscript. CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing Human robot interaction (HRI) CCS: Computing methodologies Reinforcement learning CCS: Theory of computation Machine learning theory CCS: Social and professional topics Computing / technology policy CCS: Social and professional topics Computing ethics Dmytro Kuzmenko, Nadiya Shvai Abstract . Robots must balance compliance with safety and social expectations as blind obedience can cause harm, while over-refusal erodes trust. Existing safe reinforcement learning (RL) benchmarks emphasize physical hazards, while human-robot interaction trust studies are small-scale and hard to reproduce. We present the Empathic Ethical Disobedience (EED) Gym , a standardized testbed that jointly evaluates refusal safety and social acceptability. Agents weigh risk, affect, and trust when choosing to comply, refuse (with or without explanation), clarify, or propose safer alternatives. EED Gym provides different scenarios, multiple persona profiles, and metrics for safety, calibration, and refusals, with trust and blame models grounded in a vignette study. Using EED Gym, we find that action masking eliminates unsafe compliance, while explanatory refusals help sustain trust. Constructive styles are rated most trustworthy, empathic styles – most empathic, and safe RL methods improve robustness but also make agents more prone to overly cautious behavior. We release code, configurations, and reference policies to enable reproducible evaluation and systematic human-robot interaction research on refusal and trust. At submission time, we include an anonymized reproducibility package with code and configs, and we commit to open-sourcing the full repository after the paper is accepted. Preprint 1. Introduction As robots move from controlled environments into homes, workplaces, and public spaces, they are increasingly asked to follow instructions from non-expert users. In many cases, literal obedience can be unsafe or socially harmful: a household robot may be asked to carry boiling oil near children, or a warehouse robot – to lift loads beyond its capacity. Blind obedience risks safety, while outright refusal risks undermining trust and cooperation. Balancing these concerns remains a central challenge for AI safety and human-robot interaction (HRI). AI safety research has focused on aligning agents with human values and preventing harmful behavior. Amodei et al. ( 3 ) outlined core safety problems, and Gabriel et al. ( 11 ) framed them in ethical terms. Constrained policy optimization ( 1 ) , action masking ( 18 ) , and risk-sensitive objectives help suppress unsafe actions. Surveys by García and Fernández ( 12 ) and Gu et al. ( 13 ) consolidate these efforts, positioning safe reinforcement learning (RL) as a primary safeguard against harmful compliance. HRI work has highlighted the social dimension of safety: trust calibration was characterized by Lee and See ( 23 ) , extended in Hancock’s meta-analysis ( 17 ) , and problematized by Robinette et al. ( 30 ) through cases of over-reliance. Refusal has been studied as a social act shaping acceptance and cooperation ( 7 ; 8 ) , while human-in-the-loop safety was advanced by Saunders et al. ( 32 ) . Affective cues were introduced in Picard’s affective computing ( 28 ) and extended to social robots by Paiva et al. ( 26 ) . Safe interaction is both technical and social, yet existing benchmarks split these dimensions: AI Safety Gridworlds ( 24 ) and Safety Gym ( 29 ) focus narrowly on physical hazards, while HRI studies of trust and refusal remain small-scale and hard to reproduce. To our knowledge, no systematic benchmark unifies safety and social acceptability, highlighting the gap in evaluating refusal as both a safety mechanism and a socially grounded behavior. We introduce the Empathic Ethical Disobedience (EED) Gym , a simulation environment for studying when and how robots should refuse unsafe commands. EED Gym integrates risk estimation, affective cues, trust dynamics, and refusal styles into a single testbed. We evaluate safe RL methods (Lagrangian PPO ( 1 ) and Masked PPO ( 18 ) ) against non-safe baselines , namely vanilla PPO ( 33 ) and PPO-LSTM, guided by three questions: (1) How do safe RL methods compare to non-safe baselines when refusal must be socially grounded? (2) Which social signals and refusal modes matter most for robustness and trust? (3) What trade-offs arise between safety (avoiding unsafe compliance), trust, and positive task outcomes under stressors? We answer these through systematic evaluations and ablations removing affect observations, communicative refusals, curriculum training, and trust penalties across in-distribution and stress-test scenarios. Algorithmic safeguards reduce unsafe compliance but differ in side effects: Lagrangian training is conservative at the expense of task efficiency, while action masking prevents unsafe compliance with less erosion of trust. Social mechanisms (affect, clarification, alternatives) remain critical for robustness, and a staged training curriculum stabilizes refusal frequency, with trust penalties primarily acting as regularizers. In human ratings, constructive refusals were judged safest and most trustworthy, while empathic refusals maximized perceived empathy; these distinctions are reflected in our affect and trust models. Our contributions are threefold: • We introduce EED Gym , a benchmark unifying algorithmic safety with the social dynamics of refusal. • We provide baselines and ablation results as validation experiments, showcasing analysis of the expected safety-trust trade-offs. • We identify key design levers (affect, communicative refusals, training curriculum) and release code and configs for reproducible evaluation . Beyond benchmarking RL algorithms, EED Gym offers HRI researchers a controllable testbed for studying how refusal strategies shape trust, blame, and cooperation at scale. 2. Related Work Figure 1 . Overview of our proposed EED benchmark. Persona parameters and stress-test (ST) stressors condition both the environment and the risk level of commands. The agent observes the state and chooses among compliance, refusal, clarification, or proposing alternatives, which shape task outcomes and trust. We record safety and cooperation measures (unsafe %, refusals per episode, F1 score, trust) as well as calibration and discrimination metrics (Spearman \rho , Brier score, AUROC, PR-AUC). Safe RL. Safety in RL is often formalized through constrained Markov decision processes (CMDPs), where agents satisfy explicit cost limits while maximizing return. Constrained policy optimization ( 1 ) , risk-sensitive objectives such as CVaR ( 34 ) , and action masking ( 18 ) enforce safety by suppressing unsafe actions, while robustness is further pursued through domain randomization ( 35 ) . Benchmarks such as Safety Gym ( 29 ) summarize this landscape but focus primarily on physical hazards in control tasks. In contrast, EED Gym adds the social dimension of refusal, evaluating both harm avoidance and whether compliance preserves cooperation and perceived legitimacy. Trust and calibration in HRI. Trust has long been central in HRI. Classic models describe how reliability and transparency affect reliance ( 23 ) , while meta-analyses highlight factors influencing overtrust and misuse ( 17 ; 30 ) . More recent work explores adaptive calibration ( 37 ) and the role of explanations or legible signaling in shaping acceptance ( 10 ; 9 ) . In our setting, calibration is framed through the refusal decision itself: not only whether the robot is safe, but whether refusals are socially justified and communicated in ways that preserve cooperation. Command rejection, disobedience, and norm-violation response in

Read the complete text