Deceptive Sequential Decision-Making via Regularized Policy Optimization

Yerin Kim, Alexander Benvenuti, Bo Chen, Mustafa Karabag, Abhishek Kulkarni, Nathaniel D. Bastian, Ufuk Topcu, Matthew Hale · 2025

Read Deceptive Sequential Decision-Making via Regularized Policy Optimization by Yerin Kim, Alexander Benvenuti, Bo Chen, Mustafa Karabag, Abhishek Kulkarni, Nathaniel D. Bastian, Ufuk Topcu, Matthew Hale (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18803.txt.

Introduction

Yerin Kim and seven co-authors built in 2025 a frame for autonomous systems that must hide their true goals from watchers. Enemies can learn a system's reward by watching its choices, using a known learning method in reverse. The team's frame not only hides the reward but actively plants false beliefs about it.

Three styles of deception are defined. One leads the watcher to any wrong answer, one to a chosen wrong answer, and one to hold two answers as equally likely. Each style is built into the system's planning as a penalty term. The work gives designers a menu of tricks matched to how much misdirection they need.

Catalog record

CREATORYerin Kim, Alexander Benvenuti, Bo Chen, Mustafa Karabag, Abhishek Kulkarni, Nathaniel D. Bastian, Ufuk Topcu, Matthew Hale
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18803

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deceptive Sequential Decision-Making via Regularized Policy Optimization Yerin Kim, Alexander Benvenuti, Bo Chen, Mustafa Karabag, Abhishek Kulkarni, Nathaniel D. Bastian, Ufuk Topcu, Matthew Hale Abstract Autonomous systems are increasingly expected to operate in the presence of adversaries, though adversaries may infer sensitive information simply by observing a system. Therefore, present a deceptive sequential decision-making framework that not only conceals sensitive information, but actively misleads adversaries about it. We model autonomous systems as Markov decision processes, with adversaries using inverse reinforcement learning to recover reward functions. To counter them, we present three regularization strategies for policy synthesis problems that actively deceive an adversary about a system’s reward. “Diversionary deception” leads an adversary to draw any false conclusion about the system’s reward function. “Targeted deception” leads an adversary to draw a specific false conclusion about the system’s reward function. “Equivocal deception” leads an adversary to infer that the real reward and a false reward both explain the system’s behavior. We show how each form of deception can be implemented in policy optimization problems and analytically bound the loss in total accumulated reward induced by deception. Next, we evaluate these developments in a multi-agent setting. We show that diversionary, targeted, and equivocal deception all steer the adversary to false beliefs while still attaining a total accumulated reward that is at least 98 % 98\% of its optimal, non-deceptive value. I Introduction Autonomous systems are used in various forms of vital infrastructure, including manufacturing systems [ monostori2016cyber , arkin1990autonomous ] , autonomous vehicles [ chen2017cyber , guo2022cyber ] , and smart power grids [ yu2016smart ] . One challenge in using these systems is that they can be observed by external parties that may seek to uncover and exploit sensitive information [ qayyum2020securing , SHEEHAN2019523 ] . For example, an adversary may observe the daily routines of an individual’s autonomous vehicle, which can give insight into when that individual is home or away, who their close associates are, and other sensitive information. In military or cybersecurity contexts where strict confidentiality is required, such observations can result in severe operational loss [ 10.1145/989.991 ] . For example, if an adversary infers the underlying objective in troop movements or resource allocation, then they might exploit this knowledge to make targeted attacks [ mhara2024cyber ] . Moreover, recurring patterns on supply channels or communication frequencies can unintentionally disclose strategic priorities [ 9152764 ] . Unfortunately, such observations are often unavoidable, in the sense that a user cannot stop someone else from physically observing them. Thus, another approach is needed to reduce the leakage of sensitive information. One way of mitigating risk is by adopting privacy to protect sensitive data. Traditional privacy implementations often induce uncertainty, thereby giving adversaries ambiguous information [ benvenuti2023differentially , 9304015 ] . However, adversaries may still glean sensitive information despite the implementation of privacy protections [ 9833672 , 9519418 ] , and simply inducing uncertainty may not be sufficient to deter an adversary or mitigate the effects of their efforts. For example, an adversary may still predict specific times that a user is not at home, even if the user’s exact departure and arrival times are uncertain. As a result, interest has arisen in the development of deceptive techniques that drive observers to draw incorrect conclusions about the systems that they observe [ karabag2021deception , mceneaney2005deception , lv2024optimal ] . In this paper, we develop and analyze three forms of deception for sequential decision systems. We model such systems as Markov decision processes (MDPs), and we seek to deceive an adversary about an MDP’s reward function, which encodes its objectives and intentions. The adversary is modeled as using inverse reinforcement learning (IRL) to attempt to infer objectives, and we introduce three types of deception to counter these efforts. The first is “diversionary” deception, which seeks to make an adversary draw any incorrect conclusion about a system’s reward. The second is “targeted” deception, which seeks to make an adversary draw a particular incorrect conclusion about a system’s reward. The third is “equivocal” deception, which seeks to make an adversary draw a false conclusion about the reward function that introduces ambiguity between two different objectives. For each, we formulate tractable optimization problems for policy synthesis that include regularizer terms that implement each type of deception. In detail, our contributions are: • We introduce “diversionary deception”, “targeted deception” and “equivocal deception” and formulate a family of optimization problems for synthesizing decision policies that implement each one (Definitions 5 , 6 , and 7 and Optimization Problems 4 , 5 , and 7 ). • We analytically bound the loss in total accumulated reward that is caused by behaving deceptively in terms of a user-specified parameter that is used to implement each type of deception (Theorems 1 , 2 and 3 ). • We validate the deceptiveness and performance of our method in numerical simulations of a network defense problem, and we empirically assess deception/performance tradeoffs (Section IV ). Existing work in [ kim2024defining ] has presented related definitions of deception and empirical results for them, but to the best of our knowledge the current paper is the first to analytically characterize these types of deception and the tradeoffs they create. In addition, this work expands the empirical results by exploring deception and performance under different algorithms of inverse reinforcement learning, ensuring their reliability in applications. I-A Related Works Privacy has been widely studied for protecting sensitive information of decision systems at runtime. For example, the work in [ yazdani2022differentially , hawkins2020differentially ] adopted differential privacy to protect agents’ state trajectories in multi-agent systems. By using differential privacy, the authors in [ benvenuti2023differentially ] privatize reward functions in policy synthesis for multi-agent MDPs, while those in [ chen2023differentialsymbolic , chen2023differential ] ensure protection of non-numeric or symbolic data. An entropy-based approach has been considered as another method of protecting information. In [ karabag2019least ] and [ savas2019entropy ] , the information leaked to the observer is minimized to reduce the observer’s ability to infer certain characteristics of agents, such as their transition probabilities. As described above, while privacy can induce uncertainty in an observer, a need has arisen for deceptive decision strategies that deliberately induce false beliefs in an observer. For example, the authors in [ abdulhai2024defining ] defined deception in a partially observable speaker-listener problem, and deception is formalized as the regret that the listener incurs by listening to the speaker. Deceptive strategies of agents are explored in [ karabag2021deception ] and [ karabag2022exploiting ] in environments in which a supervisor provides a reference policy for an agent to follow, and the agent attempts to follow a different policy while giving the appearance of following the policy that was specified by the supervisor. The current paper differs from privacy-based approaches because our focus is on deliberately inducing incorrect inferences, rather than generating uncertainty. In addition, this work is different from [ karabag2021deception ] and [ karabag2022exploiting ] because we consider a system operating under observation of adversaries rather than a supervisor. Thus, no reference policy is given to the agents being observed in this work, and the observer has no baseline expectations for the agents’ behavior. We differ from [ abdulhai2024defining ] because we define deception

Read the complete text