Quantum-Inspired Reinforcement Learning in the Presence of Epistemic Ambivalence

Alireza Habibi, Saeed Ghoorchian, Setareh Maghsudi · 2025

Read Quantum-Inspired Reinforcement Learning in the Presence of Epistemic Ambivalence by Alireza Habibi, Saeed Ghoorchian, Setareh Maghsudi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04219.txt.

Introduction

Alireza Habibi, Saeed Ghoorchian, and Setareh Maghsudi published a paper in 2025 on decision making when the evidence itself argues both ways. Ordinary uncertainty fades as data arrives, but ambivalence, born of conflicting experiences, can survive fresh knowledge and still sway choices. Standard learning frameworks have no slot for that state.

The authors build one, borrowing the mathematics of quantum states to hold conflicting evidence in a single object. Their decision process scores every outcome's probability and reward through quantum-style measurement, and they prove an optimal policy exists within it. A modified greedy learning rule accompanies the framework. In two test problems, a two-state task and a lattice task, the agent still converged to the best policy while ambivalence persisted.

Catalog record

CREATORAlireza Habibi, Saeed Ghoorchian, Setareh Maghsudi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04219

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantum-Inspired Reinforcement Learning in the Presence of Epistemic Ambivalence Alireza Habibi, Saeed Ghoorchian, Setareh Maghsudi Abstract The complexity of online decision-making under uncertainty stems from the requirement of finding a balance between exploiting known strategies and exploring new possibilities. Naturally, the uncertainty type plays a crucial role in developing decision-making strategies that manage complexity effectively. In this paper, we focus on a specific form of uncertainty known as epistemic ambivalence (EA), which emerges from conflicting pieces of evidence or contradictory experiences. It creates a delicate interplay between uncertainty and confidence, distinguishing it from epistemic uncertainty that typically diminishes with new information. Indeed, ambivalence can persist even after additional knowledge is acquired. To address this phenomenon, we propose a novel framework, called the epistemically ambivalent Markov decision process (EA-MDP), aiming to understand and control EA in decision-making processes. This framework incorporates the concept of a quantum state from the quantum mechanics formalism, and its core is to assess the probability and reward of every possible outcome. We calculate the reward function using quantum measurement techniques and prove the existence of an optimal policy and an optimal value function in the EA-MDP framework. We also propose the EA-epsilon-greedy Q-learning algorithm. To evaluate the impact of EA on decision-making and the expedience of our framework, we study two distinct experimental setups, namely the two-state problem and the lattice problem. Our results show that using our methods, the agent converges to the optimal policy in the presence of EA. † † affiliationnotice: Equal contribution 1 Introduction Reinforcement Learning (RL) models an agent facing the exploration-exploitation dilemma: While being uncertain about some specific factors that determine the actions’ outcomes, the agent selects between an action that, according to her current belief, maximizes the reward and a seemingly suboptimal one, which, however, can result in gaining some information that increases future rewards. The type of uncertainty plays a crucial role in solving the exploration-exploitation dilemma. Thus, the core challenge is to efficiently understand the environmental uncertainties. So far, the literature mainly emphasizes two primary forms of uncertainty: Aleatoric uncertainty and epistemic uncertainty ( Liu et al., 2024 ; Lockwood & Si, 2022 ; Kahn et al., 2017 ; Lütjens et al., 2019 ) , described below. • Aleatoric uncertainty arises from the inherent randomness or stochastic nature of the environment in which the agent operates, such as randomness in reward generation processes, or state transitions ( Zennaro & Jøsang, 2020 ; Clements et al., 2019 ) ; as such, it can neither be reduced nor predicted before starting an experiment. • Epistemic uncertainty results from the agent’s limited knowledge like incomplete understanding of the environment or an inadequate representation of some of the characteristics of the problem. This uncertainty encompasses various aspects of the RL problem, including uncertainty regarding the true model of the environment, insufficient knowledge about the outcomes of specific actions in certain states, and ambiguity concerning the optimal policy due to incomplete exploration of the state space ( Liu et al., 2024 ) . The agent can reduce epistemic uncertainty if relevant information becomes available. Effective decision-making necessitates that the agent adeptly manages both aleatoric and epistemic uncertainties, as this enhances performance and leads to more informed decisions in Reinforcement Learning (RL). Besides aleatoric and epistemic uncertainties, other obstacles might hinder optimal decision-making. One is ambivalence , which can inhibit the decision-making progress. Ambivalence differs from uncertainty in that it persists even after the information becomes available. Consequently, traditional approaches to modeling, quantifying, and addressing uncertainty may be inadequate or suboptimal in effectively managing ambivalence as a distinct challenge ( Lam & Sherman, 2020 ) . In this work, we consider a novel form of uncertainty, called as Epistemic Ambivalence (EA). EA occurs when multiple interpretations, explanations, or courses of action are possible, particularly in cases where available when the data does not clearly support one viewpoint over another. In epistemology, which studies the nature and boundaries of knowledge, EA highlights the inherent limitations and complexities involved in understanding and reasoning about the world ( Amaya, 2021 ; Williamson, 2021 ; Lam, 2013 ) . While traditional uncertainty represents imperfect knowledge, EA represents a cognitive state in which conflicting pieces of evidence coexist, resulting in an uncertain decision-making process. Individuals may experience uncertainty or skepticism when supplied with contrasting or ambiguous information. For example, on investment decisions, a person may face conflicting information about a specific stock, which leads to EA. Even after performing extra research and gathering additional data, the person may remain uncertain about whether to buy or sell the stock because of the ongoing presence of conflicting evidence. Such examples illustrate how, when faced with contradictory data from credible sources, an agent may struggle to make an optimal decision, reflecting the nature of EA. The persistence of this uncertainty challenges traditional decision-making intuitions, highlighting the complexity of navigating conflicting evidence. Thus, a deeper understanding of EA can foster more flexible and adaptive decision-making strategies in complex and uncertain environments. In quantum mechanics, quantum particles can exist in multiple states at the same time because of the principle of superposition, with their state remaining indeterminate until observed ( Nielsen & Chuang, 2010 ) . This principle makes an interesting connection to decision-making scenarios, where agents face uncertainty caused by conflicting information from diverse sources. In this paper, we exploit the analogy between quantum mechanic states and EA to develop a theoretical framework for decision-making under EA and a policy. Roughly stated, we map each piece of information to a quantum basis state and construct an EA quantum state that incorporates all of the information. This mapping, accompanied by using Dirac notation and mathematical formulations derived from quantum mechanics, assists the decision-makers in quantifying EA, i.e., calculating the probability of each option, hence enabling informed and effective decision-making. We propose the EA-MDP framework to understand and control EA. We show that the optimal policy exists within this framework. Additionally, we introduce the EA-epsilon-greedy Q-learning algorithm to evaluate the impact of EA on decision-making. Two experiments are conducted to evaluate the performance and effectiveness of our approach. The structure of the paper is as follows. In Section 2 , we provide an overview of the relevant literature on uncertainty in RL, its application in quantum mechanics, and vice versa. Section 3 provides a brief introduction to the principles of quantum mechanics and superposition. Section 4 outlines the problem formulation and implementation of EA by using quantum mechanics. In Section 5 , we present the theoretical results. Section 6 details the experimental evaluation using two distinct problems, where the agent’s reward is calculated based on EA uncertainty, followed by the training process of the agent. Finally, Section 7 provides concluding remarks. 2 Related Work Uncertainty exists in various elements of RL, including observations, actions, transition dynamics, and rewards ( Kakade & Langford, 2002 ; Puterman, 2014 ; Boutilier et al., 2000 ; Shapiro et al., 2021 ; Bertsekas & Tsitsiklis, 1996 ; Wang et al., 2016 ) . Wang et al. (2020) developed a robust reinforcement learning framework that allows agents to learn in environments with noise, where they can only observe perturbed rewards. Zhang et al. (2020) studied the vulnerability of Deep Reinforcement

Read the complete text