Quantum reinforcement learning in dynamic environments
Oliver Sefrin, Manuel Radons \nomail, Lars Simon \nomail, Sabine Wölk \nomail · 2025
Read Quantum reinforcement learning in dynamic environments by Oliver Sefrin, Manuel Radons \nomail, Lars Simon \nomail, Sabine Wölk \nomail (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01691.txt.
Introduction
Oliver Sefrin and co-authors test a quantum-assisted learning agent in worlds that change over time, in this 2025 paper. Earlier work paired quantum amplitude amplification with standard reinforcement learning, cutting the samples needed to learn some tasks. That hybrid agent, though, had only faced still worlds, where the rules never shift. Real settings drift: rewards move, and the best act changes. The authors ask whether the quantum edge survives drift.
They add a dissipation mechanism, a way for the agent to let old knowledge fade as the world moves. Trials then pit the upgraded hybrid against a plain classical agent in shifting tasks. The comparison weighs learning speed and final performance. Showing the method works in motion, not just at rest, widens its claim on real problems.
Catalog record
| CREATOR | Oliver Sefrin, Manuel Radons \nomail, Lars Simon \nomail, Sabine Wölk \nomail |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01691
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantum reinforcement learning in dynamic environments Oliver Sefrin, Manuel Radons \nomail, Lars Simon \nomail, Sabine Wölk \nomail Abstract Combining quantum computing techniques in the form of amplitude amplification with classical reinforcement learning has led to the so-called “hybrid agent for quantum-accessible reinforcement learning”, which achieves a quadratic speedup in sample complexity for certain learning problems. So far, this hybrid agent has only been applied to stationary learning problems, that is, learning problems without any time dependency within components of the Markov decision process. In this work, we investigate the applicability of the hybrid agent to dynamic RL environments. To this end, we enhance the hybrid agent by introducing a dissipation mechanism and, with the resulting learning agent, perform an empirical comparison with a classical RL agent in an RL environment with a time-dependent reward function. Our findings suggest that the modified hybrid agent can adapt its behavior to changes in the environment quickly, leading to a higher average success probability compared to its classical counterpart. keywords Quantum reinforcement learning, Hybrid algorithm, Continual reinforcement learning, Amplitude amplification 1 Introduction From the success of convolutional neural networks for computer vision tasks in the early 2010s ( Cireşan et al. 2011 ; Krizhevsky et al. 2017 ) to the nowadays ever-expanding usage of generative artificial intelligence based on diffusion ( Sohl-Dickstein et al. 2015 ) or transformer ( Vaswani et al. 2017 ) models, the vast field of machine learning (ML) has impacted many areas of modern life. Besides the continuous development of model architectures and training methods, the rapid improvement of specialized computing hardware in the form of graphics or tensor processing units (GPUs/TPUs) has led to today’s prevalence of ML. This work focuses on reinforcement learning (RL), the ML subcategory encompassing interaction-based learning. While best known in the area of games with algorithmic breakthroughs such as deep Q-learning (DQN) for Atari games ( Mnih et al. 2015 ) or the MuZero algorithm ( Schrittwieser et al. 2020 ) , RL also serves other areas such as the training of the above mentioned language models through reinforcement learning from human feedback (RLHF) ( Ziegler et al. 2020 ) . The combination of RL with quantum computing, which promises polynomial ( Grover 1997 ; Hamann and Wölk 2022 ) or even exponential ( Shor 1994 ; Dunjko et al. 2018 ; Liu et al. 2021 ; Molteni et al. 2024 ) speedups in specific problem tasks, has opened up the research field of quantum reinforcement learning (QRL) ( Dong et al. 2008 ; Dunjko et al. 2016 ; Jerbi et al. 2021 ) . Due to the restrictions of current quantum hardware in the so-called noisy, intermediate-scale quantum (NISQ) ( Preskill 2018 ) era, however, the search for “quantum advantage” remains challenging ( Schuld and Killoran 2022 ; Cerezo et al. 2024 ; Ciliberto et al. 2020 ) . Here, the so-called “hybrid agent for quantum-accessible reinforcement learning” ( Hamann and Wölk 2022 ) , or briefly hybrid learning agent, provides an interesting outlook for the post-NISQ era: by incorporating amplitude amplification into the process of sampling action sequences, it allows to find rewarded sequences quadratically faster for specific RL environment types. This hybrid learning agent, however, has only been used in stationary environments so far, i.e., environments which do not undergo changes throughout the training process. Yet, many RL problems, especially regarding real-world applications (e.g., traffic control), do possess some kind of time-dependent dynamics (e.g., closed lanes due to road maintenance or temporary differences in traffic volume due to daily commuters). In such scenarios, a well-trained agent often performs worse after a sudden change of the environment compared to an untrained or a barely trained agent. As a consequence, a fast learning quantum agent often has a disadvantage compared to a slower learning classical agent when the environment changes. To compensate for this disadvantage and outperform classical agents again, the hybrid agent needs an efficient procedure to unlearn or relearn. Therefore, in this work, we take a first step by introducing a dissipation mechanism in the hybrid learning agent and by investigating its applicability and efficiency in such non-stationary environments. In this way, the hybrid learning agent becomes more robust to changes of the problem environment without the necessity to introduce complex mechanisms for explicitly detecting anomalies. Specifically, we compare the hybrid learning agent against a classical counterpart in the so-called Gridworld or maze environment, where we add time dependency to the reward function both on the intra- and inter-episode level. Our research questions are the following: • Which modifications have to be applied to the original hybrid learning agent as defined in ( Hamann and Wölk 2022 ) to achieve high-performance learning in these kinds of non-stationary environments? • Assuming a better learning performance on a first reward function, can the hybrid learning agent overcome the resulting worse initial success probability after a change of reward function? • Does the less frequent updating of the agent’s policy due to the episodes spent on amplitude amplification hinder the hybrid learning agent’s capability to adapt in the dynamic environment? This article is structured as follows: in section 2 , we recapitulate RL theory and the initial hybrid learning agent and summarize related work on QRL and continual RL. Section 3 features a description of the basic Gridworld problem as well as the modified environment options with non-stationary reward functions. In section 4 , we present our modifications of the hybrid learning agent for the non-stationary RL problems and discuss the methodology of our numerical simulations. sections 5 and 6 comprise our simulation results and the discussion thereof. Finally in section 7 , we provide a conclusion and an outlook to future research. 2 Background 2.1 Reinforcement Learning Reinforcement Learning (RL) ( Sutton and Barto 2018 ) is a branch of machine learning focused on learning through interaction. Usually, this interaction takes place in a sequence of discrete time steps t\in\mathbb{N}_{0} and is formalized as a Markov Decision Process (MDP) ( Puterman 1994 ) . The MDP is a five-tuple \displaystyle M=\langle\,T,\,\mathcal{S},\,\mathcal{A},\,p_{t}(\cdot|s,a),\,r_{t}(s,a)\,\rangle. (1) T refers to the total amount of interaction steps and can be finite (in episodic tasks) or infinite (in continuing tasks). \mathcal{S} and \mathcal{A} denote the sets of states and actions, respectively. The transition probability function \displaystyle p_{t}:\mathcal{S}\times\mathcal{S}\times\mathcal{A}\rightarrow[0,1] (2) denotes the probability p_{t}(s^{\prime}|s,a) that upon playing action a\in\mathcal{A} in state s\in\mathcal{S} at time t , the state at time t+1 will be s^{\prime}\in\mathcal{S} . Associated with this new state is a real-valued, bounded reward R_{t+1} , with the value determined by the reward function \displaystyle r_{t}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}. (3) After playing action a in state s at time t , the agent receives the reward R_{t+1}:=r_{t}(s,a) . The agent’s behavior is described by its policy \pi:\mathcal{A}\times\mathcal{S}\rightarrow[0,1], which describes the probability \pi(a|s) to choose action a upon perceiving state s . The agent’s goal is to adapt its policy such as to maximize the expected cumulative reward \displaystyle G_{t}=\sum_{k=t+1}^{T}\chi^{k-t-1}R_{k} (4) at any time step t , where \chi\in[0,1] is a discount factor. In continuing tasks, setting \chi<1 ensures that the cumulative reward does not diverge. In episodic tasks, the interaction terminates once the final time step t=T is reached, upon which the interaction starts