Quantum Reinforcement Learning with Dynamic-Circuit Qubit Reuse and Grover-Based Trajectory Optimization

Thet Htar Su, Shaswot Shresthamali, Masaaki Kondo · 2025

Read Quantum Reinforcement Learning with Dynamic-Circuit Qubit Reuse and Grover-Based Trajectory Optimization by Thet Htar Su, Shaswot Shresthamali, Masaaki Kondo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16002.txt.

Introduction

Quantum learning agents need many qubits as tasks grow longer, and today's quantum chips have few to spare. Thet Htar Su, Shaswot Shresthamali, and Masaaki Kondo offer a design that reuses the same qubits step after step. Their system measures and resets qubits mid-task, so a fixed set of seven serves for any task length.

Good paths through a task are boosted with a quantum search trick so they get sampled more often. Computer tests kept the same paths as a fixed design while cutting qubit needs by two thirds. A run on an IBM quantum chip showed the design works on real, noisy hardware.

Catalog record

CREATORThet Htar Su, Shaswot Shresthamali, Masaaki Kondo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-16002

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantum Reinforcement Learning with Dynamic-Circuit Qubit Reuse and Grover-Based Trajectory Optimization Thet Htar Su, Shaswot Shresthamali, Masaaki Kondo Abstract A fully quantum reinforcement learning framework is developed that integrates a quantum Markov decision process, dynamic circuit-based qubit reuse, and Grover’s algorithm for trajectory optimization. The framework encodes states, actions, rewards, and transitions entirely within the quantum domain, enabling parallel exploration of state–action sequences through superposition and eliminating classical subroutines. Dynamic circuit operations, including mid-circuit measurement and reset, allow reuse of the same physical qubits across multiple agent–environment interactions, reducing qubit requirements from 7 × T \times T to 7 for T T time steps while preserving logical continuity. Quantum arithmetic computes trajectory returns, and Grover’s search is applied to the superposition of these evaluated trajectories to amplify the probability of measuring those with the highest return, thereby accelerating the identification of the optimal policy. Simulations demonstrate that the dynamic-circuit-based implementation preserves trajectory fidelity while reducing qubit usage by 66 percent relative to the static design. Experimental deployment on IBM Heron-class quantum hardware confirms that the framework operates within the constraints of current quantum processors and validates the feasibility of fully quantum multi-step reinforcement learning under noisy intermediate-scale quantum conditions. This framework advances the scalability and practical application of quantum reinforcement learning for large-scale sequential decision-making tasks. Keywords: quantum reinforcement learning, dynamic circuits, quantum Markov decision process, qubit reuse, Grover’s algorithm, policy optimization, NISQ devices I Introduction Classical reinforcement learning (RL) has achieved remarkable success in sequential decision-making tasks; however, it often suffers from slow convergence and substantial computational demands. Inefficient exploration and high sample complexity remain persistent challenges, particularly in large or complex environments where extensive interactions are required to identify optimal policies Dulac-Arnold et al. 2021 . To address these limitations, quantum reinforcement learning (QRL) has emerged as a promising paradigm that leverages quantum computing to overcome the computational bottlenecks of classical RL Jerbi et al. 2021a ; Chen . By exploiting superposition, entanglement, and amplitude amplification, QRL offers the potential to accelerate exploration, reduce sample complexity, and enhance policy optimization beyond classical capabilities. Early research has explored hybrid quantum–classical approaches, where quantum circuits are integrated into selected components of the RL pipeline or quantum-inspired policy optimization algorithms are developed Meyer et al. . While such hybrid methods provide valuable insights, they also introduce drawbacks. One critical issue is the reliance on classical–quantum data conversion, which creates input–output bottlenecks that slow computation, introduce noise, and reduce the benefits of quantum acceleration. These issues are particularly severe in the noisy intermediate-scale quantum (NISQ) regime, where devices are constrained by short coherence times, limited qubit connectivity, and error-prone operations Preskill 2018 . To fully realize the advantages of quantum acceleration, researchers have proposed fully quantum RL approaches in which the agent, environment, and learning processes are implemented natively in the quantum domain Wiedemann et al. 2023 ; Su et al. 2025 . While the previous work Wiedemann et al. 2023 established a proof of concept, its applicability to real-world problems remains limited. The study in Su et al. 2025 , in particular, revealed scalability challenges, as it requires a large number of qubits to model multiple interaction steps, causing resource demands to grow rapidly with the complexity of the environment dynamics. For instance, solving an environment with four states and two actions required seven qubits for a single interaction, and extending this to T interactions would demand 7 \times T qubits. Such qubit overhead is prohibitive on small quantum processors, where effective qubit counts are already constrained by hardware imperfections Pawar et al. 2025 . Consequently, while prior work demonstrates feasibility, it remains impractical for handling larger or more complex environments on near-term hardware. These challenges highlight a fundamental scalability barrier: richer interaction modeling in fully quantum RL has so far been achievable only at the cost of dramatically increased qubit requirements, exceeding the capacity of both NISQ devices and classical simulators. To overcome this barrier, QRL framework must be redesigned to preserve the benefits of quantum parallelism while drastically reducing physical qubit usage. A promising direction is the use of dynamic circuits, which enables mid-circuit measurement and qubit reuse during execution, allowing the same physical qubits to perform multiple roles within a computation. This capability directly addresses the qubit-scaling problem by recycling qubits for different purposes over the course of a computation, enabling more complex RL dynamics to be executed with fewer qubits. Dynamic circuits are already supported in superconducting Nation and trapped-ion Pino et al. 2021 architectures, with recent demonstrations on IBM processors Johnson confirming their practical feasibility. Building on these advances, this work proposes a scalable fully quantum RL framework based on dynamic circuit to recycle qubits across interaction steps through mid-circuit measurement and reset. This design substantially reduces qubit requirements, improves scalability, and enables the realization of a fully quantum RL system on near-term hardware. Within this framework, all computations, from state encoding to trajectory search, are performed entirely within the quantum domain, ensuring efficient resource utilization. The primary contribution lies in demonstrating that multi-step agent–environment interactions can be implemented in a quantum circuit without increasing the qubit count, thereby enabling large-scale Markov decision process (MDP) to be solved fully within the quantum domain while remaining compatible with the constraints of near-term quantum devices. The key contributions of this work are as follows: (1) Quantum agent–environment representation: A fully quantum encoding of states, actions, rewards, and state transitions using superposition, enabling parallel exploration of state–action pairs. (2) Qubit-efficient multiple interactions via dynamic circuits: A quantum MDP (QMDP) design that reuses qubits through mid-circuit measurement and reset, reducing qubit requirements and aligning with the constraints of quantum hardware. (3) Quantum trajectory search with Grover’s algorithm: An efficient method to identify the maximum-reward trajectory, enabling faster exploration of trajectories and accelerating policy optimization. This paper is organized as follows: Sec. II reviews prior work on QRL and dynamic circuits; Sec. III introduces relevant concepts from classical RL and quantum computing; Sec. IV describes the proposed QRL implementation, including the quantum MDP design, dynamic-circuit-based quantum interactions, and trajectory search; Sec. V presents demonstrations and results; Sec. VI offers discussion; and Sec. VII concludes with a summary and future directions. II Related Work Quantum reinforcement learning (QRL) has developed along two main approaches: hybrid quantum–classical methods, which combine classical RL algorithms with quantum principles, and fully quantum methods, which aim to implement all RL components entirely within the quantum domain. The following review summarizes key developments in both areas, leading to the specific gap our work addresses. II.1 Hybrid Quantum-Classical Reinforcement Learning Hybrid quantum–classical methods include quantum-inspired RL (QiRL), variational quantum circuit (VQC)-based RL, and RL methods leveraging quantum subroutines. QiRL represented the initial integration of quantum computing (QC) and RL, aiming to overcome classical limitations such as slow learning, the exploration–exploitation trade-off, and high computational costs Meyer et al. . By leveraging quantum principles such as superposition and amplitude amplification, researchers advanced the field from representing actions and policies within the quantum domain Dong et al. 2008 ; Dao-Yi et al. 2006 ; Chen and Dong 2008 ; CHEN et al. 2006 ; Dong et al. 2012 ; Ganger and Hu 2019 ; Cho et al. 2024 to embedding experiences

Read the complete text