It’s-A-Me, Quantum Mario: Scalable Quantum Reinforcement Learning with Multi-Chip Ensembles

Junghoon Justin Park, Huan-Hsin Tseng, Shinjae Yoo, Samuel Yen-Chi Chen, Jiook Cha · 2025

Read It’s-A-Me, Quantum Mario: Scalable Quantum Reinforcement Learning with Multi-Chip Ensembles by Junghoon Justin Park, Huan-Hsin Tseng, Shinjae Yoo, Samuel Yen-Chi Chen, Jiook Cha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-00713.txt.

Introduction

Junghoon Justin Park leads five authors in total, with Huan-Hsin Tseng, Shinjae Yoo, Samuel Yen-Chi Chen, and Jiook Cha, in applying quantum reinforcement learning to Super Mario Bros. Quantum learners promise compact models with vast state spaces, but current noisy quantum hardware limits qubits and adds noise.

Their multi-chip ensemble splits high-dimensional game observations across several small Quantum Convolutional Neural Networks, each on its own circuit, then classically aggregates outputs inside a Double Deep Q-Network. The modular design reduces information loss from dimensionality reduction and remains implementable on near-term hardware. It achieves better performance and learning stability than classical baselines and single-chip quantum models, opening complex environments to quantum agents.

Catalog record

CREATORJunghoon Justin Park, Huan-Hsin Tseng, Shinjae Yoo, Samuel Yen-Chi Chen, Jiook Cha
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-00713

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

It’s-A-Me, Quantum Mario: Scalable Quantum Reinforcement Learning with Multi-Chip Ensembles Thanks: The views expressed in this article are those of the authors and do not represent the views of Wells Fargo. This article is for informational purposes only. Nothing contained in this article should be construed as investment advice. Wells Fargo makes no express or implied warranties and expressly disclaims all legal, tax, and accounting implications related to this article. Junghoon Justin Park, Huan-Hsin Tseng, Shinjae Yoo, Samuel Yen-Chi Chen, Jiook Cha Abstract Quantum reinforcement learning (QRL) promises compact function approximators with access to vast Hilbert spaces, but its practical progress is slowed by NISQ-era constraints such as limited qubits and noise accumulation. We introduce a multi-chip ensemble framework using multiple small Quantum Convolutional Neural Networks (QCNNs) to overcome these constraints. Our approach partitions complex, high-dimensional observations from the Super Mario Bros environment across independent quantum circuits, then classically aggregates their outputs within a Double Deep Q-Network (DDQN) framework. This modular architecture enables QRL in complex environments previously inaccessible to quantum agents, achieving superior performance and learning stability compared to classical baselines and single-chip quantum models. The multi-chip ensemble demonstrates enhanced scalability by reducing information loss from dimensionality reduction while remaining implementable on near-term quantum hardware, providing a practical pathway for applying QRL to real-world problems. I Introduction Quantum reinforcement learning (QRL) has emerged as a promising frontier integrating quantum computing capabilities with reinforcement learning paradigms [ 1 , 2 , 3 , 4 ] . Recent advances have demonstrated viable QRL implementations through variational quantum circuits (VQC), with Chen et al. (2020) pioneering a quantum analog to deep Q-learning networks by substituting neural networks with VQCs [ 1 ] . Subsequent research has refined these approaches, validating that quantum models can effectively learn control policies through value iteration on standard benchmarks [ 3 , 5 ] . These developments highlight QRL’s potential to leverage quantum parallelism and entanglement for enhanced learning efficiency [ 4 ] . Nevertheless, QRL still faces severe scalability challenges. The limitations of noisy intermediate-scale quantum (NISQ) devices [ 6 ] have confined most QRL demonstrations to simplified problems using shallow circuits with only 4-6 qubits [ 1 , 2 , 3 ] . While it is possible to process high-dimensional data with limited number of qubits using amplitude encoding, it requires exponential circuit depth [ 7 , 8 , 9 ] . As quantum noise grows exponentially with circuit depth [ 10 ] , large-scale quantum circuits are often infeasible. Consequently, current quantum agents operate with restricted state representations and can only tackle low-dimensional environments. Even recent advancements remain limited to elementary tasks such as CartPole [ 5 , 11 ] . The fundamental challenge lies in scaling to high-dimensional spaces, which demands larger quantum circuits that inevitably encounter deeper noise effects and the barren plateau problem—the vanishing of gradients in large quantum circuits [ 12 , 13 ] . This scalability bottleneck prevents QRL from addressing complex environments where quantum advantages might be most significant. Distributed quantum computing techniques offer a promising avenue to overcome these limitations. While extensively explored in quantum computing broadly [ 14 , 15 , 16 ] —including feature-based partitioning approaches that distribute input data across multiple circuits with classical aggregation [ 17 ] —these strategies have remained largely unexplored in QRL [ 18 , 19 , 20 ] . This gap represents a missed opportunity to scale quantum agents to more challenging domains. In this work, we introduce a scalable QRL framework leveraging multi-chip ensembles. The multi-chip ensemble architecture has been shown to offer advantages including greater scalability, barren plateau mitigation, improved generalization, and noise resilience [ 21 ] . We apply this modular approach by partitioning the agent’s quantum policy model across multiple quantum processing units and classically aggregating their outputs for decision-making. Specifically, we divide observation data into subsets, process them through parallel smaller quantum circuits on separate quantum processors, then combine their expectation values to produce final Q-value estimates or policy outputs. Drawing inspiration from classical ensemble learning and recent distributed VQC method, our approach effectively constructs a larger virtual quantum model from NISQ-compatible components. We demonstrate our approach’s efficacy by training a quantum agent to play Super Mario Bros [ 22 ] —a complex, high-dimensional environment previously beyond the reach of traditional QRL methods. II Background II-A Reinforcement Learning RL is a framework where an agent learns to maximize cumulative rewards through interactions with an environment. This process is formalized as a Markov Decision Process (MDP) defined by the tuple \langle\mathcal{S,A,R,P,}\gamma\rangle where \mathcal{S} represents the state space, \mathcal{A} the action space, \mathcal{R} the reward function, \mathcal{P} the transition probability function, and \gamma\in[0,1) the discount factor. The agent’s objective is to find an optimal policy \pi^{*} that maximizes expected future rewards. This is quantified by the action-value function Q^{\pi}(s,a) , which represents the expected return when taking action a in state s and following policy \pi thereafter: \displaystyle Q^{\pi}(s,a)=\mathbb{E}\Bigg[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\Bigg], (1) \displaystyle s_{t}\sim P(s_{t-1},a_{t-1}),\quad a_{t}\sim\pi(s_{t}) This function satisfies the Bellman equation [ 23 ] : Q^{\pi}(s,a)=\mathbb{E}\Big[\mathcal{R}(s,a)\Big]+\gamma\,\mathbb{E}_{\mathcal{P},\pi}\Big[Q^{\pi}(s^{\prime},a^{\prime})\Big] (2) Q-learning, a fundamental RL algorithm, iteratively updates the action-value function using temporal difference learning: Q(s_{t},a_{t})\leftarrow Q(s_{t},a_{t})+\alpha\left(r_{t+1}+\gamma\max_{a}Q(s_{t+1},a)\right.\\ \Big.-Q(s_{t},a_{t})\Big) (3) where \alpha is the learning rate and r_{t+1} the immediate reward. Deep Q-Networks (DQN) [ 24 ] extend Q-learning by approximating the action-value function with neural networks. Double DQN (DDQN) [ 25 , 26 ] improves upon standard DQN by addressing its overestimation bias through decoupling action selection and evaluation. DDQN employs two networks: an online network with parameters \theta for action selection and a target network with parameters \bar{\theta} for value estimation. The loss function is: L(\theta)=\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{D}}\Big[\big(r+\gamma Q_{\bar{\theta}}(s^{\prime},\arg\max_{a^{\prime}}Q_{\theta}(s^{\prime},a^{\prime}))\\ -Q_{\theta}(s,a)\big)^{2}\Big] (4) where \mathcal{D} is a replay buffer of transitions (s,a,r,s^{\prime}) . By separating action selection (via \arg\max_{a^{\prime}}Q_{\theta} ) from value estimation (via Q_{\bar{\theta}} ), DDQN significantly reduces the systematic overestimation present in standard Q-learning with function approximation. II-B VQCs in RL VQCs offer a quantum approach to function approximation in RL, analogous to neural networks in classical RL [ 27 ] . A VQC consists of three key components: an encoding layer that transforms classical states into quantum states through rotational gates, a parameterized variational ansatz that processes quantum information, and a measurement layer that extracts expectation values to represent action-values. In QRL, VQCs replace neural networks in DQN frameworks [ 1 , 2 , 5 , 3 , 4 ] . The process operates as follows: first, the environment state s is encoded into a quantum state through rotation gates applied to initialized qubits. Next, the encoded state passes through a parameterized quantum circuit (the ansatz) whose parameters \boldsymbol{\theta} are trainable. Finally, Pauli operators measure each qubit (or combinations thereof) to produce expectation values that correspond to Q-values for each possible action: Q_{\theta}(s,a)=\text{Tr}[H_{a}\rho(s,\theta)] (5) where \rho(s,\theta)=|\psi(s,\theta)\rangle\langle\psi(s,\theta)| is the density matrix representing the quantum state after encoding and applying the variational circuit, and H_{a} is the Hermitian operator corresponding to action a . This formulation generalizes to both pure and mixed quantum states, making it particularly relevant for NISQ devices. The circuit parameters \boldsymbol{\theta} are updated by minimizing a loss function equivalent to that used in classical DQN or DDQN, with gradients computed through parameter-shift rules [ 28 ] or other quantum-compatible

Read the complete text