Hybrid action Reinforcement Learning for quantum architecture search

Jiayang Niu, Yan Wang, Ke Deng, Azadeh Alavi, Mark Sanderson, Yongli Ren · 2025

Read Hybrid action Reinforcement Learning for quantum architecture search by Jiayang Niu, Yan Wang, Ke Deng, Azadeh Alavi, Mark Sanderson, Yongli Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04967.txt.

Introduction

Jiayang Niu and co-authors present HyRLQAS, a learning method that designs quantum circuits by choosing gates and setting their values at the same time. Past search methods split those two jobs, which made results weak or brittle.

Their agent works in a mixed space of discrete gate picks and smooth values, and can refine gates placed in past steps. Trained to find low ground-state energy for molecules, it beats prior search methods with fewer gates. Its guided start values also cut the steps that later fine-tuning needs, reaching very low error.

Catalog record

CREATORJiayang Niu, Yan Wang, Ke Deng, Azadeh Alavi, Mark Sanderson, Yongli Ren
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04967

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hybrid action Reinforcement Learning for quantum architecture search Jiayang Niu, Yan Wang, Ke Deng, Azadeh Alavi, Mark Sanderson, Yongli Ren Abstract Designing expressive yet trainable quantum circuit architectures remains a major challenge for variational quantum algorithms, where manual or heuristic designs often lead to suboptimal performance. We propose HyRLQAS ( H ybrid- A ction R einforcement L earning for Q uantum A rchitecture S earch), a unified framework that couples discrete gate placement and continuous parameter generation within a hybrid action space. Unlike existing approaches that treat structure and parameter optimization separately, HyRLQAS jointly learns circuit topology and initialization while dynamically refining previously placed gates through a reinforcement learning process. Trained in a variational quantum eigensolver (VQE) environment, the agent constructs circuits that minimize molecular ground-state energy. Experiments show that HyRLQAS achieves consistently lower energy errors and shorter circuits than both discrete-only and continuous-only baselines. Furthermore, the hybrid action space not only leads to better circuit structures but also provides favorable parameter initializations, resulting in post-optimization energy distributions with consistently lower minima. These results suggest that hybrid-action reinforcement learning provides a principled pathway toward automated, hardware-efficient quantum circuit design. 1 Introduction In the Noisy Intermediate-Scale Quantum (NISQ) era ( 1 ), current quantum hardware remains constrained by limited qubit counts and significant noise. Consequently, hybrid quantum–classical algorithms have emerged as the dominant paradigm for achieving near-term quantum advantage. These methods rely on iterative feedback loops between a quantum processor and a classical optimizer, and their effectiveness largely depends on the design of the underlying Parameterized Quantum Circuit (PQC), also known as an ansatz . Representative examples include the Variational Quantum Eigensolver (VQE) ( 27 ), the Quantum Approximate Optimization Algorithm (QAOA) ( 8 ), and various variational quantum machine learning models ( 31 ; 23 ). Among them, the VQE framework provides a prototypical example for illustrating how PQCs are used in practice: it estimates the ground-state energy of a molecular Hamiltonian by optimizing the parameters of a quantum circuit to minimize the expected energy E(\bm{\theta})=\langle\psi(\bm{\theta})|H|\psi(\bm{\theta})\rangle, (1) where H denotes the molecular Hamiltonian and |\psi(\bm{\theta})\rangle is the variational quantum state. Hence, the performance of VQE—and more generally of hybrid quantum–classical algorithms—critically depends on the structure and expressivity of the chosen PQC. While several PQC (ansatz) families have been proposed—such as hardware-efficient, chemically inspired, and problem-specific templates—most are still designed manually or follow rigid heuristic patterns. However, such hand-crafted circuits often struggle to generalize across problem domains or adapt to hardware noise and connectivity constraints, thereby limiting scalability, expressivity, and robustness on realistic NISQ devices ( 22 ; 4 ; 13 ). These challenges underscore the need for automated Quantum Architecture Search (QAS), which seeks to discover PQC structures optimized for specific tasks and hardware characteristics. In this context, VQE is commonly employed as a downstream evaluation task, as its optimized energy E(\bm{\theta}) directly reflects the effectiveness of the discovered quantum architectures. Recent progress in Quantum Architecture Search (QAS) has explored a variety of paradigms to overcome the limitations of manually designed ansatz structures. Among these, evolutionary and genetic algorithms search for effective circuit configurations through iterative mutation and selection processes ( 37 ; 29 ; 2 ). In parallel, sampling-based strategies ( 34 ; 10 ), such as Monte Carlo sampling and supernet weight sharing ( 39 ), have been developed to reduce search complexity and enable more efficient benchmarking. Although both paradigms have shown promising results, evolutionary methods often suffer from poor scalability as the number of parameterized gates increases ( 40 ; 28 ), while sampling-based approaches still struggle to achieve chemically accurate results under realistic noise, limiting their practical applicability on NISQ devices ( 21 ). In contrast to search-based approaches, reinforcement learning (RL) ( 22 ; 36 ) offers a dynamic framework for QAS. In RL-based QAS ( 41 ), an agent sequentially places quantum gates, while an external optimizer (e.g., COBYLA, SPSA ( 32 ), Adam ( 15 )) adjusts parameters, and the final energy serves as the reward for policy updates. Nonetheless, existing frameworks ( 25 ; 26 ; 6 ) exhibit three major issues: (i) they focus solely on learning gate placement, ignoring how gate parameters interact with circuit structure, thus failing to capture the intrinsic dependency between parameter initialization and gate topology; (ii) parameter optimization is entirely delegated to an external optimizer that restarts from scratch in every episode, preventing experience reuse and leading to redundant computation; and (iii) external optimizers themselves suffer from inherent limitations—as circuit depth increases, they often encounter barren plateaus ( 22 ; 18 ), where gradient information vanishes and convergence becomes highly sensitive to initialization (see Fig. 1 ). To overcome these issues, we propose the H ybrid- A ction R einforcement L earning for Q uantum A rchitecture S earch ( HyRLQAS ) framework. HyRLQAS expands the agent’s action space to jointly learn where to place gates and how to initialize their parameters during training. For the first issue, our agent explicitly learns both gate placement and parameter initialization. Before conducting the main experiments, we initially attempted to let the agent directly learn optimal parameter values without relying on any external optimizer; however, this approach did not converge successfully (see Section 4.3 ), indicating the necessity of using learned parameters as effective warm-starts rather than final optima. For the second issue, by enabling the agent to learn parameter initialization distributions, HyRLQAS reuses optimization knowledge that was previously discarded in existing works. Prior frameworks rely solely on external optimizers to fine-tune circuit parameters but never allow the agent to capture the underlying distribution of successful initializations—an inefficiency that HyRLQAS eliminates by incorporating this learning process directly into training. For the third issue, employing the learned initialization parameters significantly increases the likelihood of achieving lower post-optimization energies. In practice, circuits initialized by the learned policy consistently converge to better energy minima compared to those with random initialization, yielding more stable and reliable outcomes. Building upon this foundation, HyRLQAS formulates quantum architecture search as a unified hybrid-action reinforcement learning problem, where discrete gate selection and continuous parameter generation are jointly optimized under a single policy. Unlike prior approaches that treat gate placement and parameter tuning as two independent processes, HyRLQAS embeds parameter generation directly into the RL loop, enabling the agent to iteratively refine initialization strategies as training progresses. The main contributions of this work are summarized as follows: • We propose HyRLQAS , a hybrid-action reinforcement learning framework that unifies discrete gate placement and continuous parameter generation within a single policy, enabling the agent to jointly learn circuit structures and parameter initialization strategies during training. • We introduce a parameter-initialization learning mechanism in which the parameters learned by the agent act as informed warm-starts for external optimizers. This design leverages optimization experience accumulated during training and mitigates the inefficiency of restarting from random initializations in existing QAS frameworks. • We validate the effectiveness of HyRLQAS through extensive experiments on multiple molecular Hamiltonians, showing that its performance gains stem from both the learned initialization parameters and the hybrid learning–induced improvements in circuit structure. (a) LiH 4-qubit (b) LiH 6-qubit (c) LiH 8-qubit Figure 1: Distribution of final energies under three initialization strategies for the same circuit. Near-zero : parameters

Read the complete text