Quantum Hierarchical Reinforcement Learning via Variational Quantum Circuits Thanks: The views expressed in this article are those of the authors and do not represent the views of Wells Fargo. This article is for informational purposes only. Nothing contained in this article should be construed as investment advice. Wells Fargo makes no express or implied warranties and expressly disclaims all legal, tax, and accounting implications related to this article. Yu-Ting Lee 1 , Samuel Yen-Chi Chen 2 , Fu-Chieh Chang 1,3 Abstract Reinforcement learning is one of the most challenging learning paradigms where efficacy and efficiency gains are extremely valuable. Hierarchical reinforcement learning is a variant that leverages temporal abstraction to structure decision-making. While parametrized quantum computations have shown success in non-hierarchical reinforcement learning, whether these advantages adapt to hierarchical decision-making remains a critical open question. In this work, we develop a hybrid hierarchical agent based on the option-critic architecture. This hybrid agent substitutes classical components with variational quantum circuits for feature extractors, option-value functions, termination functions, and intra-option policies. Evaluated on standard benchmarking environments, results show that a hybrid agent utilizing a quantum feature extractor can outperform classical baselines while saving up to 66% trainable parameters. We also identify an architectural bottleneck that quantum option-value estimation severely degrades performance. Further ablation studies reveal how architectural choices of the quantum circuits affect performance. Our work establishes design principles for parameter-efficient hybrid hierarchical agents. r14942088@ntu.edu.tw, yen-chi.chen@wellsfargo.com, d09942015@ntu.edu.tw I Introduction Reinforcement learning (RL) has achieved remarkable success in solving complex decision-making problems [ 39 , 34 , 31 ] . However, standard RL algorithms often struggle with long-horizon tasks and sparse rewards [ 1 , 29 ] . Hierarchical reinforcement learning (HRL) mitigates these issues through temporal abstraction, representing knowledge as courses of action spanning multiple time scales [ 29 , 28 , 9 , 41 , 2 ] . A prominent example is the option-critic architecture [ 2 ] , which enables an end-to-end, gradient-based learning of temporal abstraction called options . Fig. 1 : Training a hybrid RL agent with quantum circuits. We consider a hybrid framework in which the RL agent is enhanced by quantum circuits trained with a deep RL algorithm. In this work, we consider a hybrid quantum-classical option-critic architecture. Current HRL research remains predominantly classical. In standard non-hierarchical settings, quantum reinforcement learning (QRL) has shown promise in accelerating learning and achieving performance comparable to or exceeding classical approaches with significantly fewer parameters [ 23 , 21 , 15 , 18 , 7 , 19 ] . Whether HRL can be similarly enhanced by leveraging the expressivity inherent in quantum computing [ 15 , 35 ] remains a critical and largely unexplored open question. In this work, we introduce a hybrid quantum-classical option-critic framework by integrating variational quantum circuits (VQCs) into an end-to-end hierarchical pipeline. Figure 1 illustrates the general paradigm of a hybrid RL agent, where quantum circuits are integrated into the learning process. Within our hierarchical architecture, we selectively substitute classical components—feature extractors, option-value functions, termination functions, and intra-option policies—with VQCs. Our modular approach enables us to evaluate the distinct impacts of quantum components on the learning dynamics of hierarchical agents: where quantum circuits benefit learning and where they impose a bottleneck. We release our code for reproducibility. 1 1 1 Code is available at https://github.com/Yu-TingLee/quantum-option-critic . The main contributions of this work are: (1) We demonstrate that a hybrid hierarchical model with a quantum feature extractor outperforms classical baselines while saving up to 66% trainable parameters, and is only surpassed by classical models with greater capacity. (2) We identify a critical performance bottleneck that using VQCs for option-value function approximation fails to provide useful learning signals and causes learning failure. (3) Ablation studies show how architectural choices, specifically circuit depth, learnable input scaling, and entanglement, affect the hybrid agent’s efficacy. The remainder of this paper is organized as follows. Section II reviews related work in RL, HRL, and QRL. Section III provides necessary background on Markov decision processes (MDPs), the option-critic architecture, and VQCs. We detail the training algorithm in Section IV and our proposed VQC architecture in Section V . Experimental settings and results are presented in Sections VI and VII , respectively. Finally, Section VIII offers a broader discussion, followed by concluding remarks in Section IX . II Related Work II-A Reinforcement Learning Classical RL encompasses a broad range of tabular and linear methods, with foundational temporal-difference (TD) algorithms such as SARSA [ 39 ] and Q-learning [ 46 ] establishing the core principles of value estimation. However, scaling these traditional techniques to high-dimensional environments often requires deep neural networks to approximate value functions and policies. A prominent example is the Deep Q-Network (DQN) [ 25 ] , which integrated Q-learning with deep function approximation to handle complex state spaces. To mitigate the sample inefficiency and overestimation biases inherent in Q-learning, subsequent value-based models introduced structural and algorithmic modifications, yielding architectures such as Double DQN, Dueling DQN, and the Rainbow agent [ 13 , 45 , 14 ] . Parallel to value-based methods, direct policy optimization frameworks, stemming from the REINFORCE algorithm [ 47 ] , explicitly optimize the policy an RL agent uses, rather than focusing on learning value functions. However, pure policy gradient methods frequently suffer from high variance during training. To resolve this, actor-critic methods were developed as hybrid solutions, where a learned value baseline (the critic) is used to stabilize the policy gradient updates (the actor). Within this hybrid paradigm, algorithms like DDPG [ 20 ] specifically target continuous domains, while frameworks such as A3C/A2C, SAC, and TD3 [ 24 , 12 , 11 ] provide versatile formulations for both discrete and continuous environments. More recently, RL has expanded into complex sequence planning, with frameworks like the Self-Taught Reasoner applying iterative RL refinement to optimize the reasoning capabilities of large language models [ 48 , 5 ] . II-B Hierarchical Reinforcement Learning Hierarchical reinforcement learning has a rich history [ 29 , 41 , 38 , 28 , 9 , 43 ] . Early methods typically rely on manually defined task graphs or finite state machines [ 28 , 9 ] . The options framework [ 41 ] emerged as a prominent two-level solution which enables agents to execute temporally extended actions, called options . Although early options were labor-intensive to design via explicitly provided sub-goals and pseudo-rewards, this mathematical formalization ultimately laid the groundwork for automated option discovery and option learning . Specifically, the option-critic architecture [ 2 ] extended the options framework, allowing an end-to-end, gradient-based learning and discovery of options. This mechanism was subsequently expanded into parallel augmented MDPs in the double actor-critic architecture [ 49 ] . Alternative HRL paradigms include manager-worker architectures, such as FeUdal Networks [ 44 ] , and goal-conditioned frameworks like HIRO [ 26 ] . Empirical analyses suggest that the primary benefit of hierarchy lies in improved exploration toward meaningful intermediate states [ 27 ] . II-C Quantum Reinforcement Learning Quantum reinforcement learning seeks to leverage quantum computing to enhance the performance of RL agents and to accelerate training [ 7 , 36 , 10 ] . In particular, hybrid QRL architectures frequently employ VQCs to encode and process classical state observations to utilize the inherent expressivity of quantum states [ 30 , 15 ] . This expressivity can be tied to specific data encoding strategies; for instance, architectures utilizing data re-uploading have been shown to map classical inputs to partial Fourier series [ 32 ] . Crucially, VQCs can serve as parameter-efficient alternatives to classical deep neural networks with hardware-efficient ansatz that minimizes circuit depth [ 16 ] . In value-based methods, quantum models can match classical performance with significantly fewer trainable parameters [ 21 , 7 ] . Similarly, in hybrid actor-critic architectures, VQCs have demonstrated substantial performance increase and superior sample efficiency, requiring fewer environmental interactions to learn effective policies [ 18 , 8 , 4 ] . Recent research has also expanded QRL into complex new domains, such as in relation extraction [ 50 ] . Despite these advantages, practical scalability is currently limited by hardware noise and the barren plateau phenomenon [ 22 , 37 , 33 ] . As existing QRL architectures are predominantly non-hierarchical, a shift toward hierarchical agents is necessary to evaluate how quantum benefits manifest in structured decision-making. Input obs. shape: (B,in) Classical NN / VQC (B,in) \to (B,in) Feature Extractor Classical NN / VQC (B,in) \to (B, |\Omega| ) Sigmoid (B, |\Omega| ) \to (B, |\Omega| ) Output logits (B, |\Omega| ) Termination Function Classical NN / VQC (B,in) \to (B, |\Omega| ) Output logits (B, |\Omega| ) Option-Value Function Classical NN / VQC (B,in) \to (B,out) Softmax (B,out) \to (B,out) Action probs. (B,out) Intra-Option Policies Fig. 2 : Model architecture for the hybrid option-critic agent. The input observations are processed by a shared feature extractor and then passed to each downstream component. The Option-Value and Termination Functions are implemented as single networks that output logits for all |\Omega| options simultaneously. Conversely, the Intra-Option Policies are |\Omega| independent networks. In our experiments, we selectively substitute with VQCs the four components in light blue: (i) Feature Extractor, (ii) Option-Value Function, (iii) Termination Function, and (iv) Intra-Option Policies. III Preliminaries III-A Markov Decision Process A Markov decision process (MDP) consists of a state space \mathcal{S} , an action space \A , a transition probability P\colon\mathcal{S}\times\A\to\Delta(\mathcal{S}) , a reward function r\colon\mathcal{S}\times\A\to\R , and a discount factor \gamma\in[0,1) . Here, \Delta(\mathcal{S}) is the set of distributions (probability measures) over \mathcal{S} . A stationary, Markovian policy \pi\colon\mathcal{S}\to\Delta(\A) specifies a distribution over actions conditioned on the current state. In the discounted setting, the value function of a policy \pi is defined as V_{\pi}(s)=\E_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\mid s_{0}=s\right] (1) and the action-value function (also known as the Q-function) is Q_{\pi}(s,a)=\E_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\mid s_{0}=s,a_{0}=a\right]. (2) To address the problem of finding a good policy, policy gradient methods optimize a parameterized policy by maximizing an objective function using gradient-based updates. Let \pi(\cdot\mid\theta) be a policy parameterized by \theta and J(\theta) be the expected discounted return. In the discounted setting, the policy gradient theorem [ 40 ] states that \frac{\partial J}{\partial\theta}=\E_{\pi(\cdot\mid\theta)}\left[\nabla_{\theta}\log\pi(a\mid s,\theta)Q_{\pi_{\theta}}(s,a)\right]. (3) Consequently, this result allows the policy to be optimized directly by estimating the gradient from sampled trajectories. Modern algorithms typically introduce a separate function approximator, also known as a critic, to estimate value functions (such as the state-value or the action-value function). This technique establishes the core paradigm of actor-critic methods. III-B The Option-Critic Architecture The options framework [ 41 ] extends MDPs with temporally extended actions. A Markovian option \omega\in\Omega is a triple (\I_{\omega},\pi_{\omega},\beta_{\omega}) in which \I_{\omega}\subset\mathcal{S} is an initiation set, \pi_{\omega} is an intra-option policy, and \beta_{\omega}\colon\mathcal{S}\to[0,1] is a termination function. Under a call-and-return workflow, an agent selects an option \omega , executes actions according to \pi_{\omega} until termination is triggered by \beta_{\omega} , and then repeats the process. Within this setting, the option-critic architecture [ 2 ] enables an end-to-end learning and discovery of options using policy gradient results. Let Q_{\Omega}(s,\omega) be the option-value function, representing the expected return of executing option \omega in state s . The expected value of a state s evaluated over the option selection policy is denoted as V_{\Omega}(s) . Denote \mu_{\Omega}(s,\omega\mid s_{0},\omega_{0})=\sum_{t=0}^{\infty}\gamma^{t}\PR(s_{t}=s,\omega_{t}=\omega\mid s_{0},\omega_{0}) the discounted occupancy measure over state-option pairs. The gradient of the expected discounted return J with respect to the parameters \xi of the intra-option policy \pi_{\omega}(\cdot\mid\xi) is: \frac{\partial J}{\partial\xi}=\sum_{s,\omega}\mu_{\Omega}(s,\omega\mid s_{0},\omega_{0})\sum_{a}\nabla_{\xi}\pi_{\omega}(a\mid s,\xi)Q_{U}(s,\omega,a), (4) where Q_{U} is the value of executing action a in state s under option \omega : \begin{split}&Q_{U}(s,\omega,a)=r(s,a)\,+\\ &\gamma\sum_{s^{\prime}}\PR(s^{\prime}\mid s,a)\left[(1-\beta_{\omega}(s^{\prime}))Q_{\Omega}(s^{\prime},\omega)+\beta_{\omega}(s^{\prime})V_{\Omega}(s^{\prime})\right].\end{split} (5) Similarly, the gradient with respect to the parameters \vartheta of the termination function \beta_{\omega}(\cdot\mid\vartheta) is: \frac{\partial J}{\partial\vartheta}=-\sum_{s,\omega}\mu_{\Omega}(s,\omega\mid s_{0},\omega_{0})\nabla_{\vartheta}\beta_{\omega}(s\mid\vartheta)\left(Q_{\Omega}(s,\omega)-V_{\Omega}(s)\right). (6) In practice, we can estimate these gradients directly from sampled transitions. Specifically, in this work, we approximate the action-value function Q_{U}(s,\omega,a) in Eq. ( 4 ) using a 1-step TD error. The policy gradient in Eq. ( 6 ) is computed using an option-value critic Q_{\Omega} with an off-policy \max_{\omega^{\prime}}Q_{\Omega}(s,\omega^{\prime}) baseline. Details are deferred to Sec. IV and Algorithm 1 . III-C Variational Quantum Circuits Fig. 3 : General structure of a variational quantum circuit (VQC). A classical state s is encoded into a quantum state via the encoding unitary U(s) , evolved by the variational circuit V(\theta) , and then measured by evaluating expectation values of Hermitian observables. Variational quantum circuits (VQCs), also referred to as parameterized quantum circuits (PQCs), are a class of trainable quantum models capable of processing classical input data. In our framework, a VQC maps an input state s\in\mathcal{S} to RL outputs. A standard VQC is typically decomposed into three components. First, a data encoding unitary circuit U(s) encodes a classical state s into an n -qubit system, yielding the encoded state U(s)|0\rangle^{\otimes n} , where |0\rangle^{\otimes n} stands for the ground state of the quantum system. Then, a parameterized unitary circuit V(\theta) evolves this state and transforms the encoded states into V(\theta)U(s)|0\rangle^{\otimes n} . This variational circuit V(\theta) typically consists of alternating layers of trainable single-qubit rotations and multi-qubit entangling gates. Finally, a measurement layer is applied to extract classical information by evaluating the expectation values of predefined Hermitian observables \{O_{k}\}_{k=1}^{K} , such as Pauli-Z matrices. The computation of a VQC can be summarized as a quantum function f(s;\theta)=(\langle O_{1}\rangle,\dots,\langle O_{K}\rangle) where \langle O_{k}\rangle=\langle 0|U^{\dagger}(s)V^{\dagger}(\theta)O_{k}V(\theta)U(s)|0\rangle. (7) The expectation values \langle O_{k}\rangle can be estimated by conducting multiple measurements (shots) on physical quantum devices or by direct calculation when using quantum simulation tools. Specifically, in this work, we implement our circuits in PennyLane [ 3 ] , which treats the quantum circuit as a differentiable computational graph, enabling efficient computation of VQC gradients via classical methods such as backpropagation. IV Option-Critic Algorithm Algorithm 1 Option-Critic (DQN-style) [ 2 ] 1: Number of options |\Omega| , replay buffer \mathcal{D} , batch size |B| , update frequencies N_{\mathrm{critic}} , N_{\mathrm{target}} , discount factor \gamma 2: Randomly initialize model parameters \theta 3: Initialize target network parameters \theta^{\prime}\leftarrow\theta 4: Initialize s\leftarrow s_{0} 5: Choose \omega according to an \epsilon -greedy option \pi_{\Omega}(\cdot\mid s,\theta) over Q_{\Omega}(s,\cdot) 6: for each step t do 7: Choose a\sim\pi_{\omega}(\cdot\mid s,\theta) and execute a 8: Observe s^{\prime},r 9: Store (s,\omega,r,s^{\prime}) in \mathcal{D} \triangleright Store transition to buffer 10: y_{\mathrm{actor}}\leftarrow r+\gamma\left[(1-\beta_{\omega}(s^{\prime}\mid\theta))Q_{\Omega}(s^{\prime},\omega\mid\theta^{\prime})+\beta_{\omega}(s^{\prime}\mid\theta)\max_{\omega^{\prime}}Q_{\Omega}(s^{\prime},\omega^{\prime}\mid\theta^{\prime})\right] \triangleright Compute TD target 11: \mathcal{L}_{\mathrm{policy}}\leftarrow\left[-\log\pi_{\omega}(a\mid s,\theta)(y_{\mathrm{actor}}-Q_{\Omega}(s,\omega\mid\theta))\right] \triangleright Compute policy loss 12: \mathcal{L}_{\mathrm{term}}\leftarrow\left[\beta_{\omega}(s\mid\theta)(Q_{\Omega}(s,\omega\mid\theta)-\max_{\omega^{\prime}}Q_{\Omega}(s,\omega^{\prime}\mid\theta))\right] \triangleright Compute termination loss 13: \mathcal{L}_{\mathrm{total}}\leftarrow\mathcal{L}_{\mathrm{policy}}+\mathcal{L}_{\mathrm{term}} \triangleright Compute actor loss 14: if t mod N_{\mathrm{critic}}=0 then 15: Randomly sample a batch B=\{(s_{j},\omega_{j},r_{j},s^{\prime}_{j})\} from \mathcal{D} 16: y_{j}\leftarrow r_{j}+\gamma\left[(1-\beta_{\omega_{j}}(s^{\prime}_{j}\mid\theta))Q_{\Omega}(s^{\prime}_{j},\omega_{j}\mid\theta^{\prime})+\beta_{\omega_{j}}(s^{\prime}_{j}\mid\theta)\max_{\omega^{\prime}}Q_{\Omega}(s^{\prime}_{j},\omega^{\prime}\mid\theta^{\prime})\right] \triangleright Compute batch TD targets 17: \mathcal{L}_{\mathrm{total}}\leftarrow\mathcal{L}_{\mathrm{total}}+\frac{1}{2|B|}\sum_{j}(Q_{\Omega}(s_{j},\omega_{j}\mid\theta)-y_{j})^{2} \triangleright Compute critic loss 18: end if 19: Update \theta by minimizing \mathcal{L}_{\mathrm{total}} using Adam 20: if \beta_{\omega}(s^{\prime}\mid\theta) terminates in s^{\prime} then 21: Choose \omega according to an \epsilon -greedy option \pi_{\Omega}(\cdot\mid s^{\prime},\theta) over Q_{\Omega}(s^{\prime},\cdot) \triangleright Choose a new option 22: end if 23: if t mod N_{\mathrm{target}}=0 then 24: \theta^{\prime}\leftarrow\theta \triangleright Synchronize target network 25: end if 26: s\leftarrow s^{\prime} 27: end for The option-critic algorithm we use is detailed in Algorithm 1 . For clarity, we have omitted some details on regularization terms, which are specified in Sec. VI-C . The algorithm proceeds as follows. The agent maintains two models, a main network parameterized by \theta and a target network parameterized by \theta^{\prime} . Each model maintains an option-value function Q_{\Omega} and a set of |\Omega| options, where each option \omega consists of an intra-option policy \pi_{\omega} and a termination function \beta_{\omega} . At the start, and whenever the active option terminates, the agent selects a new option \omega using an \epsilon -greedy policy over the option-values Q_{\Omega}(s,\cdot\mid\theta) of the current state. Because Q_{\Omega} represents the expected return of each option, this simple policy exploits the most promising skill based on current estimates while remaining exploratory. Then, at each step, the agent executes an action from the active intra-option policy and stores the resulting transition (s,\omega,r,s^{\prime}) in a replay buffer. This design makes the algorithm more data-efficient through reuse of experience. The core learning mechanism relies on a unified loss by integrating online actor updates with periodic off-policy critic updates. At every step, the algorithm computes a one-step off-policy TD target y_{\mathrm{actor}} using the slow target network parameterized by \theta^{\prime} . The TD target y_{\mathrm{actor}} effectively approximates the action-value Q_{U}(s,\omega,a\mid\theta) (cf. Eq. ( 4 ) and Eq. ( 5 )) with the stable target network. The actor’s learning is further driven by two components: (i) a policy loss that uses the TD loss (y_{\mathrm{actor}}-Q_{\Omega}(s,\omega\mid\theta) as an action-level advantage to update the intra-option policy, and (ii) a termination loss that encourages option switching if the active option’s value falls below the best available alternative, \max_{\omega^{\prime}}Q_{\Omega}(s,\omega^{\prime}\mid\theta) . To ensure stable value approximation, the critic is updated periodically by sampling a mini-batch from the replay buffer to compute a MSE loss with respect to the main network’s Q_{\Omega} against batch TD targets. This MSE is directly added to \mathcal{L}_{\mathrm{total}} , allowing the optimizer to update the parameters \theta end-to-end. Finally, the agent terminates the active option based on \beta_{\omega}(s^{\prime}\mid\theta) , and the target network is synchronized periodically. V Variational Quantum Circuits for Hybrid Option-Critic \left|0\right\rangle_{0} R_{x}(\lambda_{0}s_{0}) R_{x}(\lambda_{0}s_{0}) R_{y}(\theta_{0,0}) R_{y}(\theta_{0,0}) R_{z}(\theta_{1,0}) R_{z}(\theta_{1,0}) A \left|0\right\rangle_{1} R_{x}(\lambda_{1}s_{1}) R_{x}(\lambda_{1}s_{1}) R_{y}(\theta_{0,1}) R_{y}(\theta_{0,1}) R_{z}(\theta_{1,1}) R_{z}(\theta_{1,1}) A \left|0\right\rangle_{2} R_{x}(\lambda_{2}s_{2}) R_{x}(\lambda_{2}s_{2}) R_{y}(\theta_{0,2}) R_{y}(\theta_{0,2}) R_{z}(\theta_{1,2}) R_{z}(\theta_{1,2}) A \left|0\right\rangle_{3} R_{x}(\lambda_{3}s_{3}) R_{x}(\lambda_{3}s_{3}) R_{y}(\theta_{0,3}) R_{y}(\theta_{0,3}) R_{z}(\theta_{1,3}) R_{z}(\theta_{1,3}) A Fig. 4 : VQC structure with 4 qubits and 1 layer. The unitaries in the dashed block constitute a single layer. This block is repeated n times based on the environment and the VQC’s position in the model architecture. Our hybrid option-critic agent relies on four primary components: a shared feature extractor, an option-value function, a termination function, and a set of intra-option policies. For parameter efficiency, the option-value function and termination functions each output values for all options simultaneously rather than maintaining independent networks per option. See Fig. 2 for an illustration of our model architecture. Specifically, any of these components can be instantiated as a VQC to utilize the potential of hybrid systems. We adopt a specific VQC architecture with a data re-uploading structure [ 15 , 32 ] . A VQC layer consists of R_{x} encoding gates, with inputs scaled by trainable scaling parameters \lambda , followed by CNOT gates for entanglement, and parameterized R_{y} and R_{z} rotation blocks. For measurement, we evaluate the expectation values of the Pauli-Z observables with respect to each qubit. See Fig. 4 for a visual illustration of our VQC structure. Because quantum rotation gates ( R_{x} , R_{y} , R_{z} ) are periodic, we normalize all inputs to the range [-\pi,\pi] to serve as valid rotation angles. We apply two mapping strategies depending on a VQC’s position within the network. When a VQC serves as the feature extractor, it directly receives the environmental observations. In the tested environments, there are only two kinds of input variables from the observation space: (i) unbounded continuous variables and (ii) symmetrically bounded continuous variables. For unbounded continuous variables o_{\mathrm{unbound}} , we apply the transformation 2\arctan(o_{\mathrm{unbound}})\in(-\pi,\pi) . For symmetrically bounded continuous variables o_{\mathrm{bounded}}\in[-c,c] , we apply a scaling \frac{\pi}{c}\cdot o_{\mathrm{bounded}}\in[-\pi,\pi] . Conversely, when VQCs are utilized as downstream components, they process the latent feature vector h generated by the preceding feature extractor. This feature extractor may be classical, which yields unbounded activations, or quantum, which yields Pauli-Z expectation values bounded in [-1,1] . To accommodate both seamlessly, we uniformly apply the transformation 2\cdot\arctan(h)\in(-\pi,\pi) before passing intermediate state representations to the downstream VQCs. VI Experimental Settings VI-A Hybrid Model Variants We define eight hybrid variants based on the substitution of classical components with VQCs. Each replaced component is labeled with one of the letters: F for the feature extractor, O for the option-value function, T for the termination function, and P for the intra-option policies. We partition our evaluation into two groups. The first group examines compound architectures where a quantum feature extractor is paired with subsequent quantum downstream components: Hybrid_FO, Hybrid_FT, Hybrid_FP, as well as Hybrid_FOTP, in which all parameterized components are instantiated as VQCs. The second group isolates the effect of single-component substitutions: Hybrid_F, Hybrid_O, Hybrid_T, and Hybrid_P. VI-B Tested Environments and Baselines We consider two classical benchmarking environments from Gymnasium [ 42 ] : CartPole and Acrobot. 2 2 2 https://gymnasium.farama.org/ Our choice of environments is consistent with prior QRL work [ 21 , 15 , 7 , 36 ] . The two environments have continuous state spaces and discrete action spaces. In the CartPole task, the objective is to balance a pole on a moving cart by applying horizontal forces. The state space is 4-dimensional: cart position, cart velocity, pole angle, and pole angular velocity, and the action space consists of two discrete actions: pushing left or right. The reward range for CartPole is [0,500] . The Acrobot environment consists of a two-link chain with a fixed base and an actuated central joint. The goal is to apply torques on the actuated joint to swing the free end of the linear chain above a given height, starting from a downward-hanging position. Its state space is 6-dimensional, comprising the trigonometric functions of the two joint angles and their angular velocities, while the action space consists of three possible torques. The reward range for Acrobot is [-500,0] , with a penalty of -1 applied at each timestep until the goal is reached. See Fig. 5 for illustrations of the two environments. Across both environments, we benchmark our hybrid models against classical baselines and a random baseline (an agent sampling actions uniformly). All models are trained using Algorithm 1 . (a) CartPole (b) Acrobot Fig. 5 : The two tested environments: CartPole and Acrobot. VI-C Implementation Details We scale model architectures to match the complexity of each environment. For the classical baselines and any classical components within a hybrid agent, we use standard linear layers and multilayer perceptrons (MLPs) with ReLU activations. Detailed architectural configurations and their corresponding trainable parameter counts are provided in Table I and Table II , respectively. To encourage temporal abstraction and prevent option degeneracy, we apply a termination regularization of 0.01. Furthermore, we use an entropy regularization coefficient of 0.01 on the intra-option policies to encourage exploration. Actions are sampled from a softmax over the policy logits. All models are trained using the Adam optimizer [ 17 ] with a learning rate of 0.0005. We set the update frequency N_{\mathrm{critic}}=4 , and the target network is synchronized every N_{\mathrm{target}}=200 steps. We apply a standard discount factor \gamma=0.99 . The exploration rate \epsilon decays exponentially from an initial value of 1.0 to a minimum of 0.05. TABLE I : Architectural configurations. We summarize the model architectures for the classical and quantum variants. Note that all MLPs consist of a single hidden layer. Model parameters are reported in Table II . Environment Component Classical Quantum CartPole Feature Extractor MLP (8 neurons) VQC (4 layers) Downstream Linear layers VQC (1 layer) Acrobot Feature Extractor MLP (8 neurons) VQC (5 layers) Downstream MLP (3 neurons) VQC (2 layers) TABLE II : Model parameters. We report the number of trainable parameters in the classical and quantum components for option-critic with 2 options (Fig. 6 ). CartPole Acrobot Component Classical Quantum Classical Quantum Feature Extactor 76 48 110 90 Option-Value Function 10 12 29 36 Termination Function 10 12 29 36 Intra-Option Policies 20 24 66 72 VII Experiments Fig. 6 presents the results of the eight hybrid models and the two baselines, while Fig. 9 investigates the impact of the number of options on the classical and quantum intra-option policies. Further analyses and ablation studies are presented in Fig. 7 , Fig. 8 , and Fig. 10 . Table III reports the mean episodic reward, standard deviation (SD), and the relative reward against the classical baseline ( 1.00\times ) across both environments. Note that the relative reward ratio reflects the relative penalty incurred in Acrobot. Our results yield several distinct architectural insights regarding the efficacy of VQCs for quantum HRL. (a) Hybrid models with a quantum feature extractor and subsequent downstream quantum components. (b) Hybrid models with single quantum components. Fig. 6 : Performance of option-critic with different model architectures. We report the learning curves in CartPole and Acrobot. Curves represent mean episodic reward, and the shaded area represents the standard deviation. Each curve is averaged with a trailing window of 2000 steps and averaged over 10 runs with different random seeds. Remarkably, hybrid models can greatly outperform classical baselines in both environments. VII-A The Advantage of the Quantum Feature Extractor Empirical results across both environments demonstrate that utilizing a VQC exclusively for the feature extractor (Hybrid_F) is the most effective architectural strategy. Hybrid_F achieves 2.95 \times relative reward in CartPole and a 46% penalty reduction in Acrobot. This dominant performance is likely attributable to the expressivity of VQCs when processing classical environmental observations. Furthermore, coupling downstream quantum components with a quantum feature extractor consistently improves their performance. For instance, Hybrid_T achieves an mean reward of 59.25 in CartPole, and Hybrid_FT improves this to 201.99 (2.76 \times relative reward). However, utilizing VQCs solely for downstream components yields inconsistent results: Hybrid_T and Hybrid_P each outperform the classical baseline in one environment but not the other (Table III ), suggesting their effects may be task-specific. Overall, only utilizing a VQC for the feature extractor remains the optimal architecture. We further investigate the classical network size required to close the performance gap. We scale the classical feature extractor to 16, 24, and 32 hidden neurons while keeping all other components fixed. See Fig. 7 and Table III for the results. These correspond to 188, 260, and 332 model parameters in CartPole and 338, 442, and 546 in Acrobot, compared to 88 and 214 for Hybrid_F. We see that Hybrid_F exceeds the 24-neuron baseline in both environments (mean reward: 216.10 vs. 208.36 in CartPole; -125.71 vs. -150.10 in Acrobot) while saving 66% and 52% trainable parameters, respectively, and is surpassed by the 32-neuron baselines. Fig. 7 : Comparison with scaled classical baselines. We scale the classical feature extractor to 16, 24, and 32 hidden neurons while keeping all other components fixed. Hybrid_F exceeds the 24-neuron classical baselines in both environments (Table III ) while saving 66% and 52% trainable parameters in CartPole and Acrobot. VII-B Performance Degradation from the Quantum Option-Value Function While a quantum feature extractor yields substantial performance gains, substituting the option-value function with a VQC (Hybrid_O) within our architecture results in severe performance degradation. In CartPole, Hybrid_O fails to surpass even the Random baseline. Its performance in Acrobot is similarly close to the Random baseline. Consequently, employing a VQC for the option-value function is strongly discouraged. We further compare the learning dynamics of Hybrid_O against the classical baseline in terms of actor loss, critic loss, and policy entropy. Results are shown in Fig. 8 . In both environments, Hybrid_O exhibits policy entropy near the theoretical maximum of \ln(|\A|) throughout training (effectively sampling actions uniformly), flat critic loss close to zero, and flat actor loss with small standard deviations. In contrast, the classical baselines show active learning dynamics across all three metrics. These observations suggest that the quantum critic likely does not provide a useful learning signal. The precise root cause remains an open question; we note that barren plateaus are unlikely the cause given our shallow, few-qubit circuits [ 22 ] . One promising direction is to investigate whether trainable observables [ 6 ] can resolve this bottleneck with a more flexible spectral range. This bottleneck also explains the failure of the fully quantum-parameterized model. Substituting all parameterized parts as VQCs (Hybrid_FOTP) is not a viable choice; it achieves a mere 21.05 mean reward in CartPole and -498.94 in Acrobot, functionally equivalent to a random model. This collapse in learning is likely driven by the inclusion of the quantum option-value function, which negates the benefits provided by the quantum feature extractor. Fig. 8 : Analysis of the option-value bottleneck. We compare policy entropy, actor loss, and critic loss between Hybrid_O and the classical baselines in both environments. Hybrid_O shows high policy entropy (close to theoretical maximum), flat near-zero critic loss, and flat actor loss throughout training. In contrast, classical baselines show active learning dynamics. VII-C Impact of the Number of Options We investigate how expanding the number of options beyond |\Omega|=2 affects the learning dynamics of classical and quantum intra-option policies. As illustrated in Fig. 9 and Table III , scaling to 3 or 4 options does not yield a consistent pattern of improvement or degradation. This indicates that simply increasing available temporal abstraction does not reliably enhance the agents in these environments. Fig. 9 : Impact of the number of options on performance. We investigate how classical and quantum intra-option policies (Hybrid_P) scale with increased temporal abstraction. Results suggest no consistent pattern of improvement or degradation. TABLE III : Mean episodic reward and relative reward to the classical baseline. Bold indicates the best model among the eight hybrid variants and the two baselines. The relative reward in Acrobot reflects relative penalty, where lower is better. CartPole Acrobot Model Mean \pm SD Rel. Reward Mean \pm SD Rel. Reward Classical 73.13 \pm 117.52 1.00 \times -233.44 \pm 181.03 1.00 \times Random 22.23 \pm 11.83 0.30 \times -499.02 \pm 10.24 2.14 \times Hybrid_FOTP 21.05 \pm 11.25 0.29 \times -498.94 \pm 10.95 2.14 \times Hybrid_FO 20.51 \pm 11.10 0.28 \times -498.54 \pm 13.38 2.14 \times Hybrid_FT 201.99 \pm 169.23 2.76 \times -148.58 \pm 126.44 0.64 \times Hybrid_FP 142.75 \pm 124.84 1.95 \times -245.62 \pm 124.87 1.05 \times Hybrid_F 216.10 \pm 161.22 2.95 \times -125.71 \pm 99.99 0.54 \times Hybrid_O 18.21 \pm 12.09 0.25 \times -498.16 \pm 15.17 2.13 \times Hybrid_T 59.25 \pm 94.53 0.81 \times -154.09 \pm 110.33 0.66 \times Hybrid_P 126.12 \pm 128.26 1.72 \times -273.76 \pm 158.80 1.17 \times Classical (16 neurons) 167.25 \pm 178.29 2.29 \times -162.14 \pm 139.54 0.69 \times Classical (24 neurons) 208.36 \pm 191.10 2.85 \times -150.10 \pm 139.93 0.64 \times Classical (32 neurons) 221.80 \pm 191.75 3.03 \times -107.18 \pm 81.14 0.46 \times Classical (3 options) 36.38 \pm 75.31 0.50 \times -218.71 \pm 183.35 0.94 \times Classical (4 options) 88.58 \pm 112.75 1.21 \times -196.63 \pm 163.02 0.84 \times Hybrid_P (3 options) 87.43 \pm 103.77 1.20 \times -309.20 \pm 161.85 1.32 \times Hybrid_P (4 options) 65.83 \pm 99.32 0.90 \times -304.70 \pm 165.51 1.31 \times VII-D Impact of Architectural Choices The previous experiments do not reveal how architectural choices influence the performance of our hybrid agents. To address this, we run ablation experiments on the best-performing hybrid model, Hybrid_F, presented in Fig. 10 . Specifically, we vary one architectural choice at a time: incrementing or decrementing the VQC depth by 2 layers, fixing the input-scaling parameters \lambda to 1, or removing the entangling CNOT gates. By comparing the performance of these agents against the original Hybrid_F from Fig. 6 , we make the following observations: Fig. 10 : Influence of Architectural Choices. We ablate the best-performing hybrid model Hybrid_F (Fig. 6 ) in both environments by (i) incrementing or decrementing model depth by 2 layers, (ii) fixing the input-scaling parameters \lambda to 1, or (iii) removing the CNOT entangling gates. All three choices affect performance. (1) Influence of model depth: Increasing depth does not consistently improves performance, while decreasing depth degrades performance in both environments. This suggests that sufficient depth is necessary, but additional depth does not guarantee improvement. (2) Influence of scaling parameters \boldsymbol{\lambda} : Training the scaling parameters \lambda greatly benefits the learning performance of the hybrid agents. This observation is consistent with prior work [ 15 ] . (3) Influence of entanglement: Removing the entanglement degrades performance in both environments. This confirms that multi-qubit entanglement is beneficial for the hybrid agent. These results indicate that all three architectural choices—circuit depth, learnable input scaling, and entanglement—contribute to the quantum feature extractor’s advantage, and should be carefully considered when designing hybrid hierarchical agents. In conclusion, our results show that VQC placement within the hierarchy is a decisive factor. The quantum feature extractor is the only component where quantum substitution consistently improves performance, consistent with prior findings that VQCs can achieve competitive RL performance with fewer parameters than classical networks [ 21 , 7 ] . Further scaling experiments show that Hybrid_F outperforms the 24-neuron classical baseline while saving 66% and 52% parameters in CartPole and Acrobot, respectively. In contrast, the option-value bottleneck persists even when paired with a quantum feature extractor. Preliminary analysis shows that a quantum critic fails to learn meaningful option-value estimates, leaving policies near-uniform throughout training. Our ablations suggest that circuit depth, entanglement, and learnable input scaling are important architectural choices for maintaining the quantum advantage. VIII Discussion There are several limitations. First, since our choice of environments is consistent with prior QRL work [ 21 , 15 , 7 , 36 ] , whether our observations carry to more complex settings remains open. Second, although we provide a preliminary analysis of the option-value bottleneck, the precise root cause remains unidentified. Third, we only employed one training algorithm; the option-value bottleneck may behave differently under alternative learning schemes. Additionally, hybrid models require longer training time due to the cost of simulating quantum circuits. For future work, there are several key directions. First, more complex environments (e.g., MiniGrid, sparse-reward continuous control) could benefit more from temporal abstraction and better test our hybrid agents. The root cause of the option-value bottleneck deserves further investigation, including whether trainable observables can resolve it by providing learnable spectral range. Extending this work to other HRL paradigms would test the generality of our findings. Lastly, our simulations do not include any source of quantum noise. Investigating the influence of hardware or simulated noise on hybrid agents is an important and interesting direction. IX Conclusion This work introduces a hybrid quantum-classical option-critic framework that integrates VQCs into HRL. Our key findings are threefold: (1) a hybrid model utilizing a quantum feature extractor outperforms classical baselines while saving up to 66% parameters, and is only surpassed by classical models with substantially more capacity, (2) replacing the option-value function with a VQC causes severe learning failure, which is a critical architectural bottleneck, and (3) circuit depth, entanglement, and learnable input scaling are important factors for hybrid agents. Together, these results provide architectural guidelines for integrating quantum circuits into hierarchical RL agents. We believe our work brings practical quantum advantage in RL closer to realization on near-term quantum devices.