Formal Verification of Noisy Quantum Reinforcement Learning Policies
Dennis Gross · 2025
Read Formal Verification of Noisy Quantum Reinforcement Learning Policies by Dennis Gross (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01502.txt.
Introduction
Dennis Gross studies how to verify policies learned by quantum reinforcement learning. Such policies use quantum traits to pick actions in sequence. They aim to beat classic policies on some tasks. But quantum hardware adds noise from measures and flaws.
That noise makes a policy's acts less sure. Gross works on formal checks that hold under that noise. A checked policy can be trusted to meet set bounds. Trust matters if quantum agents are to run in labs or plants where a bad act has a real cost. Noise-aware checks also state plainly what they cannot promise, so users know which policy acts stay open to hardware drift over time.
Catalog record
| CREATOR | Dennis Gross |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01502
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Formal Verification of Noisy Quantum Reinforcement Learning Policies Dennis Gross Abstract . Quantum reinforcement learning (QRL) aims to leverage quantum phenomena to develop sequential decision-making policies that achieve task objectives more effectively than their classical counterparts. Unlike reinforcement learning policies implemented on classical hardware, QRL policies are subject to additional uncertainty from quantum measurements and hardware imperfections. These uncertainties, including bit-flip, phase-flip, and depolarizing noise, can alter agent behavior and lead to unsafe policy behaviour. However, there do not exist approaches that provide systematic methods for verifying exactly whether trained QRL policies satisfy safety requirements under specific quantum noise conditions. We present QVerifier , a formal verification method that uses probabilistic model checking to rigorously analyze trained QRL policies, with and without modeled quantum noise, to verify satisfaction or violation with safety properties. Our method incrementally constructs a formal model of the policy–environment interaction by expanding all states reachable under the policy and assigning their transition probabilities using the model’s dynamics. Quantum measurement uncertainty and optional quantum hardware noise are incorporated directly into these probabilities. The resulting model is then verified using the Storm model checker to assess satisfaction of the safety properties. Experiments across multiple QRL environments show that our approach precisely quantifies how different models of quantum noise affect safety guarantees. The results reveal not only how specific noise models degrade policy performance, but also cases where certain noise can improve it. By enabling rigorous safety verification before deployment, QVerifier addresses a critical need: because access to quantum hardware is prohibitively expensive, pre-deployment verification is essential for any safety-critical use of quantum reinforcement learning. It targets a potential classical–quantum sweet spot: trained QRL policies that execute efficiently on quantum hardware, yet remain tractable for classical probabilistic model checking despite being too slow for real-time classical deployment. 1. Introduction As quantum computing [ 56 ] transitions from theory to commercial reality [ 46 , 55 , 1 ] , it opens new opportunities for advancing machine learning [ 13 , 10 , 2 ] . Quantum reinforcement learning (QRL) combines reinforcement learning (RL) with quantum computing to train sequential decision-making policies that may achieve task objectives more effectively than their classical counterparts by leveraging quantum phenomena such as superposition and entanglement [ 36 , 14 , 50 , 65 , 15 ] . In general, an RL agent aims to learn a near-optimal policy to achieve a fixed objective by taking actions and receiving feedback in the form of rewards and state observations from the environment [ 67 ] . A policy is memoryless if it depends only on the current state, and it can be stochastic , assigning action probabilities from which the agent samples its next action. In QRL, policies are encoded by variational quantum circuits (VQCs) [ 65 ] , which are parameterized circuits optimized during training. A VQC takes a quantum encoding of the environment state and produces a quantum state, from which action probabilities are estimated via repeated measurements (shots). Each shot measures the circuit on the same encoded input , yielding a single classical outcome by projecting the quantum state onto one of its possible results; many such shots are needed to approximate the underlying action distribution. VQCs can be executed on quantum simulators or quantum hardware [ 75 ] , though simulators become exponentially slower for large circuits [ 16 ] . As research in QRL progresses into safety-critical systems [ 40 , 57 , 45 ] , it is crucial to ensure that these QRL policies are developed with safety in mind [ 10 , 66 ] . However, trained policies may exhibit unsafe behavior [ 22 ] , such as collisions [ 9 , 64 ] , since rewards rarely capture complex safety requirements [ 69 ] . In QRL, additional uncertainty from stochastic measurements and device noise [ 68 , 5 ] on costly quantum computing hardware [ 52 ] can further distort action selection. To resolve the safety issues mentioned above, formal verification methods like probabilistic model checking [ 22 ] have been proposed to reason about the safety of RL on classical hardware [ 73 , 33 , 12 , 30 ] and the correctness of quantum systems [ 21 , 8 ] . Model checking is not limited by properties that rewards can express but supports a broader range of properties that can be expressed by probabilistic computation tree logic (PCTL) [ 32 ] , a temporal logic designed to express probabilistic branching-time properties. At its core, probabilistic model checking uses mathematical models to verify a system’s correctness with respect to a given safety property [ 7 ] . However, in the context of QRL policy verification, probabilistic model checking has not yet been applied, and there are limited tools for systematic comparisons of QRL policies [ 38 , 11 , 41 ] . In this paper , we introduce a method, QVerifier , for model-checking trained QRL policies against safety properties while explicitly accounting for quantum uncertainty arising from measurements and quantum hardware noise. Given a trained QRL policy, a quantum noise model, and a formal environment model, we incrementally build a formal representation of the combined policy–noise-environment system, exploring only the states actually reachable under the trained QRL policy. We then use the probabilistic model checker Storm [ 34 ] to verify whether the resulting formal model satisfies the desired safety property. Our experiments demonstrate that probabilistic model checking enables rigorous verification of trained QRL policies with and without noise with respect to safety properties, before deploying them on costly quantum hardware. We verify six policies (three QRL and three classical RL baselines) across three environments (Frozen Lake, Ski, and Freeway) under four quantum noise models: bit-flip, phase-flip, depolarizing, and amplitude damping that are applied independently at each quantum circuit gate, though QVerifier is not limited to these noise models. Our results reveal that noise effects are policy- and task-dependent: while bit-flip and depolarizing noise consistently degrade performance, we observe that low-level amplitude-damping noise can sometimes improve QRL policy performance, with the Ski environment showing a 27% improvement over the classical RL baseline. These findings demonstrate that pre-deployment verification not only identifies safety violations but also uncovers beneficial quantum noise regimes. Our main contributions are a method for formally verifying trained QRL policy behaviors in modeled environments, with and without quantum noise models, and an exact comparison approach for QRL and classical RL performance. By enabling rigorous safety verification before deployment, QVerifier addresses a critical need: access to quantum hardware is prohibitively expensive, making pre-deployment verification essential for safety-critical applications. QVerifier targets a potential sweet spot between classical and quantum computation, where trained QRL policies execute efficiently on quantum hardware but remain feasible to simulate classically for offline model checking. Moreover, the methodology underlying QVerifier could inform the design of future quantum model checkers. 2. Related Work Research at the intersection of quantum computing, machine learning, and software engineering has grown rapidly in recent years [ 10 ] . A key challenge across these areas is dealing with quantum oise, which affects both quantum circuits and the software built on top of them. Noise-aware quantum software engineering, therefore, forms an active research direction [