QuantFPFlow: Quantum Amplitude Estimation for Fokker–Planck Policy Optimisation in Continuous Reinforcement Learning
Abraham Itzhak Weinberg · 2026
Read QuantFPFlow: Quantum Amplitude Estimation for Fokker–Planck Policy Optimisation in Continuous Reinforcement Learning by Abraham Itzhak Weinberg (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16429.txt.
Introduction
Weinberg introduces QuantFPFlow, a reinforcement learning framework using quantum amplitude estimation for Fokker-Planck policy optimization. Agents estimate the Fokker-Planck partition function at O(1/epsilon squared) cost; QuantFPFlow uses a Grover-amplified estimator at O(1/epsilon) for a quadratic speedup. The stationary distribution drives an exploration bonus steering the agent toward optimal regions. On a local-optima task, QuantFPFlow beats Soft Actor-Critic and finds the optimum 10.4 percent more often, keeping entropy near 6.5 nats versus 1.5 for SAC.
Catalog record
| CREATOR | Abraham Itzhak Weinberg |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29011 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16429
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QuantFPFlow: Quantum Amplitude Estimation for Fokker–Planck Policy Optimisation in Continuous Reinforcement Learning Abraham Itzhak Weinberg Abstract We introduce QuantFPFlow , a reinforcement learning framework that integrates quantum amplitude estimation into the Fokker–Planck (FP) formulation of stochastic policy optimisation. Classical continuous-space RL agents must estimate the FP partition function Z = ∫ e − V ( 𝐱 ) / D d 𝐱 Z=\int e^{-V(\mathbf{x})/D}\,d\mathbf{x} at cost 𝒪 ( 1 / ε 2 ) \mathcal{O}(1/\varepsilon^{2}) ; QuantFPFlow replaces this with a Grover-amplified amplitude estimator achieving 𝒪 ( 1 / ε ) \mathcal{O}(1/\varepsilon) —a provable quadratic speedup. While the full quantum acceleration requires fault-tolerant hardware, the quantum-inspired classical simulation demonstrated here already exhibits the 𝒪 ( 1 / ε ) \mathcal{O}(1/\varepsilon) algorithmic structure. The estimated stationary distribution ρ ∗ \rho^{*} drives a theoretically grounded exploration bonus r aug = r env + α log ( 1 / ρ ∗ ( s ) ) r_{\mathrm{aug}}=r_{\mathrm{env}}+\alpha\log(1/\rho^{*}(s)) . This bonus steers the agent toward globally optimal regions of multimodal reward landscapes while simultaneously constraining policy variance through FP diffusion matching. On a continuous-control task specifically designed to expose local-optima failure, QuantFPFlow achieves mean reward 1,295.7 ± 423.2 1{,}295.7\pm 423.2 versus 1,284.0 ± 474.0 1{,}284.0\pm 474.0 for Soft Actor-Critic (SAC), while discovering the global optimum 10.4 % more frequently (33.9 % vs. 30.7 %). Policy entropy remains near H ( π ) ≈ 6.5 H(\pi)\approx 6.5 nats throughout training, whereas SAC collapses to 1.5 1.5 nats, confirming that FP diffusion matching actively prevents premature convergence. Dimensionality experiments further show computational scaling of 𝒪 ( d 0.35 ) \mathcal{O}(d^{0.35}) for QuantFPFlow versus 𝒪 ( d 0.76 ) \mathcal{O}(d^{0.76}) for classical FP estimation. Keywords: Fokker–Planck equation, Quantum amplitude estimation, Reinforcement learning, Stochastic optimal control, Multimodal optimisation, Exploration 1 Introduction Reinforcement learning in continuous state–action spaces remains challenging due to the curse of dimensionality and the tendency of gradient-based policy optimisation to converge to local optima. Soft Actor-Critic (SAC) [ 4 ] and DDPG [ 8 ] address exploration through entropy regularisation and Ornstein–Uhlenbeck noise respectively, but both are ultimately greedy methods that collapse onto dominant reward modes when the landscape is multimodal. The Fokker–Planck (FP) equation offers a principled alternative [ 10 ] . Rather than optimising a deterministic policy, it characterises the evolution of a probability distribution over states under stochastic dynamics. When the drift is a gradient field f(\mathbf{x})=-\nabla V(\mathbf{x}) , the unique stationary solution is the Boltzmann distribution \rho^{*}(\mathbf{x})\propto\exp(-V(\mathbf{x})/D) , which encodes the globally optimal stochastic policy [ 6 , 3 ] . The computational bottleneck is the partition function Z=\int e^{-V(\mathbf{x})/D}\,d\mathbf{x} , which requires \mathcal{O}(1/\varepsilon^{2}) Monte Carlo samples for \varepsilon -precision estimation. Quantum amplitude estimation (QAE) [ 1 ] provides a quadratic speedup: the partition function can be encoded as the squared norm of a quantum state and estimated in \mathcal{O}(1/\varepsilon) queries via Grover-like amplification. This paper operationalises this connection, yielding QuantFPFlow , an FP-Actor-Critic with a quantum-enhanced exploration mechanism. Contributions. 1. QuantFPFlow algorithm : an FP-Actor-Critic that uses temperature-annealed QAE to compute a theoretically grounded exploration bonus, FP-regularised policy gradients, and adaptive diffusion matching. 2. Proven quadratic query speedup (Theorem 1 ): \mathcal{O}(1/\varepsilon) partition function estimation versus classical \mathcal{O}(1/\varepsilon^{2}) , with an empirical demonstration spanning three orders of magnitude in precision ( Figure 5 ). 3. Superior global optimum discovery : 10.4 % relative improvement over SAC on a multimodal continuous control benchmark, with sustained high policy entropy ( Figures 2 and 3 ). 4. Favourable dimensionality scaling : \mathcal{O}(d^{0.35}) versus classical FP \mathcal{O}(d^{0.76}) ( Figure 6 ). 2 Background This section reviews the two theoretical pillars of QuantFPFlow: the Fokker–Planck equation as a framework for stochastic policy optimisation, and quantum amplitude estimation as a tool for quadratic speedup in partition function computation. Together they motivate the design of our algorithm in Section 3 . 2.1 Fokker–Planck Equation in Reinforcement Learning Consider a continuous-state agent with Itô dynamics d\mathbf{x}=f(\mathbf{x})\,dt+\sqrt{2D}\,dW, (1) where f(\mathbf{x}) is the policy-induced drift, D>0 is the diffusion coefficient, and W is a standard Wiener process. The probability density \rho(\mathbf{x},t) satisfies the Fokker–Planck equation: \frac{\partial\rho}{\partial t}=-\nabla\cdot\bigl(f(\mathbf{x})\,\rho\bigr)+D\,\nabla^{2}\rho. (2) For gradient drift f(\mathbf{x})=-\nabla V(\mathbf{x}) , the unique stationary solution is \rho^{*}(\mathbf{x})=\frac{1}{Z}\,e^{-V(\mathbf{x})/D},\quad Z=\int_{\mathbb{R}^{d}}e^{-V(\mathbf{x})/D}\,d\mathbf{x}, (3) provided Z<\infty . This is the optimal stochastic policy for the control problem with running cost V(\mathbf{x}) and diffusion D [ 6 , 11 ] . 2.2 Quantum Amplitude Estimation Definition 1 (Quantum oracle for Z ) . Let \mathcal{X}=\{x_{1},\ldots,x_{N}\} be a uniform discretisation of the state space with N=2^{n} points. Define the oracle \mathcal{A} as the unitary preparing \mathcal{A}|0\rangle=\frac{1}{\sqrt{N}}\sum_{i=1}^{N}\left(\sqrt{1-p_{i}}\,|0\rangle+\sqrt{p_{i}}\,|1\rangle\right)|i\rangle,\quad p_{i}=\frac{e^{-V(x_{i})/D}}{M}, (4) where M=\max_{i}e^{-V(x_{i})/D} normalises probabilities to [0,1] . With this oracle, the amplitude a=\sqrt{\mathbb{E}[p_{i}]}=\sqrt{Z/(NM)} encodes the partition function. Quantum amplitude estimation [ 1 ] estimates a^{2} to additive precision \varepsilon using \mathcal{O}(1/\varepsilon) applications of \mathcal{A} and its inverse, via the quantum phase estimation subroutine applied to the Grover operator Q=\mathcal{A}(I-2|0\rangle\langle 0|)\mathcal{A}^{-1}(I-2|\chi_{1}\rangle\langle\chi_{1}|) , where |\chi_{1}\rangle marks the “good” states. We encode the FP stationary distribution as |\psi\rangle=\frac{1}{\sqrt{Z}}\sum_{x\in\mathcal{X}}e^{-V(x)/2D}|x\rangle, (5) so that |\langle x|\psi\rangle|^{2}=\rho^{*}(x) . Grover-like amplitude amplification then estimates these squared amplitudes with \mathcal{O}(1/\varepsilon) queries. 2.3 Related Work FP in RL. Kappen [ 6 ] showed stochastic optimal control reduces to a Boltzmann path integral. Todorov [ 11 ] connected this to KL 1 1 1 Kullback-Leibler -regularised RL. Our work operationalises this connection computationally via quantum estimation. Quantum RL. Dunjko et al. [ 2 ] established quantum speedups for model-based RL. Jerbi et al. [ 5 ] showed quadratic speedups for policy evaluation. QuantFPFlow is the first to exploit QAE specifically for FP partition function computation in continuous-space RL. Entropy-regularised RL. SAC [ 4 ] maximises an entropy-augmented reward. QuantFPFlow differs: entropy emerges from the FP diffusion structure via a consistency constraint rather than being added as a reward term. 3 QuantFPFlow Algorithm Building on the FP framework and QAE speedup established in Section 2 , we next present the full QuantFPFlow algorithm. The algorithm couples three components---a quantum amplitude estimator, an FP-guided actor, and a TD 2 2 2 Temporal Difference critic—into a unified training loop whose key innovation is using the FP stationary distribution both as a theoretically grounded exploration signal and as a structural constraint on policy variance. 3.1 Overview QuantFPFlow has three coupled components: (i) a QAE computing \rho^{*}(s) ; (ii) an FP-Actor updating policy mean \mu and log-std \log\sigma via FP-guided gradients; (iii) a linear Critic V_{\phi}(s)=\phi^{\top}s trained via TD learning. 3.2 Temperature-Annealed Quantum Amplitude Estimator Algorithm 1 Temperature-Annealed QAE 1: potential V(x) , range [a,b] , inverse temperature \beta , n_{\mathrm{qubits}} 2: estimated \hat{\rho}^{*}(x) 3: \mathbf{x}\leftarrow\mathrm{linspace}(a,b,\,2^{n_{\mathrm{qubits}}}) ; \mathbf{V}\leftarrow[V(x_{i})] 4: \rho_{\mathrm{acc}}\leftarrow\mathbf{0} 5: for \beta_{i} in \mathrm{linspace}(0.3\beta,\,\beta,\,6) do 6: \mathbf{a}\leftarrow\exp(-\beta_{i}\mathbf{V}/2) ; \mathbf{a}\leftarrow\mathbf{a}/\left\|\mathbf{a}\right\| 7: for k=1 to 5 do 8: \mathbf{a}\leftarrow 2\bar{a}\,\mathbf{1}-\mathbf{a} \triangleright inversion about mean 9: \mathbf{a}\leftarrow\max(\mathbf{a},0) ; \mathbf{a}\leftarrow\mathbf{a}/\left\|\mathbf{a}\right\| 10: end for 11: \rho_{\mathrm{acc}}\leftarrow\rho_{\mathrm{acc}}+\mathbf{a}^{2} 12: end for 13: return \rho_{\mathrm{acc}}/\sum\rho_{\mathrm{acc}} The outer temperature annealing loop ( Algorithm 1 , line 3) initialises with a flat distribution at \beta_{i}=0.3\beta and progressively sharpens the estimate toward \rho^{*} at \beta . This prevents premature collapse to a single mode—analogous to simulated annealing—while the inner Grover loop (lines 5–9) amplifies the probability mass of high-density regions. 3.3 Complexity Theorem We now state and prove the main theoretical result. Theorem 1 (Quadratic query