BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous control
Yifan Zhang, Liang Zheng · 2026
Read BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous control by Yifan Zhang, Liang Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16170.txt.
Introduction
Zhang, Zheng, and Zhang propose BAPR, a reinforcement learning method for control systems whose dynamics stay stable for long stretches then shift abruptly. A globally cautious policy wastes performance in stable periods, while a locally adaptive one risks failure at undetected shifts. BAPR unifies Bayesian online change detection with robust ensemble learning, turning maximally cautious after detected change-points and relaxing as confidence grows. All results are formally verified in Lean 4, with 22 machine-checked theorems and no placeholders.
Catalog record
| CREATOR | Yifan Zhang, Liang Zheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108017 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16170
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous control Code available at https://github.com/erzhu419/BAPR Yifan Zhang, Liang Zheng Abstract Real-world control systems frequently operate under piecewise stationary conditions, where dynamics remain stable for extended periods before undergoing abrupt regime changes. Standard robust RL methods face a practical dilemma: a globally conservative policy wastes performance during stable periods, while a locally adaptive policy can fail sharply when the regime changes undetected. We propose BAPR (Bayesian Amnesic Piecewise-Robust SAC), which unifies Bayesian Online Change Detection (BOCD) with robust ensemble RL. The abstract BAPR operator—a convex combination of mode-conditional Bellman operators weighted by a frozen belief distribution—is a γ \gamma -contraction when the belief and penalty terms are frozen during each Bellman backup. A complementary counterexample, machine-verified in Lean 4, gives a sharp threshold for a deliberately Q-dependent belief operator: in that construction the Lipschitz factor is γ + λ Δ \gamma+\lambda\Delta (where Δ \Delta is the mode reward gap), so contractivity is lost once γ + λ Δ ≥ 1 \gamma+\lambda\Delta\geq 1 . We assemble a component-wise error budget for the abstract mode-mixture operator, with the algebraic pieces mechanized in Lean; applying this budget to the shared-critic implementation relies on the stated frozen-parameter and function-approximation assumptions. BOCD drives an adaptive conservatism mechanism that becomes more conservative after detected surprise events and relaxes as confidence grows, with detection delay O ( log ( 1 / δ ) ) O(\log(1/\delta)) under mode-separability assumptions. A context-conditioning module trained via RMDM loss provides mode-aware representations from simulator-provided mode IDs at training time and requires no mode labels at deployment. Keywords: non-stationary reinforcement learning; Bayesian change-point detection; robust ensemble; piecewise stationarity; adaptive conservatism; context-conditioned policy. 1 Introduction Non-stationarity is a pervasive challenge in real-world control systems. From urban transit networks experiencing sudden traffic incidents to robotic systems undergoing mechanical wear, the environment dynamics that an agent faces are rarely fixed. Instead, many practical settings exhibit piecewise stationarity : the system operates under one regime for an extended period, then abruptly transitions to a different regime with substantially different dynamics [ 17 , 12 ] . Standard Reinforcement Learning (RL) algorithms implicitly assume stationarity—that the transition kernel P(s^{\prime}|s,a) and reward function R(s,a) remain fixed throughout training. When this assumption is violated, the agent’s learned value function becomes stale after a regime change, leading to severe performance degradation. This problem is especially acute for off-policy methods like Soft Actor-Critic (SAC) [ 7 ] , where the replay buffer accumulates transitions from expired regimes, effectively “poisoning” the critic with outdated data. Definition 1.1 (Regime Staleness Problem) . Let \mathcal{M}_{1},\mathcal{M}_{2},\ldots be a sequence of MDPs (regimes) encountered by the agent, with regime \mathcal{M}_{k} active during time interval [t_{k},t_{k+1}) . An off-policy agent with replay buffer \mathcal{D} suffers from regime staleness if, after a switch at time t_{k+1} : (i) Buffer contamination: A fraction \eta>0 of samples in \mathcal{D} come from \mathcal{M}_{k} , whose dynamics differ from the current \mathcal{M}_{k+1} ; (ii) Value misalignment: The learned Q-function Q_{t} satisfies \|Q_{t}-Q^{*}_{k+1}\|_{\infty}>\epsilon even as the number of samples from \mathcal{M}_{k+1} grows, due to the persistent bias from stale transitions. This is structurally distinct from the Q -value poisoning identified in stationary stochastic environments [ 24 ] : staleness arises from temporal model mismatch , not from conflating aleatoric and epistemic uncertainty. Existing approaches to non-stationary RL fall into two broad categories, each with significant limitations: 1. Reactive adaptation methods (e.g., context-conditioned policies [ 13 , 20 ] , meta-learning [ 15 , 5 ] ) adapt the policy based on recent observations but lack a principled mechanism for detecting when a regime change has occurred. They continuously adapt even during stable periods, introducing unnecessary variance. 2. Robust methods (e.g., Robust MDPs [ 21 ] , distributionally robust RL [ 26 , 18 ] ) hedge against worst-case uncertainty but use a static uncertainty set calibrated for the entire training process. In piecewise stationary environments, this leads to excessive conservatism during stable periods (when the current regime is well-characterized) and insufficient conservatism immediately after a regime change (when the uncertainty set from the previous regime no longer applies). We argue that the missing ingredient is a time-aware uncertainty quantification mechanism that can: • Detect regime changes in real-time from observable signals; • Modulate the degree of conservatism based on confidence in the current regime; • Forget stale information from expired regimes in a principled manner. To this end, we propose BAPR (Bayesian Amnesic Piecewise-Robust SAC), which integrates Bayesian Online Change Detection (BOCD) [ 1 ] into the robust ensemble SAC framework of RE-SAC [ 24 ] . BOCD maintains a posterior distribution \rho(h) over run-lengths h (time since the last change-point), and this belief distribution serves as a bridge between change-point detection and adaptive conservatism. We use the variance-growing likelihood \sigma^{2}_{0}+\sigma_{g}h (Eq. 8 ) so that anomalous surprise pushes posterior mass toward larger run-lengths, raising the expected run-length \bar{h} above its steady-state baseline; the deviation \bar{h}/(H{-}1)-\bar{\lambda}_{w}^{\text{EMA}} is the change-point signal that drives \beta_{\text{eff}} (§ 4.3 ). The semantics is therefore “surprise raises \bar{h} above its EMA baseline,” not “surprise resets \bar{h} to zero”; we adopt this design because the EMA-deviation signal is less brittle to surprise-model mis-specification than a fixed absolute threshold in our experiments. Building upon RE-SAC’s disentangled treatment of aleatoric and epistemic risk [ 24 ] , and extending the Environment-aware Soft actor-Critic with stochastic Policy (ESCP) [ 13 ] framework for context-conditioning, our contributions are: • Operator-level characterization of frozen beliefs: We establish contraction for the BAPR operator—a convex combination of mode-conditional operators with frozen belief weights—under the finite-state assumptions of Theorem 4.2 . For a constructed Q-dependent belief operator, the contraction factor becomes exactly \gamma+\lambda\Delta (where \Delta is the mode reward gap), and contractivity is lost when \gamma+\lambda\Delta\geq 1 (Theorem D.1 ). This identifies a concrete stability boundary that is distinct from the frozen-penalty requirement in RE-SAC. The core finite-state/operator lemmas are machine-verified in Lean 4 with no sorry ( BAPR.lean : 560 lines; BAPR-Counterproof.lean : 265 lines). • Conditional piecewise convergence-rate bound: Under three structural assumptions—Mode Separability (Assumption B.1 ), Lipschitz Surprise (Assumption B.2 ), and Metastable Period (Assumption B.3 )—we state a Piecewise Q-Value Convergence Rate Theorem (Appendix A ) showing that after each regime switch, the abstract Q-value error decays as \gamma^{n}\cdot E_{\text{switch}}+(\varepsilon_{\text{proj}}+\sigma)/(1-\gamma) . Under these assumptions, the recovery-count expression is polynomial in the listed parameters rather than exponential in the risk-sensitive horizon term. The algebraic components of this bound correspond to machine-verified Lean 4 lemmas. • Adaptive conservatism via BOCD: We derive a belief-weighted penalty \lambda_{w} from the BOCD posterior that modulates the LCB coefficient \beta in real-time. We prove (machine-verified) that \beta_{\text{eff}}\leq\beta_{\text{base}} and that \beta_{\text{eff}} is monotonically decreasing in surprise magnitude, so false positives can only increase conservatism in the surrogate LCB objective rather than make it more optimistic than the RE-SAC baseline (Appendix G ). • Context-conditioning with RMDM: We introduce a context-conditioning module trained via a Representation learning for Mode Detection and Matching (RMDM) loss; training uses the simulator-provided mode IDs as a supervised target, while inference at deployment requires no mode labels (the embedding is computed from observed transitions). • Detection delay and operator-level error budget (Theorem E.2 ):