KFCPO: Kronecker-Factored Approximated Constrained Policy Optimization
Joonyoung Lim, Younghwan Yoo · 2025
Read KFCPO: Kronecker-Factored Approximated Constrained Policy Optimization by Joonyoung Lim, Younghwan Yoo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00880.txt.
Introduction
Joonyoung Lim and Younghwan Yoo propose KFCPO, an algorithm for safe reinforcement learning, where an agent must earn reward while respecting cost limits. Second order methods can train such agents steadily, but the standard versions are slow, and clashes between reward and safety goals can destabilize learning.
KFCPO uses a Kronecker factored approximation to make natural gradient updates cheap and closed form, layer by layer. A margin aware rule blends the reward and cost gradients according to how close the agent sits to a safety boundary, avoiding harsh fixed thresholds. A rollback step keeps updates inside a trust region. On Safety Gymnasium tasks, returns rose ten to fifty percent over the best baseline that stayed within the safety limits.
Catalog record
| CREATOR | Joonyoung Lim, Younghwan Yoo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-00880
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KFCPO: Kronecker-Factored Approximated Constrained Policy Optimization Thanks: Corresponding author: Younghwan Yoo ( [email protected] ). Joonyoung Lim, Younghwan Yoo Abstract . We propose KFCPO, a novel Safe Reinforcement Learning (Safe RL) algorithm that combines scalable Kronecker-Factored Approximate Curvature (K-FAC) based second-order policy optimization with safety-aware gradient manipulation. KFCPO leverages K-FAC to perform efficient and stable natural gradient updates by approximating the Fisher Information Matrix (FIM) in a layer-wise, closed-form manner, avoiding iterative approximation overheads. To address the trade-off between reward maximization and constraint satisfaction, we introduce a margin-aware gradient manipulation mechanism that adaptively adjusts the influence of reward and cost gradients based on the agent’s proximity to safety boundaries. This method blends gradients using a direction-sensitive projection, eliminating harmful interference and avoiding abrupt changes caused by fixed hard thresholds. Additionally, a minibatch-level KL rollback strategy is adopted to ensure trust region compliance and to prevent destabilizing policy shifts. Experiments on Safety Gymnasium using OmniSafe show that KFCPO achieves 10.3%–50.2% higher average return across environments compared to the best baseline that respected the safety constraint, demonstrating superior balance of safety and performance. † † footnotetext: MSC2020: Primary 00A05, Secondary 00A66. 1. Introduction Reinforcement Learning (RL) has achieved remarkable success across a variety of domains, including robotics, autonomous systems, and industrial control. However, most RL algorithms remain confined to simulation due to the risks associated with unsafe behavior during training or deployment in real-world environments. This has led to increasing interest in Safe Reinforcement Learning (Safe RL), also known as constrained Reinforcement learning, which seeks to maximize expected return while adhering to predefined safety constraints, typically formulated as cumulative cost thresholds. To address safety concerns, many Safe RL algorithms have been proposed. Achiam et al. [ 1 ] introduced Constrained Policy Optimization (CPO), which applies trust region updates with second order approximations to enforce safety constraints. Subsequent works, including Projected Constrained Policy Optimization (PCPO) by Yang et al. [ 2 ] , extended the CPO method through projection-based techniques. Meanwhile, Ray et al. [ 3 ] explored Lagrangian-based methods, notably PPO-Lag and TRPO-Lag, which integrate cost constraints into the reward objective via dual variables. These methods build upon PPO [ 4 ] and TRPO [ 5 ] , two widely used first-order and second order policy optimization algorithms, respectively. While Lagrangian methods are attractive for their computational simplicity and compatibility with existing RL frameworks, CPO and its variants offer stronger theoretical guarantees for constraint satisfaction and more precise control over policy updates. This makes second order approaches particularly appealing in complex, safety-critical scenarios, when computational resources are sufficient. However, recent benchmark evaluations by Ji et al. [ 6 , 7 ] and Ray et al. [ 3 ] have shown that CPO and similar second order methods often fail to enforce constraints reliably in high dimensional or partially observable environments. This has revealed two key challenges in constrained RL: (1) approximation errors introduced by the use of conjugate gradient based approximations with limited iterations, and (2) the difficulty of dynamically balancing reward maximization and safety enforcement when the objectives conflict. These findings contradict the results reported in the original CPO paper [ 1 ] and emphasize the need for new approaches that better address both optimization accuracy and reward-safety coordination. Addressing the first challenge optimization accuracy, requires a stable and efficient second order optimizer. To this end, we examine the Kronecker-Factored Approximate Curvature (K-FAC) method, which approximates the Fisher Information Matrix (FIM) using structured Kronecker products to achieve efficient computation with reduced complexity and faster processing. Originally introduced by Martens and Grosse [ 8 ] , K-FAC has been extended to support convolutional [ 9 ] and recurrent layers [ 10 ] , enabling its application to a wide range of neural architectures. K-FAC was later extended to reinforcement learning by Wu et al. [ 11 ] , who applied it in the Actor-Critic using Kronecker Factored Trust Region (ACKTR) algorithm. Their work demonstrated that K-FAC not only improves training stability and sample efficiency but also provides strong experimental evidence of its effectiveness when applied to reinforcement learning. However, to the best of our knowledge, K-FAC has not yet been applied in the context of Safe RL. For the second challenge, reward-safety coordination, researchers have recently investigated gradient manipulation techniques for coordinating conflicting optimization signals. Yu et al. [ 12 ] proposed gradient surgery algorithm as Projecting Conflicting Gradient (PCGrad), which reduces interference by projecting each task’s gradient onto the normal plane of others in multi-task learning. Building on this direction, Gu et al. [ 13 ] proposed a soft switching gradient manipulation method for Safe RL, which blends reward and cost gradients by removing conflicting components before combining them. However, their method applies fixed blending ratios within a predefined slack region, making it insensitive to the agent’s actual safety state. Additionally, the introduction of extra hyperparameters (e.g., slack size) increases tuning complexity and reduces robustness across environments. Building on these insights, we propose KFCPO, a novel Safe RL algorithm that integrates K-FAC for scalable and stable second order optimization with a safety margin aware gradient manipulation mechanism. Our method dynamically adjusts the optimization direction based on the agent’s proximity to constraint violation, avoiding reliance on hard thresholds or manually tuned slack regions. To further improve stability, we introduce a minibatch level Kullback-Leibler (KL) divergence rollback strategy to prevent unexpected policy shifts and preserve trust region behavior. Together, these components enable effective and stable learning in complex, safety-constrained, and partially observable environments. Our contributions are summarized as follows: • We introduce KFCPO, the first Safe RL algorithm that incorporates K-FAC into Safe RL. • We propose a novel margin-aware gradient manipulation mechanism that dynamically balances reward and cost gradients based on the agent’s proximity to the safety threshold. • We conduct experiments on the Safety Gymnasium benchmark using the OmniSafe framework, comparing KFCPO with a wide range of baselines including state-of-the-art Safe RL algorithms. Across all environment, KFCPO achieved 10.3% to 50.2% higher average return compared to the best performing baseline that respected the cost limit, demonstrating its superior ability to balance safety and performance. 2. Preliminaries In this section, we present the background and notations necessary to describe our Safe RL algorithm, including the problem formulation and the core optimization techniques it builds upon. We briefly review the formulation of Constrained Markov Decision Processes (CMDPs), natural gradient based optimization and K-FAC method, which together form the foundation of our approach. 2.1. Constrained Markov Decision Process RL is commonly formulated as a Markov Decision Process (MDP), defined by the tuple (\mathcal{S},\mathcal{A},P,R,\gamma) , where \mathcal{S} is the state space, \mathcal{A} is the action space, P(s^{\prime}|s,a) is the transition probability, R(s,a) is the reward function, and \gamma\in[0,1) is the discount factor. The objective is to learn a policy \pi(a|s) that maximizes the expected discounted return: (1) J(\pi)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\right]. To incorporate safety constraints, this framework is extended to a CMDPs, introduced by Altman [ 14 ] , where a cost function C(s,a) quantifies constraint violations and a threshold d bounds the expected cumulative cost. The constrained objective becomes: (2) \max_{\pi}J(\pi)\quad\text{s.t.}\quad J_{C}(\pi)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}C(s_{t},a_{t})\right]\leq d. This constrained formulation introduces a trade-off between maximizing performance and satisfying safety requirements. 2.2. Natural Gradient Sutton et al. [ 15 ] proposed the use