Central Path Proximal Policy Optimization
Nikola Milosevic, Johannes Müller, Nico Scherf · 2025
Read Central Path Proximal Policy Optimization by Nikola Milosevic, Johannes Müller, Nico Scherf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00700.txt.
Introduction
Nikola Milosevic, Johannes Müller, and Nico Scherf present a 2025 machine learning paper on training AI agents that must respect safety limits. In constrained decision problems, obeying the limits is usually assumed to cost final performance. Recent theory showed the limits can be built into the geometry of the policy itself, keeping the learning path near the ideal central route of a barrier method.
Their algorithm, C3PO, tweaks the popular PPO training loss so each update stays close to that central path. Against other on-policy methods it earned higher returns while enforcing constraints more tightly. The authors present central-path guidance as a promising direction for safe reinforcement learning in robotics and other limit-bound settings.
Catalog record
| CREATOR | Nikola Milosevic, Johannes Müller, Nico Scherf |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00700
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Central Path Proximal Policy Optimization Nikola Milosevic, Johannes Müller, Nico Scherf Abstract In constrained Markov decision processes, enforcing constraints during training is often thought of as decreasing the final return. Recently, it was shown that constraints can be incorporated directly in the policy geometry, yielding an optimization trajectory close to the central path of a barrier method, which does not compromise final return. Building on this idea, we introduce Central Path Proximal Policy Optimization (C3PO), a simple modification of the PPO loss that produces policy iterates, which stay close to the central path of the constrained optimization problem. Compared to existing on-policy methods, C3PO delivers improved performance with tighter constraint enforcement, suggesting that central path-guided updates offer a promising direction for constrained policy optimization. 1 Introduction Reinforcement learning (RL) has demonstrated impressive capabilities across a wide range of domains, yet real-world applications increasingly demand more than just reward maximization. In many real-world high-stakes environments—such as autonomous vehicles, healthcare, or robotic manipulation—agents must also avoid violating domain-specific safety or resource constraints. This motivates the study of constrained Markov decision processes (CMDPs), an extension of the standard RL framework that imposes expected cost constraints alongside the goal of reward maximization Altman 1999 . By treating feasibility and reward objectives separately, CMDPs provide a principled framework for specifying agent behavior in complex environments. CMDPs are especially relevant in deep reinforcement learning, where designing reward functions that reliably encode which states to avoid is notoriously difficult. Prior work has emphasized the importance of explicit constraint modeling for safe behavior. For instance, Ray et al. 2019 argue that CMDPs offer a natural formalism for benchmarking safe exploration in deep RL and introduce the Safety Gym suite to evaluate algorithms based on both task performance and cumulative safety cost. Similarly, Roy et al. 2022 demonstrate that task specifications expressed via constraints are often more natural and easier to design, especially as tasks increase in complexity, e.g. in finetuning LLMs Dai et al. 2023 . Despite their relatively low sample efficiency, model-free on-policy algorithms continue to play a foundational role in constrained RL. They offer conceptual clarity, support rigorous theoretical analysis, and provide strong baselines for studying the balance between performance and constraint satisfaction. As the field moves toward more scalable and sample-efficient approaches, insights developed in the on-policy setting remain central to both algorithm design and our broader understanding of safe learning, such as the policy improvement guarantees and constraint violation bounds introduced by Achiam et al. 2017 . In this context, there is a growing need for simple, scalable, and effective algorithms for solving CMDPs—ideally with properties similar to widely used algorithms such as proximal policy optimization (PPO; Schulman et al. 2017b ). PPO’s robustness, ease of implementation, and scalability have made it the method of choice in many deep RL and RLHF pipelines Ouyang et al. 2022 . We aim to extend these strengths to the constrained setting by developing an algorithm that shares PPO’s practical benefits while enforcing constraints in a principled CMDP framework. Specifically, we seek to achieve high final reward while approximately satisfying constraints, at least at convergence. To frame this problem, we distinguish between two commonly conflated settings in constrained RL: (i) safe exploration , where constraints must be satisfied throughout training, and (ii) safe convergence , where only the final policy is required to satisfy the constraints. Much of the literature has focused on the former, motivated by safety-critical applications in the real world. In contrast, safe convergence—where exploration may be unsafe—better reflects settings like simulation-based training or alignment finetuning Dai et al. 2023 .Typically, ensuring safety during training is considered to decrease the final performance achieved by an algorithm. We show the contrary and present an algorithm that exhibits strict feasibility during training as well as reliable feasibility and high return at convergence. Figure 1: Pictorial visualization of the central path argument from the main text. While a wide range of methods technically converge to an optimal feasible solution in the linear programming formulation of finite CMDPs (left), in the function approximation setting (right), approaching the constraint surface too early may result in higher sensitivity to local optima. In nonlinear CMDPs, the constraint surface is typically curved and nonconvex in policy parameter space. Converging prematurely or oscillating near the constraint boundary during training—as is often the case with Lagrangian or most trust region methods—can lead to unreliable constraint satisfaction at convergence. Furthermore, it can lead the optimizer to local optima that satisfy the constraints but fail to achieve high reward, see Figure 1 . Penalty and barrier methods avoid this by maintaining a feasible trajectory toward the constraint surface, yielding feasible solutions more reliably. However, barrier methods introduce bias ( Müller and Cayci 2024 ) , meaning the optimization problem obtained by adding a barrier penalty does not have the same solution set as the original problem, which can lead to degraded reward in policy optimization Milosevic et al. 2025 . Barrier methods either require careful tuning or an interior point approach Liu et al. 2020 to avoid harming reward performance. The recently proposed C-TRPO ( Milosevic et al. 2025 ) addresses these challenges by combining the strengths of trust-region and barrier methods by deriving a barrier-inspired trust-region formulation using strictly feasible trust regions. This results in an algorithm that acts like a barrier method with an adaptively receding barrier. This introduces no bias in the optimal feasible solution but still produces strictly feasible policies just like a barrier method. Further, C-TRPO produces policies, which are close to the regularization path obtained by altering the regularization strength, which is commonly known as the central path Boyd and Vandenberghe 2004 ; Müller and Cayci 2024 . This nicely illustrates how the constraints are incorporated in the algorithm’s geometry and ensures that C-TRPO and variants thereof produce policies which don’t prematurely approach the constraint surface. However, C-TRPO’s scalability remains limited due to computational overhead and sensitivity to large network architectures or batch sizes introduced by the TRPO-inspired update, and the update is defined only in the feasible set. To address the need for a simple and scalable CMDP solver, we propose a proximal version of C-TRPO. It also follows the central path, which we therefore call Central Path Proximal Policy Optimization (C3PO) . C3PO is a minibatch-based method that approximates the C-TRPO update using an exact penalty formulation, combining the simplicity and efficiency of PPO-style updates with the feasible geometry of central path methods. At its core, C3PO leverages the central path property of natural policy gradients to gradually guide the policy toward the constraint surface without inducing oscillations or premature convergence. The result is a practical algorithm that retains high reward performance while satisfying constraints at convergence, and which has the potential to scale well to large neural networks and modern deep RL settings. 2 Background We consider the infinite-horizon discounted constrained Markov decision process (CMDP) and refer the reader to Altman 1999 for a general treatment. The CMDP is given by the tuple \mathcal{M}\cup\mathcal{C} , consisting of a finite MDP \mathcal{M} and a set of constraints \mathcal{C} . The finite MDP \mathcal{M}=\{\mathcal{S},\mathcal{A},P,r,\mu,\gamma\} is defined by a finite state-space \mathcal{S} , a finite action-space \mathcal{A} , a transition kernel