Partially Equivariant Reinforcement Learning in Symmetry-Breaking Environments

Junwoo Chang, Minwoo Park, Joohwan Seo, Roberto Horowitz, Jongmin Lee, Jongeun Choi · 2025

Read Partially Equivariant Reinforcement Learning in Symmetry-Breaking Environments by Junwoo Chang, Minwoo Park, Joohwan Seo, Roberto Horowitz, Jongmin Lee, Jongeun Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00915.txt.

Introduction

Junwoo Chang and five coauthors blend symmetry into robot learning in a 2025 paper. Reinforcement learning teaches agents by reward, and many tasks have group symmetries, motions that leave the task unchanged. Respecting those symmetries helps an agent generalize, since it can reuse lessons across symmetric states and actions.

Real settings, though, almost never keep symmetries perfectly. Dynamics, actuator limits, and reward design break them, often only in places. The team's partially equivariant method keeps symmetry where it holds and relaxes it where it breaks. That balance could make robot policies both data-efficient and honest about real-world messiness.

Catalog record

CREATORJunwoo Chang, Minwoo Park, Joohwan Seo, Roberto Horowitz, Jongmin Lee, Jongeun Choi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00915

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Partially Equivariant Reinforcement Learning in Symmetry-Breaking Environments Junwoo Chang, Minwoo Park, Joohwan Seo, Roberto Horowitz, Jongmin Lee, Jongeun Choi Abstract Group symmetries provide a powerful inductive bias for reinforcement learning (RL), enabling efficient generalization across symmetric states and actions via group-invariant Markov Decision Processes (MDPs). However, real-world environments almost never realize fully group-invariant MDPs; dynamics, actuation limits, and reward design usually break symmetries, often only locally. Under group-invariant Bellman backups for such cases, local symmetry-breaking introduces errors that propagate across the entire state–action space, resulting in global value estimation errors. To address this, we introduce Partially group-Invariant MDP (PI-MDP), which selectively applies group-invariant or standard Bellman backups depending on where symmetry holds. This framework mitigates error propagation from locally broken symmetries while maintaining the benefits of equivariance, thereby enhancing sample efficiency and generalizability. Building on this framework, we present practical RL algorithms – Partially Equivariant (PE)-DQN for discrete control and PE-SAC for continuous control – that combine the benefits of equivariance with robustness to symmetry-breaking. Experiments across Grid-World, locomotion, and manipulation benchmarks demonstrate that PE-DQN and PE-SAC significantly outperform baselines, highlighting the importance of selective symmetry exploitation for robust and sample-efficient RL. 1 Introduction Group symmetries provide a powerful inductive bias in machine learning, enabling models to generalize efficiently. In robotics and continuous control, leveraging equivariance has been shown to improve data efficiency in both behavior cloning ( 40 ; 23 ; 22 ; 33 ; 30 ; 9 ; 41 ; 26 ; 25 ; 24 ; 27 ) , where the data collection is costly, and reinforcement learning (RL) ( 32 ; 11 ; 34 ; 35 ; 28 ; 16 ; 5 ; 17 ) , where exploration can be inefficient. Most existing equivariant RL methods are grounded in the notion of a group-invariant Markov Decision Process (MDP) ( 35 ; 36 ) , where invariance of the reward and transition functions implies symmetry in the optimal policy. In practice, however, these symmetry assumptions rarely hold exactly. Real-world environments introduce symmetry-breaking factors such as dynamics, actuation limits, or reward shaping. Under the Bellman backups based on the group-invariant MDP, even local violations of symmetry can introduce errors that propagate across the state–action space, leading to degraded value estimates, suboptimal policies, or even training failure. Prior works on approximate equivariance ( 5 ; 17 ) attempt to mitigate this challenge by relaxing equivariance globally, e.g., by modifying architectures to tolerate violations. While effective to some extent, these methods often lose the sample efficiency benefits of strict equivariance and can become unstable when symmetry-breaking is extensive, since equivariance is still applied indiscriminately across the entire space. To overcome this limitation, we introduce the framework of the Partially group-Invariant MDP (PI-MDP) , which selectively applies the group-invariant structure only in regions where symmetry is preserved (Fig. 1 ). Our approach builds on the derivation that local symmetry-breaking leads to one-step backup errors that propagate globally. By routing updates to the standard updates under the true MDP, we limit the propagation of one-step backup errors across the space. In particular, we detect symmetry-breaking regions via predictor disagreement outliers between an equivariant and an unconstrained one-step predictor, and apply standard rather than equivariant updates on those outliers while retaining equivariance elsewhere. Building on this framework, we develop practical reinforcement learning algorithms for both discrete and continuous control that retain the benefits of equivariance in symmetric regions while remaining robust to substantial symmetry-breaking. The contributions of our work are summarized as follows: 1) We analyze how local symmetry violations induce global value error via one-step backups, clarifying when selective symmetry is beneficial. 2) We introduce the Partially group-Invariant MDP (PI-MDP) and a practical RL formulation that uses equivariance where symmetry holds and falls back to standard updates where it breaks. 3) Across state-based discrete and continuous control experiments, we show that our method retains the sample efficiency gains of equivariance in symmetric regions and remains robust as symmetry-breaking increases, outperforming strict and approximate-equivariant baselines. Figure 1: Overview of partial equivariance in reinforcement learning. Equivariant networks provide strong inductive bias and sample efficiency in environments with exact symmetry. Left: In the symmetric case, the equivariant policy \pi_{E} exploits this structure and learns an optimal action a to reach the goal. Right: When the agent and goal are rotated by 90^{\circ} but a fixed obstacle (not represented in the agent’s state) remains in place, the symmetry of the true dynamics is broken. An exactly equivariant policy is forced to output the rotated action ga , which is invalid due to the obstacle in some cases, thereby corrupting training. Our framework introduces a gating function \lambda that detects such symmetry-breaking and activates the non-equivariant policy \pi_{N} , preserving robustness while retaining the sample efficiency benefits of equivariance in symmetric regions. 2 Related Work Group equivariance in continuous control. Recent works have applied group equivariance to imitation learning and classical control ( 40 ; 23 ; 22 ; 33 ; 30 ; 9 ; 41 ; 26 ; 25 ; 24 ) , demonstrating high data efficiency and generalization over baseline models. Parallel efforts have investigated group equivariance in reinforcement learning (RL) ( 32 ; 11 ; 34 ; 35 ; 28 ; 16 ) , showing improved sample efficiency compared to the conventional RL approaches. However, the effectiveness of equivariant RL remains limited in more general settings, such as robotic control tasks, where inherent symmetry-breaking often arises from factors including occlusions, environmental asymmetries, kinematic singularities, and complex dynamics. Approximate equivariance. Recent studies have proposed relaxing strict group equivariance to handle symmetry-breaking in data ( 5 ; 17 ; 38 ; 21 ; 31 ; 8 ) . Such approaches introduce approximate equivariance, enabling models to remain robust when exact symmetries do not hold. In reinforcement learning, approximate equivariant architectures have also shown improved robustness and efficiency against symmetry-breaking ( 5 ; 17 ) . For instance, 5 introduced residual pathways to the equivariant linear layers, while 38 proposed a relaxed equivariant convolutional layer with expanded kernel parameterizations, which were later adopted in the RL setting by 17 . However, these methods primarily focus on global relaxations of equivariance at the representation level. In contrast, our approach addresses symmetry-breaking by minimizing local equivariance errors during the Bellman backup, thereby preventing their global propagation through value updates. 3 Preliminaries Reinforcement learning. We consider a Markov decision process (MDP) defined as \mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma) where \mathcal{S} is the state space, \mathcal{A} is the action space, R:\mathcal{S}\times\mathcal{A}\to\mathbb{R} is the reward function, P(\cdot\mid s,a) is the transition kernel, and \gamma\in(0,1) is the discount factor. The agent learns a policy \pi to maximize the expected return, J=\mathbb{E}_{\pi,P}\Big[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\,\Big|\,s_{0}=s,a_{0}=a\Big] . The Bellman operator under a policy \pi is (\mathcal{T}^{\pi}Q)(s,a)=R(s,a)+\gamma\,\mathbb{E}_{s^{\prime}\sim P(\cdot\mid s,a)}\Big[\,\mathbb{E}_{a^{\prime}\sim\pi(\cdot\mid s^{\prime})}[Q(s^{\prime},a^{\prime})]\Big] , while the optimal (hard) Bellman operator is (\mathcal{T}Q)(s,a)=R(s,a)+\gamma\,\mathbb{E}_{s^{\prime}\sim P(\cdot\mid s,a)}\big[\max_{a^{\prime}}Q(s^{\prime},a^{\prime})\big] . Group equivariance. A symmetry is a transformation that preserves certain properties of a system ( 2 ) . The set of all symmetries forms a group , which satisfies associativity, identity, inverses, and closure. A group representation is a homomorphism \rho:G\to GL(n) that maps each group element g\in G to an invertible n\times n matrix. A function f:X\to Y is equivariant if \rho_{Y}(g)f(x)\;=\;f(\rho_{X}(g)x),\quad\forall g\in G,x\in X , where \rho_{X} and \rho_{Y} are

Read the complete text