Soft Equivariance Regularization for Invariant Self-Supervised Learning
Joohyung Lee, Changhun Kim, Hyunsu Kim, Kwanhyung Lee, Juho Lee · 2026
Read Soft Equivariance Regularization for Invariant Self-Supervised Learning by Joohyung Lee, Changhun Kim, Hyunsu Kim, Kwanhyung Lee, Juho Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-06693.txt.
Introduction
Joohyung Lee and colleagues propose Soft Equivariance Regularization, a small add-on that splits two training goals. The main self-taught goal stays on the final code, while a soft push toward shape-aware behavior acts on an earlier map of image parts. That split eases a trade-off seen when both goals share one layer. Tests show gains on image sorting, warped images, and object finding.
Catalog record
| CREATOR | Joohyung Lee, Changhun Kim, Hyunsu Kim, Kwanhyung Lee, Juho Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56225 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.06693
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Soft Equivariance Regularization for Invariant Self-Supervised Learning Joohyung Lee, Changhun Kim, Hyunsu Kim, Kwanhyung Lee, Juho Lee Abstract Self-supervised learning (SSL) typically learns representations invariant to semantic-preserving augmentations (e.g., random crops and photometric jitter). While effective for recognition, enforcing strong invariance can suppress transformation-dependent structure that is useful for robustness to geometric perturbations and spatially sensitive transfer. A growing body of work, therefore, augments invariance-based SSL with equivariance objectives, but these objectives are often imposed on the same final (typically spatially-collapsed) representation. We empirically observe a trade-off in this coupled setting: pushing equivariance regularization toward deeper layers improves equivariance scores but degrades ImageNet-1k linear evaluation, motivating a layer-decoupled design. Motivated by this trade-off, we propose Soft Equivariance Regularization (SER) , a plug-in regularizer that decouples where invariance and equivariance are enforced: we keep the base SSL objective unchanged on the final embedding, while softly encouraging equivariance on an intermediate spatial token map via analytically specified group actions ρ g \rho_{g} (e.g., 90 ∘ 90^{\circ} rotations, flips, and scaling) applied directly in feature space. SER learns/predicts no per-sample transformation codes/labels, requires no auxiliary transformation-prediction head, and adds only 1.008 × \times training FLOPs. On ImageNet-1k ViT-S/16 pretraining, SER improves MoCo-v3 by +0.84 Top-1 in linear evaluation under a strictly matched 2-view setting and consistently improves DINO and Barlow Twins; under matched view counts, SER achieves the best ImageNet-1k linear-eval Top-1 among the compared invariance+equivariance add-ons. SER further improves ImageNet-C/P by +1.11/+1.22 Top-1 and frozen-backbone COCO detection by +1.7 mAP. Finally, applying the same layer-decoupling recipe to existing invariance+equivariance baselines (e.g., EquiMod and AugSelf) improves their accuracy, suggesting layer decoupling as a general design principle for combining invariance and equivariance. Code is available at https://github.com/aitrics-chris/SER . 1 Introduction Self-supervised learning (SSL) is a central paradigm for visual representation learning ( 5 ; 4 ; 34 ) , enabling strong features to be learned from large-scale unlabeled data. Many successful SSL methods follow a simple principle: learn representations that are invariant to a distribution of semantic-preserving augmentations, such as random crops and photometric jitter. This invariance is highly effective for recognition, but it can also suppress transformation-dependent structure (e.g., orientation, reflection, or scale cues) useful for geometric robustness and spatially sensitive transfer. A complementary principle is equivariance : rather than discarding transformation information, an equivariant representation changes in a predictable and structured way when the input is transformed ( 10 ; 27 ) . Prior work that incorporates equivariance into SSL can be broadly grouped into two families ( 36 ) . Implicit approaches encourage transformation awareness through auxiliary objectives (e.g., predicting or contrasting transformations) ( 10 ; 25 ) . Explicit approaches aim to model how features transform in latent space, often introducing auxiliary transformation/action modules or learning/predicting per-sample transformation codes/labels as training targets ( 12 ; 31 ; 15 ; 36 ) . At ImageNet scale, equivariant SSL has also been explored in predictive/world-model settings ( 16 ) . In this work, we study a complementary problem: how to incorporate an explicit equivariance regularizer on top of strong invariance-based SSL backbones (MoCo-v3, DINO, Barlow Twins) for ViTs, without changing the architecture or the baseline invariance objective. A common design choice in prior invariant+equivariant SSL methods is to impose both objectives on the final representation. However, the final representation is typically spatially collapsed, and thus poorly aligned with spatial group actions. More importantly, we empirically observe a trade-off when equivariance regularization is pushed toward the output: equivariance scores increase, but ImageNet-1k linear-evaluation accuracy consistently decreases ( Figures 3 and 4 ). This motivates a decoupled design in which invariance and equivariance are encouraged at different layers . We propose Soft Equivariance Regularization (SER) , a simple yet scalable regularizer that decouples where invariance and equivariance are enforced. SER trains the final embedding with an invariant baseline SSL objective, while softly regularizing spatially structured intermediate representations toward equivariance via analytically known feature-space group actions \rho_{g} . Since some common SSL augmentations (notably random cropping) are non-invertible and do not form a group, SER uses a simple batch-partition strategy: the full augmentation pipeline contributes to the invariance objective, while equivariance is enforced only for an invertible geometric subset. Transformation labels. Following the terminology of STL ( 36 ) , by “no transformation labels” we mean that SER does not learn or predict per-sample transformation labels/codes as additional supervision. SER still assumes the standard (known) augmentation pipeline and uses the resulting relative transform g=g_{2}g_{1}^{-1} , but does not introduce a transformation-prediction head. We evaluate SER on ImageNet-1k pretraining of ViT-S/16 and a suite of robustness and transfer benchmarks. Because the number of views can significantly affect SSL performance, we emphasize matched-view comparisons and report results under strictly matched settings. Across strong invariance-based SSL methods (MoCo-v3, DINO, Barlow Twins), SER consistently improves ImageNet-scale classification and robustness (e.g., ImageNet-C/P), and yields larger gains on spatially sensitive transfer such as frozen-backbone COCO detection. Furthermore, we show that the same layer-decoupling modification can improve existing invariance+equivariance baselines, suggesting a general recipe for combining invariance and equivariance in practice. Contributions. Our contribution is fourfold: • Empirical trade-off at the final layer. We show that imposing invariance and equivariance on the same final representation can be suboptimal: pushing equivariance regularization toward later layers increases equivariance scores but reduces ImageNet-1k linear-evaluation accuracy ( Figures 3 and 4 ). • Layer-decoupled soft equivariance regularization. We propose SER, which promotes equivariance on an intermediate spatial representation while leaving the baseline SSL loss unchanged on the final embedding. • Analytic feature-space group actions without extra modules. SER applies analytically specified actions \rho_{g} to intermediate feature maps, avoiding auxiliary transformation/action modules and avoiding learned/predicted per-sample transformation codes/labels. • Layer decoupling upgrades prior inv+equiv methods. Beyond our loss design, we show that moving the equivariance objective of prior invariant+equivariant methods (e.g., EquiMod, AugSelf) from the final layer to an intermediate layer improves their accuracy as well, indicating that layer decoupling is broadly beneficial. 2 Backgrounds 2.1 Self-Supervised Learning Self-supervised learning (SSL) leverages supervisory signals derived from the data itself, reducing reliance on human-annotated labels. Early SSL methods used pretext tasks such as rotation prediction ( 17 ) or solving jigsaw puzzles ( 28 ) . Modern SSL in vision predominantly learns representations by enforcing invariance across augmented views of the same image, often via instance discrimination or redundancy-reduction objectives ( 5 ; 20 ; 18 ; 37 ; 1 ) . Concretely, a training sample is transformed into multiple views by independently sampling augmentations from a predefined distribution (e.g., crops and photometric jitter), and the model is optimized so that representations of these views agree. Different SSL frameworks instantiate this invariance principle with different objectives. Contrastive approaches such as SimCLR and MoCo (including MoCo-v3) use noise-contrastive estimation losses ( 5 ; 20 ; 6 ) , while non-contrastive methods such as BYOL and SimSiam minimize a similarity loss between predicted and target embeddings ( 18 ; 6 ) . Redundancy-reduction methods such as Barlow Twins and VICReg enforce agreement while discouraging collapse through covariance-based regularization ( 37 ; 1 ) . Many frameworks use two global views by default, while increasing the number of views (e.g., multi-crop training) can improve performance at the cost of additional compute