Learning Group Actions In Disentangled Latent Image Representations

Farhana Hossain Swarnali, Miaomiao Zhang, Tonmoy Hossain, Genuity Systems Limited, Computer Science, Electrical, Computer Engineering, School of Engineering, Applied Science, University of Virginia, [email protected], {mz8rr,tonmoy}@virginia.edu · 2025

Read Learning Group Actions In Disentangled Latent Image Representations by Farhana Hossain Swarnali, Miaomiao Zhang, Tonmoy Hossain, Genuity Systems Limited, Computer Science, Electrical, Computer Engineering, School of Engineering, Applied Science, University of Virginia, [email protected], {mz8rr,tonmoy}@virginia.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-04015.txt.

Introduction

To edit an image in a controlled way, say rotating only the object, a model must know which parts of its internal representation a transformation touches. Farhana Hossain Swarnali, Miaomiao Zhang, and Tonmoy Hossain automate that discovery in a 2025 computer vision preprint. Earlier latent-space methods forced researchers to split the representation into changing and fixed parts by hand.

Their framework learns group actions directly on latent image manifolds. Learnable binary masks, trained with a straight-through estimator, partition the representation into transformation-sensitive and invariant components without manual labels. Disentanglement and the transformation mappings are learned jointly in one optimisation, and the method wraps around any standard encoder-decoder. Tests on five image datasets show the system finding the right factors alone. Downstream classification confirmed the representations stay useful, and code is publicly available.

Catalog record

CREATORFarhana Hossain Swarnali, Miaomiao Zhang, Tonmoy Hossain, Genuity Systems Limited, Computer Science, Electrical, Computer Engineering, School of Engineering, Applied Science, University of Virginia, [email protected], {mz8rr,tonmoy}@virginia.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-04015

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Group Actions In Disentangled Latent Image Representations Farhana Hossain Swarnali, Miaomiao Zhang, Tonmoy Hossain, Genuity Systems Limited, Computer Science, Electrical, Computer Engineering, School of Engineering, Applied Science, University of Virginia, [email protected], {mz8rr,tonmoy}@virginia.edu Abstract Modeling group actions on latent representations enables controllable transformations of high-dimensional image data. Prior works applying group-theoretic priors or modeling transformations typically operate in the high-dimensional data space, where group actions apply uniformly across the entire input, making it difficult to disentangle the subspace that varies under transformations. While latent-space methods offer greater flexibility, they still require manual partitioning of latent variables into equivariant and invariant subspaces, limiting the ability to robustly learn and operate group actions within the representation space. To address this, we introduce a novel end-to-end framework that for the first time learns group actions on latent image manifolds, automatically discovering transformation-relevant structures without manual intervention. Our method uses learnable binary masks with straight-through estimation to dynamically partition latent representations into transformation-sensitive and invariant components. We formulate this within a unified optimization framework that jointly learns latent disentanglement and group transformation mappings. The framework can be seamlessly integrated with any standard encoder-decoder architecture. We validate our approach on five 2D/3D image datasets, demonstrating its ability to automatically learn disentangled latent factors for group actions in diverse data, while downstream classification tasks confirm the effectiveness of the learned representations. Our code is publicly available at GitHub . 1 Introduction Understanding and modeling the symmetries underlying high-dimensional image data is fundamental to learning robust, interpretable, and controllable feature representations [ 18 , 17 ] . Group actions provide a principled mathematical framework for capturing such symmetries within a given image distribution [ 26 , 19 , 21 ] . Recent developments have focused on incorporating group-theoretic inductive priors into deep networks, including Group-Equivariant CNNs, LieConv, and SE-Transformers [ 7 , 11 , 12 ] , to build equivariant representations under various transformation groups. Other approaches aim to learn group actions directly by modeling transformations between data points, enabling compositional generalization across factor combinations [ 39 , 19 ] . While effective, these methods primarily operate in high-dimensional data space, where applying group actions uniformly to the entire input leads to high computational cost, reduced interpretability, and susceptibility to irrelevant variations [ 36 ] . Besides, such models struggle to handle scenarios where transformations are semantically localized, where only a subset of the scene is expected to change while the rest remains fixed. For example, in an image of a cat rotating around a stationary ball, the transformation should apply only to the cat, not the background. Capturing such behavior requires learning group actions on latent representations that can disentangle and isolate transformation-relevant factors from invariant content, enabling finer control and more scalable generalization. Learning group actions in the latent space offers a semantically meaningful alternative to modeling them directly in the data domain to capture transformation-varying representations [ 42 , 20 , 16 ] . Building upon encoder-decoder frameworks, recent approaches have demonstrated that group actions can be learned over latent representations without requiring group-specific architectural layers or constraints, achieving diverse transformation results [ 21 ] . Nevertheless, it primarily depends on manually subdividing the latent features into transformation-sensitive versus invariant subspaces, demanding prior domain knowledge about the expected structure of latent representations that is rarely available in practice [ 21 ] . This approach falls apart in scenarios where transformation-relevant factors are unknown or entangled with invariant content. For instance, modeling car rotation assumes that latent variables for orientation are pre-identified, which rarely holds without supervision. This limitation motivates the need for a method that can automatically identify and disentangle transformation-sensitive components during training without relying on manual partitioning of the latent space, specifically in the latent space of image representations. To this end, we propose a novel end-to-end framework that automatically learns a transformation-relevant latent subspace for group actions, eliminating the manual partitioning requirement that limits existing approaches. Different from existing disentangled learning methods such as \beta -VAE and Factor-VAE [ 14 , 22 ] that focus on separating general factors of variation in latent space, our approach specifically targets the learning of latent components that should be equivariant versus invariant under a wide variety of group actions. Our method simultaneously learns the disentanglement of latent subspace through learnable partitioning and the corresponding group action mappings via joint optimization, enabling automatic discovery of transformation-relevant representations without architectural constraints. While our approach generalizes to a wide variety of transformation groups, we focus on the rotation group for empirical validation, demonstrating improved performance in both reconstruction and downstream classification tasks. We summarize our four primary contributions as follows: • Develop an end-to-end framework that automatically identifies transformation-sensitive and invariant latent representations through learnable partitioning. • Design a joint optimization approach that simultaneously learns latent disentanglement and group action mappings. • Demonstrate the generalizability of our approach across both synthetic and real-world datasets. • Validate the effectiveness of our automatically learned representations through comprehensive evaluation, including downstream classification tasks. Our proposed framework can be efficiently incorporated into any encoder-decoder architecture to learn group actions within the latent representation space. To validate our approach, we conduct experiments across five diverse 2D/3D image datasets, covering both synthetic and real-world scenarios, including high-dimensional volumetric medical images. Experimental results demonstrate that our method effectively learns transformation-specific latent partitions, achieving improved transformation control and representation quality across a broad range of visual domains. 2 Related Works Group-Equivariant Networks: Group-equivariant neural networks incorporate symmetry constraints directly into network architectures through specialized layers and convolution operations. These approaches leverage group representation theory to build equivariant representations under various transformation groups [ 7 , 33 , 30 , 37 , 38 , 34 , 29 ] , including steerable CNNs for continuous groups [ 6 ] , SE(3)-transformers for 3D transformations [ 12 ] , and quantum equivariant networks [ 28 ] . While effective, these methods require group-specific architectural constraints and mostly operate in high-dimensional data space. Group Action on Data vs. Latent Space: Traditional approaches learn group actions directly on high-dimensional data through specialized convolution operations and data augmentation strategies [ 32 , 13 ] . However, applying transformations uniformly to entire inputs leads to computational inefficiency and limits the ability to handle localized transformations [ 39 , 36 , 19 ] . To address these limitations, recent work has shifted toward learning group actions in latent space, enabling more flexible transformation modeling [ 9 , 42 , 21 ] . Despite these advances, existing latent space approaches either require manual specification of transformation-sensitive versus invariant latent components or suffer from poor learned group action quality due to inefficient group action operations in the latent space, both of which limit their practical applicability, that further requires real-world domain knowledge. Disentangled Representation Learning: Another line of work addresses disentangled representation learning, seeking to decompose latent representations into independent factors of variation [ 4 , 23 , 5 ] . Central to this approach, \beta -VAE and its variants enforce disentanglement through regularization that balances reconstruction fidelity against latent independence [ 14 , 2 , 40 , 23 , 4 ] . Subsequent developments include Factor-VAE’s treatment of total correlation [ 22 ] ,

Read the complete text