Disentangled Representation Learning via Modular Compositional Bias

Whie Jung Dong Hoon Lee Seunghoon Hong · 2025

Read Disentangled Representation Learning via Modular Compositional Bias by Whie Jung Dong Hoon Lee Seunghoon Hong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21402.txt.

Introduction

Whie Jung, Dong Hoon Lee, and Seunghoon Hong teach vision models to split a scene into separate factors, such as object identity and style, a task called disentangled representation learning. Older methods hard-wire a special goal or network shape for each factor, so a new kind of factor forces a costly redesign.

Their fix is a modular bias that lives outside both the goals and the network. The key observation is that factors recombine by different rules: a face has one nose, so global traits exclude each other, while objects can appear in any combination. They shuffle the model's inner codes by factor-specific rules and train it so every shuffle still decodes into a believable image that matches its code. Changing only the shuffling rules separates traits, objects, or both at once.

Catalog record

CREATORWhie Jung Dong Hoon Lee Seunghoon Hong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21402

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Disentangled Representation Learning via Modular Compositional Bias Whie Jung Dong Hoon Lee Seunghoon Hong Abstract Recent disentangled representation learning (DRL) methods heavily rely on factor-specific strategies—either learning objectives for attributes or model architectures for objects—to embed inductive biases. Such divergent approaches result in significant overhead when novel factors of variation do not align with prior assumptions, such as statistical independence or spatial exclusivity, or when multiple factors coexist, as practitioners must redesign architectures or objectives. To address this, we propose a compositional bias, a modular inductive bias decoupled from both objectives and architectures. Our key insight is that different factors obey distinct "recombination rules" in the data distribution: global attributes are mutually exclusive, e.g., a face has one nose, while objects share a common support (any subset of objects can co-exist). We therefore randomly remix latents according to factor-specific rules, i.e., a mixing strategy, and force the encoder to discover whichever factor structure the mixing strategy reflects through two complementary objectives: (i) a prior loss that ensures every remix decodes into a realistic image, and (ii) the compositional consistency loss introduced by Wiedemer et al. 2023 , which aligns each composite image with its corresponding composite latent. Under this general framework, simply adjusting the mixing strategy enables disentanglement of attributes, objects, and even both, without modifying the objectives or architectures. Extensive experiments demonstrate that our method shows competitive performance in both attribute and object disentanglement, and uniquely achieves joint disentanglement of global style and objects. Code is available at https://github.com/whieya/Compositional-DRL . 1 Introduction Understanding the underlying structure of data has become increasingly important for building robust and interpretable machine learning models. A key approach to addressing this challenge is unsupervised disentangled representation learning (DRL) [ 1 , 16 ] , which aims to factorize data into its fundamental compositional concept representations. By interpreting the world through compositional concepts, it becomes feasible to decompose unseen data into simpler, more interpretable components. Moreover, this approach dramatically improves the data efficiency of learning [ 26 , 24 ] , as unseen data can be explained as combinations of already learned concepts. As a key theoretical insight in DRL, Locatello et al. 2019 show that disentangled representations aligned with underlying ground-truth factors of variation cannot be reliably achieved without appropriate inductive biases or explicit supervision. Consequently, designing a method to embed suitable inductive biases, which impose constraints on the model or assumptions about the data, has become a central challenge in DRL. Specifically, attribute disentanglement methods often enforce information-theoretic objectives [ 29 , 52 ] , while object disentanglement methods leverage spatial exclusivity through architectural components, such as slot attention encoders [ 31 , 41 , 19 ] , or additive decoders [ 25 , 2 ] . Despite impressive progress in each domain, current approaches are constrained by divergent factor-specific strategies for embedding inductive biases, either through learning objectives or architectural components. These vastly different, factor-specific designs embedded within architectures or objective functions require substantial engineering when encountering a novel factor of variation that may violate prior assumptions, such as statistical independence or spatial exclusivity, or when different factors co-occur, as practitioners must redesign architectures or loss functions. Identifying suitable objective functions or architectural components for new factors is challenging, given the vast design space. This motivates the need for a general framework that simplifies inductive biases into a modular component, applicable to various disentanglement factors. As a first step toward this goal, we propose a compositional bias , a modular inductive bias decoupled from both learning objectives and architectures, enabling disentanglement of different factors within a single framework. In particular, we formulate DRL as a process of maximizing compositionality, where the compositional bias characterizes factor-specific biases by defining valid ways to compose disentangled representations. Given two sets of latent representations from different images, we construct a composite representation by exchanging random subsets of latents and maximizing the validity of the resulting composite image, measured by data likelihood and compositional consistency [ 50 ] . Analyzing the compositional structures of attributes and objects, we derive specific compositional biases, or mixing strategies , that determine valid compositions for attribute and object disentanglement. Incorporating factor-specific biases into this modular mixing strategy, rather than into the architecture or objective function, allows our method to disentangle objects and attributes under a single framework by simply switching the mixing strategy. Our contributions are as follows: • We propose a disentanglement framework that decouples factor-specific inductive biases from learning objectives and architectures, enabling both attribute and object disentanglement under a single set of objectives and architectures. • We derive mixing strategies as a compositional bias, embedding factor-specific biases in a modular way and propose specific strategies for attribute, object, and joint disentanglement. • We compare our methods against baselines specifically designed for attribute or object disentanglement. For attribute disentanglement, our method achieves the best DCI [ 10 ] , while on object-level tasks, it performs comparably to state-of-the-art methods. Notably, our method demonstrates joint disentanglement of both factors within a single framework. Figure 1: Overview of our method. To derive a compositional bias, we analyze the compositional structure of attribute and object, and implement it as a mixing strategy. Given a mixing strategy, we decode composite representations into an image and minimize our prior loss and consistency loss to ensure it is both realistic and aligned with the latents. Note that the figure illustrates a specific example for object mixing strategy. 2 Background In this section, we review the two main streams of disentangled representation learning: attribute and object 1 1 1 We refer to attribute factors as properties shared globally across an entire scene, e.g. , color, style, while object factors are distinct spatial entities within a scene, such as individual objects. disentanglement. In particular, we explain how current approaches incorporate factor-specific inductive biases into either learning objectives or model architectures, and why this tight coupling limits the ability to generalize these biases across different factors of variation. For a more comprehensive review of the literature, we refer the reader to Appendix A.3 . Attribute disentanglement In attribute disentanglement [ 3 , 6 , 22 , 5 , 37 ] , scenes are assumed to consist of a fixed set of random variables [ 22 ] , with methods typically enforcing statistical independence among latent dimensions through learning objectives to achieve attribute representations. For example, [ 3 , 22 , 5 ] incorporate Total Correlation [ 48 ] into the VAE framework, while [ 29 , 37 ] use contrastive regularization to ensure that variations in each latent lead to distinct changes in the output space. More recently, [ 52 ] proposed minimizing an upper bound on mutual information among latent variables. These information-theoretic objectives work well when the data consist of a fixed set of factors, with each latent variable corresponding to a specific factor. However, incorporating such biases directly into the learning objective is not straightforward in general. For instance, in object-centric scenes with varying numbers of objects and permutation invariance, defining information-theoretic objective becomes non-trivial. As a result, practitioners must devise entirely new objectives or models for novel scenarios, such as object-centric scenes. This underscores how embedding the bias directly into the objective function limits

Read the complete text