Constrained latent state modeling: A unifying perspective on representation learning under competing constraints

Gwenolé Quellec · 2026

Read Constrained latent state modeling: A unifying perspective on representation learning under competing constraints by Gwenolé Quellec (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15995.txt.

Introduction

Gwenolé Quellec proposes Constrained Latent State Modeling. It is a framework for judging what a learned hidden state should keep, drop, and sort. He defines six traits: predictive sufficiency, minimality, temporal coherence, observation fit, invariance to nuisance factors, and structural limits. He separates these traits from the training goals and tests used to impose or check them. He rereads major representation groups through this lens. A synthetic benchmark then shows how distinct training goals yield distinct hidden layouts and trade-offs.

Catalog record

CREATORGwenolé Quellec
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72421
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15995

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Constrained latent state modeling: A unifying perspective on representation learning under competing constraints Gwenolé Quellec Abstract Learning latent representations from complex data is central to modern machine learning, spanning temporal, multimodal, and partially observed systems. In such settings, representations are better understood as latent states capturing underlying system dynamics, rather than as mere compressed summaries of observations. Yet current approaches remain fragmented, relying on distinct—and often implicit—assumptions about what these states should represent. We argue that this fragmentation reflects a more fundamental limitation: latent representations are typically learned from underconstrained objectives that fail to specify the properties that meaningful latent states should satisfy. As a result, multiple representations can satisfy the same objective, leading to ambiguity in their structure and interpretation. While many of the underlying principles have been explored in isolation, their interactions have not been explicitly formalized. In this work, we propose constrained latent state modeling (CLSM) as a unifying perspective. We identify a set of core properties—predictive sufficiency, minimality, temporal coherence, observation compatibility, invariance to nuisance factors, and structural constraints—and show that they are intrinsically coupled through fundamental trade-offs. Revisiting major modeling families through this lens, we show that existing approaches can be interpreted as enforcing different subsets of constraints, thereby occupying distinct regions of a common design space. This perspective reframes persistent challenges such as lack of identifiability as consequences of underconstrained formulations, rather than isolated technical limitations. More broadly, CLSM provides a principled framework to make design choices explicit, to analyze trade-offs, and to guide the development of more interpretable, robust, and task-aligned latent state models. keywords Representation learning; Latent state modeling; Identifiability; Multi-objective optimization; Self-supervised learning 1 Introduction Learning latent representations from complex, high-dimensional data is a central problem in modern machine learning, with applications spanning temporal, multimodal, and partially observed systems. In many such settings, latent representations are more naturally interpreted as latent states , capturing the underlying system dynamics rather than merely compressing observations. This perspective is particularly prominent in sequential modeling, control, and modern representation learning paradigms, including world models rao_predictive_1999 ; hafner_learning_2019 ; ha_world_2018 . In this work, we focus on settings where such an interpretation is meaningful, and restrict our analysis to this regime. Although many examples in this work are drawn from clinical and biomedical applications, the proposed framework is general and applies broadly to latent state modeling across domains. Despite substantial progress, the literature on latent representation learning remains fragmented. Different modeling frameworks—ranging from probabilistic latent variable models kingma_auto-encoding_2014 ; rezende_stochastic_2014 and autoencoder-based approaches hinton_reducing_2006 ; vincent_extracting_2008 to predictive self-supervised methods oord_representation_2019 ; assran_self-supervised_2023 and multimodal learning architectures ngiam_multimodal_2011 ; radford_learning_2021 —emphasize distinct and often implicit assumptions about what a latent state should represent. As a result, models that perform well empirically may differ substantially in the properties they enforce, making it difficult to compare approaches or to interpret the representations they produce baltrusaitis_multimodal_2019 ; uelwer_survey_2025 . A central difficulty is that latent states are expected to satisfy multiple, complementary requirements. They should retain information relevant for prediction, provide compact summaries of past observations, evolve coherently over time, remain compatible with observed data, admit interpretable structure, and exhibit invariance to nuisance factors. This latter aspect is particularly critical in real-world applications, especially in the medical domain, where domain shifts arising from acquisition conditions, populations, or annotation variability can substantially affect model performance matta_systematic_2024 . These requirements are not independent: they define interacting design principles that jointly characterize desirable latent state representations. Crucially, these principles are intrinsically coupled through fundamental trade-offs. For example, enforcing predictive sufficiency may require retaining detailed information from the past, while minimality encourages compression, as formalized in information-theoretic approaches such as the information bottleneck alemi_deep_2017 . Similarly, promoting invariance to nuisance factors may conflict with sensitivity to subtle but meaningful changes, a tension that underlies many self-supervised and contrastive learning methods tschannen_mutual_2020 ; henaff_data-efficient_2020 . As a result, latent state modeling cannot be reduced to a single objective, but must instead be understood as a multi-objective problem in which competing constraints are explicitly balanced. Existing approaches address these challenges only partially. Reconstruction-based models emphasize observation fidelity, often capturing fine-grained variability without explicitly enforcing predictive relevance alain_what_2014 . Predictive methods, such as contrastive predictive coding, prioritize future-relevant information but may discard aspects of the data that are not directly predictive oord_representation_2019 . Multimodal approaches focus on integrating heterogeneous signals, yet often lack explicit control over invariance or structure baltrusaitis_multimodal_2019 . In contrast, domain-specific structured models in fields such as medicine enforce strong priors on latent states, improving interpretability at the cost of flexibility fonteijn_event-based_2012 ; young_uncovering_2018 . This diversity suggests that existing methods occupy different regions of a common design space, each emphasizing specific constraints while leaving others only weakly enforced. A recurring issue in latent variable modeling is the lack of identifiability: different models or training runs can yield representations that are equally valid in terms of the objective, yet fundamentally different in structure and interpretation. This phenomenon is well documented in the context of disentangled representation learning, where identifiability is shown to be fundamentally impossible without additional inductive biases locatello_challenging_2019 . More generally, optimizing objectives based on mutual information or reconstruction alone does not uniquely determine the structure of latent representations tschannen_mutual_2020 . We argue that this lack of identifiability reflects a more fundamental limitation: current approaches do not explicitly specify the constraints that define meaningful latent states. As a result, multiple representations can satisfy the same objective, revealing an underconstrained problem in which the desired properties of the latent state are only implicitly defined. This suggests that progress in latent modeling may be limited not by model capacity, but by the absence of a clear and explicit formulation of these constraints. In this work, we propose a unifying perspective based on constrained latent state modeling (CLSM) (pronounced “clasm”). We identify a set of core design principles for latent state representations and analyze the trade-offs that connect them. We then revisit major families of models through this lens, showing how each approach emphasizes particular constraints. Building on this analysis, we introduce a formulation in which latent state modeling is expressed as a constrained or multi-objective optimization problem, making these trade-offs explicit and controllable. CLSM does not introduce fundamentally new constraints, but provides a unified view that makes their interactions explicit and controllable across modeling frameworks. Rather than seeking a universally optimal representation, this perspective offers a principled way to navigate the design space of latent states, providing both a structured interpretation of existing methods and a foundation for the systematic development of new approaches. 2 Design Principles for Latent State Representations 2.1 From representations to states Most representation learning approaches aim at constructing latent variables that are useful for a downstream task, often defined implicitly through an objective function. However, in many applications involving temporal, multimodal, or partially observed processes, latent variables are more naturally interpreted as states , capturing the underlying system rather than merely compressing observations. This shift implies stronger requirements: a latent state should not only summarize observations, but do so in a way that is dynamically consistent, informative, and robust. In this section, we identify a set of core principles that characterize desirable latent state representations. These principles can

Read the complete text