Disentanglement by means of action-induced representations Gorka Muñoz-Gil, Hendrik Poulsen Nautrup, Arunava Majumder, Paulin de Schoulepnikoff, Florian Fürrutter, Marius Krumm, Hans J. Briegel Abstract Learning interpretable representations with variational autoencoders (VAEs) is a major goal of representation learning. The main challenge lies in obtaining disentangled representations, where each latent dimension corresponds to a distinct generative factor. This difficulty is fundamentally tied to the inability to perform nonlinear independent component analysis. Here, we introduce the framework of action-induced representations (AIRs) which models representations of physical systems given experiments (or actions) that can be performed on them. We show that, in this framework, we can provably disentangle degrees of freedom w.r.t. their action dependence. We further introduce a variational AIR architecture (VAIR) that can extract AIRs and therefore achieve provable disentanglement where standard VAEs fail. Beyond state representation, VAIR also captures the action dependence of the underlying generative factors, directly linking experiments to the degrees of freedom they influence. † † affiliationnotice: Equal contribution Creating compressed representations of high-dimensional observations has become essential across a range of machine learning tasks, from improving computational efficiency in downstream pipelines ( 22 ; 16 ) , to enabling interpretable approaches that aim to uncover structure within the input data. In this context, variational autoencoders (VAEs) ( 14 ) have emerged as a hallmark architecture for unsupervised representation learning. VAEs aim to compress input data into a low-dimensional latent space, creating a minimal representation of it. In cases where a dataset is characterized by few hidden variables, also known as factors of variation, VAEs are expected to extract these by encoding them in their latent space ( 6 ) . In physics, this translates to identifying the key features of a system, typically associated with its effective degrees of freedom ( 9 ; 11 ; 19 ; 26 ; 18 ) or order parameters ( 27 ; 4 ) . However, several challenges remain. One that has attracted significant attention, due to its foundational importance, is the disentangled nature of the learned representations ( 17 ) . This refers to a representation in which each neuron in the VAE’s latent space encodes a single factor of variation, in contrast to an entangled representation, where the factors are mixed within the latent neurons. Indeed, such a problem is closely related to independent component analysis (ICA) ( 8 ) , which aims to recover the underlying components from observed data. This connection has revealed that, under typical conditions where the mapping from factors of variation to observations is nonlinear, recovering the true generative factors becomes fundamentally ill-posed, due to the undecidable nature of nonlinear ICA ( 12 ) . From a physics standpoint, this implies that the VAE may not uncover the system’s actual degrees of freedom, but instead learn entangled combinations of them, thus hindering the effective interpretability of the model. Indeed, substantial efforts have been made to address the issue, and numerous VAE variants have been proposed to mitigate it ( 3 ; 7 ; 13 ) . From a broader perspective, VAEs and their more advanced variants are typically trained on a fixed dataset of observations. This is analogous to analyzing a collection of experimental measurements. However, in scientific practice, the process by which data is acquired is often as important as the data itself. In experimental settings, one commonly probes a system by applying a perturbation, or action, and observing the system’s response. The nature of these actions thus encodes essential information that can provide deeper insight into the physical system under investigation. For example, certain actions may selectively couple to specific degrees of freedom, such as heating an object affecting its temperature, but not its mass. Interestingly, this perspective aligns with concepts from reinforcement learning (RL) ( 24 ) , where an agent interacts with an environment by performing actions to achieve specific goals. When combined with representation learning techniques, such frameworks can be used to uncover factors of variation of the environment from observed interactions ( 25 ) , or to understand how different actions relate through their effects on the system ( 10 ) . Closely related is the field of causal representation learning ( 23 ) , which seeks to recover hidden causal variables through interventions. In such settings, actions are typically assumed to modify the hidden causal variables (or its value), and algorithms are designed to infer these transformations ( 15 ) . In contrast, physical experiments often operate under a different paradigm: the intrinsic properties of the system, i.e., its hidden variables (or degrees of freedom), remain fixed. Actions or experimental interventions do not alter these properties, but instead yield different observations that reflect them (see Figure 1 a). In this work, we show that incorporating actions is key for learning disentangled representations, and we introduce a framework specifically designed to extract physical concepts in experimental settings. To this end, we develop a novel VAE variant, drawing inspiration from recent advances in representation learning within reinforcement learning and causal inference. The model is trained on pairs consisting of an action applied to the system and the resulting observation or measurement. The proposed architecture is built with two main objectives: first, to enforce the emergence of disentangled representations of the observed data; and second, to explicitly link each action to the specific degrees of freedom it affects. We begin by presenting in Section 1 the theoretical analysis of the framework. We describe how and why disentanglement arises, and identify the conditions under which a set of actions induces a disentangled latent space. We then present in Section 2 a VAE architecture capable of producing such disentangled representations. Last, in Section 3 we numerically demonstrate the model’s capabilities in three experiments: an abstract example designed to benchmark the capabilities of the model and show its advantage in disentanglement w.r.t. state-of-the-art VAE approaches, and two physics-inspired scenarios: one involving the recovery of the degrees of freedom of a classical particle from its trajectory, and another concerning the representation of small quantum systems from tomographic data. 1 Theory of action-induced representations Figure 1 : a) Schematic representation of the problem considered. A physical system, uniquely described by its factors of variation \mathbf{c} , is observed in a certain state x . Some elementary actions a_{i}\in\mathbb{A} are applied to the system, leading to outcomes y_{i}(\mathbf{c}) that may only depend on a subset of \mathbf{c} . b) Diagrammatic overview of the theoretical representation of the problem. See main text for details. c) Top scheme represents the factors of variation \mathbf{c} of a), together with the subsets I_{A} corresponding to each action {a_{i}} . Because c_{2} is required for both actions, it is in the intersection I_{a_{1}}\cap I_{a_{2}} and by Theorem 1 will be disentangled in any minAIR (yellow color). The two lower schemes represent the same picture, here for Example 3 (main text), and for two different action combination sets P_{\mathbb{A}}=\{\{1\},\{2\},\{1,2\}\} and P_{\mathbb{A}}=\{\{1\},\{1,2\}\} , respectively. d) VAIR: two encoders ( E_{X} and E_{A} ) compute the latent neurons’ mean and variance from an input sample x and action combination A (or elementary action a ), respectively. A latent sample z and the A are then fed into the decoder D to compute the output y_{A} . We consider observations of a physical system x , assumed to lie in a continuous input data manifold X\subseteq\mathbb{R}^{d_{x}} and sampled from a probability distribution p_{\text{data}} . We further consider that the input manifold X has a much lower dimension than d_{x} , a widely-accepted tenet in computer science referred to as the Manifold Hypothesis ( 28 ) . Its coordinates are typically referred to as factors of variation c\in C and uniquely describe any given data point x\in X . The goal of our proposal is to obtain a disentangled representation of these factors of variation without relying on independent component analysis. To do so, we take an operational approach and consider a finite set of integers \mathbb{A}=\{1,2,\dots,n_{A}\} which label actions a (also referred to as experiments or tasks in the following). Some of these actions can be implemented together, giving us a subset P_{\mathbb{A}}\subseteq\mathcal{P}(\mathbb{A}) specifying allowed actions and action combinations, where \mathcal{P}(\mathbb{A}) is the power set of \mathbb{A} . For each allowed element A\in P_{\mathbb{A}} and for each input data point x , we consider that there is a unique, deterministic ground truth y_{A}\in Y_{A}\subseteq\mathbb{R}^{d_{Y}} (see Figure 1 a). Here, Y_{A} is assumed to be a continuous manifold embedded in a raw output data space \mathbb{R}^{d_{Y}} , similarly as for the input data. Example 1 . Consider an experiment a=1 measuring the gravitational force acting on a resting object, i.e. y_{1}:=F_{G}=mg with g the constant gravitational acceleration, which only depends on the mass m . An experiment a=2 measures the force exerted by a fixed, homogeneous electric field E , given by y_{2}:=F_{E}=qE and therefore only depends on the charge q . Moreover, elementary actions a=1,2 can be combined into one experiment A=\{1,2\} , measuring first the gravitational force and then the electrical force. Then, the set of allowed action combinations is P_{\mathbb{A}}=\{\{1\},\{2\},\{1,2\}\} . Note that the form of the data Y_{A} is not relevant as long as it is a continuous manifold. For instance, we could have also used the object’s trajectory in the respective fields as the outcomes y_{i} . An implementation of a very similar scenario is presented in Section 3.2 . This setting gives rise to a collection of datasets as follows: Definition 1 . Let X\subseteq\mathbb{R}^{d_{x}} be the continuous manifold of the set of input data points , and \mathbb{A}=\{1,2,\dots,n_{\mathbb{A}}\} the set of possible elementary actions . Consider a set P_{\mathbb{A}}\subseteq\mathcal{P}(\mathbb{A}) specifying the allowed action combinations . For each allowed action combination A\in P_{\mathbb{A}} , let there be a deterministic ground truth y_{A}:X\rightarrow Y_{A}\subset\mathbb{R}^{d_{Y_{A}}},x\mapsto y_{A}(x) such that Y_{A} is also a continuous manifold. We call D_{A}:=\{(x,y_{A}(x))\ |\ \forall x\in X\} an action-induced dataset for A\in P_{\mathbb{A}} . The notation for this section is summarized in Figure 1 b. From the following, we see that an action-induced dataset has some inherent structure: in Example 1 , to reproduce the data y_{1}(x) , we do not need to know the charge q , and we do not need to know m to reproduce y_{2}(x) . This motivates the concept of action-induced representations, which may filter unused parameters for a given element of P_{\mathbb{A}} : Definition 2 . Let D_{A}\subset X\times y_{A}(X) be an action-induced dataset for A\in P_{\mathbb{A}}\subseteq\mathcal{P}(\mathbb{A}) where P_{\mathbb{A}} is the set of allowed action combinations and \mathbb{A}=\{1,2,...,n_{\mathbb{A}}\} is the set of elementary actions. Consider a continuous manifold Z\subset\mathbb{R}^{d_{Z}} , referred to as a latent space . Let I_{A}\subseteq\{1,...,d_{Z}\} with A\in P_{\mathbb{A}} denote a subset of indices in \mathbb{R}^{d_{Z}} . Given such an index set, we introduce the projector \pi_{A}:Z\rightarrow Z_{A}\subseteq\mathbb{R}^{|I_{A}|} , \displaystyle\pi_{A}(z)\equiv z_{A}:=(z_{i})_{i\in I_{A}} (1) where Z_{A}:=\{(z_{i})_{i\in I_{A}}|z\in Z\} . Let us assume that there exist continuous maps \psi:X\rightarrow Z and \phi_{A}:Z_{A}\rightarrow Y_{A} for all A\in P_{\mathbb{A}} such that, \displaystyle(\phi_{A}\circ\pi_{A}\circ\psi)(x)=y_{A}(x)\>\>\forall x\in X. (2) Then, we refer to \left(Z,\psi,(I_{A},\phi_{A})_{A\in P_{\mathbb{A}}}\right) as an action-induced representation (AIR) of \{D_{A}\}_{A\in P_{\mathbb{A}}} . Figure 1 b, lower row, illustrates a mapping \psi(x) that transforms an input x into a latent representation z (shown in blue). Given an action combination A\in P_{\mathbb{A}} , the corresponding AIR z_{A} (shown in yellow) is obtained by applying the projection \pi_{A}^{(z)} to z . As we will further elaborate in Section 2 , AIRs fit naturally into the framework of VAEs, where \psi represents an encoder, Z the image of the encoder, i.e., its latent space, and \phi_{A} represents a decoder. As we have motivated above for the Example 1 , AIRs can give rise to an operational form of disentanglement. However, AIRs themselves do not guarantee it, so one needs to further restrict to what we define as minimal AIR: Definition 3 . An AIR \left(Z,\psi,(I_{A},\phi_{A})_{A\in P_{\mathbb{A}}}\right) is a minimal AIR (minAIR) if the following conditions are satisfied: 1. \psi:X\rightarrow Z is surjective and d_{Z}=|\bigcup_{A\in P_{\mathbb{A}}}I_{A}| . All \phi_{A}:Z_{A}\rightarrow Y_{A} are invertible and the inverses are continuous. 2. Z is open in \mathbb{R}^{d_{Z}} , and uses the (subset) topology of \mathbb{R}^{d_{Z}} In simple terms, condition (1) implies that minAIRs are a faithful parameterization of the Y_{A} , and (2) imposes that all z -entries are free parameters (without redundancies). A detailed motivation is discussed in Appendix A . With this definition at hand, we can revisit the Example 1 dataset and discuss some minAIRs: Example 2 . Consider the data from Example 1 . An acceptable minAIR would be z=(m,\,q) , with \phi_{1}^{(z)}(m)=mg , \phi_{2}^{(z)}(q)=qE , \phi^{(z)}_{1,2}(m,q)=(mg,\,qE) . Under the assumption that m>0 , all of these functions are continuous with continuous inverses. Similarly, c=(m^{2},\,2q) is a valid minAIR. A non-example is v=(m,m+q) , which is not a minAIR because Y_{2} is one-dimensional, but Z_{2} is two-dimensional, and therefore \phi_{2} cannot be invertible. However, if one changes the setup and considers a dataset that only includes P_{\mathbb{A}}=\{\{1\},\{1,2\}\} , then, v becomes a minAIR with \phi_{1}^{(v)}=\phi_{1}^{(z)} and \phi^{(v)}_{1,2}(m,m+q)=\phi^{(z)}_{1,2}\circ\varphi(m,m+q) where \varphi(m,m+q)=(m,q) is invertible Operational disentanglement in minAIRs We now show that minAIRs have a very specific structure that explicitly disentangles parameters w.r.t. their action-dependence. This disentanglement is summarized by our main theorem: Theorem 1 . Consider two minAIRs, \left(Z,\psi^{(z)},(I_{A}^{(z)},\phi^{(z)}_{A})_{A\in P_{\mathbb{A}}}\right) with convex set Z and \left(C,\psi^{(c)},(I_{A}^{(c)},\phi^{(c)}_{A})_{A\in P_{\mathbb{A}}}\right) with convex set C . For any nonempty set of action combinations \mathcal{A}:=\{A_{1},...,A_{m}\}\subseteq P_{\mathbb{A}} , let I_{\mathcal{A}}^{(z)}=\bigcap_{A\in\mathcal{A}}I^{(z)}_{A} (and analogously I^{(c)}_{\mathcal{A}} ) be the index sets of latent vector components they have in common. Then, for any A_{j}\in\mathcal{A} , the map g_{\mathcal{A}}:Z_{A_{j}}\rightarrow C_{\mathcal{A}} , \displaystyle g_{\mathcal{A}}:=\pi^{(c)}_{\mathcal{A}}\circ(\phi^{(c)}_{A_{j}})^{-1}\circ\phi^{(z)}_{A_{j}} (3) only depends on the overlap z_{\mathcal{A}} , but not on j and not on the completion to z_{A_{j}} . It is bijective as a function of z_{\mathcal{A}}\in Z_{\mathcal{A}} only. The proof of the Theorem and a more general version that does not assume full convexity of Z and C is given in Appendix B . This Theorem implies that the neurons that are shared for different action combinations (those in I_{\mathcal{A}} ) will be disentangled from the rest for any minAIR (see Figure 1 c). In other words, it shows that the z_{\mathcal{A}}\in Z_{\mathcal{A}} and c_{\mathcal{A}}\in C_{\mathcal{A}} uniquely determine each other via g_{\mathcal{A}} through their mutual connection to y_{A}\ \forall A\in P_{\mathbb{A}} (see Figure 1 b). Example 3 . Consider the dataset from Example 1 and the minAIRs in Example 2 . Notably, for P_{\mathbb{A}}=\{\{1\},\{2\},\{1,2\}\} all minAIRs disentangle m and q ( Figure 1 c, middle scheme), while for P_{\mathbb{A}}=\{\{1\},\{1,2\}\} , they only disentangle m , since v is also a minAIR ( Figure 1 c, bottom scheme). The reduced disentanglement in the latter case is exactly a consequence of m being shared by both action combinations A_{1}=\{1\} and A_{1,2}=\{1,2\} while q alone is in no intersection. 2 VAIR: a model for minAIR Given the problem stated above, we introduce an architecture, called variational AIR (VAIR), designed to extract such minAIR and hence, for suitable sets of actions, disentangle the hidden factors of variation of the input data. Background on VAE. The proposed model is based on the variational autoencoder (VAE) ( 14 ) , which consists of an encoder p_{\theta} that compresses input samples x into a latent representation z , and a decoder q_{\phi} that reconstructs the original input from this latent space. In VAEs, latent variables are probabilistic neurons whose distribution is typically modeled by a multivariate Gaussian. In practice, the encoder outputs the mean \mu_{i} and variance \sigma_{i} of the latent variables from which the values for each latent neuron z_{i} are sampled and then passed to the decoder. The model is trained via the Evidence Lower Bound (ELBO), \mathcal{L}=\frac{1}{N}\sum_{n=1}^{N}\mathbb{E}_{q}[\log p_{\theta}(x^{(n)}\mid z)]-\beta\mathrm{KL}[q_{\phi}(z\mid x^{(n)})||p(z)] (4) with N being the number of samples x^{(n)} in the training set and \beta the regularization parameter introduced in 6 . The loss function introduced in Equation 4 contains two terms. The first is a reconstruction loss, which enforces similarity between the predicted and target outputs. In standard VAEs, this corresponds to the likelihood p_{\theta}(x^{(n)}\mid z) of reconstructing the input x^{(n)} from the latent representation. The second term is a regularization loss, given by the Kullback-Leibler divergence between the approximate posterior q_{\phi}(z_{i}\mid x^{(n)})\sim\mathcal{N}(\mu_{i},\sigma_{i}) and a standard normal prior p(z)=\mathcal{N}(0,1) . This term encourages the latent representation to align with the prior, while competing with the reconstruction objective. This trade-off results in a polarized latent space ( 21 ) , where only the minimal set of neurons required for reconstruction deviates from the prior ( active neurons), while the remaining ones collapse to it ( passive neurons). Additionally, an intermediate regime of mixed neurons, active only for subsets of the data, has been identified ( 2 ) . VAIR: adapting VAE for AIR. VAIR is a variant of VAE purposely built to produce minAIR and hence produce disentangled representations by Theorem 1 . The main change compared to standard VAE is the use of two separate encoders (see Figure 1 d). The first, E_{X} , takes the observations x and outputs only the mean \mu_{i} of the latent neurons. The second, E_{A} , takes the action combination A\in P_{\mathbb{A}} and outputs their variances \sigma_{i}^{2} , which determine which neurons are noised out for a given action. For brevity, we often use elementary actions a\in\mathbb{A} and action combinations A\subseteq P_{\mathbb{A}} interchangeably. Following the common definitions of the polarized regime, we call neurons active if \sigma_{i}^{2}\rightarrow 0 for all actions. As defined in Definition 2 , for some AIRs not all latent neurons are required for every action a , and the regularization in Equation 4 pushes the model to noise out the unnecessary ones. Neurons that are active only for a subset of actions are referred to as mixed neurons. Finally, we feed the latent vector to the decoder D . Moreover, because the action cannot be inferred from the sampled latent vector, and is indeed needed to reconstruct y_{a} , we also input it to the D . We note here that compared to the latent model of Equation 4 , VAIR defines a likelihood p_{\theta}(x^{(n)}|z,a^{(n)}) and a latent posterior q_{\phi}(z|x^{(n)},a^{(n)}) , which can be nonetheless optimized in the same fashion. Interestingly, related conditioning ideas appear in iVAE ( 12 ) . There, identifiability relies on a context-dependent prior p(z|a) , while our approach instead uses action-dependent encoder and decoder components to induce disentanglement, without modifying the prior. Given the previous discussion, we identify E_{X}(x) with the mapping \psi ( Section 1 ), which transforms the input x into a compressed latent representation z . Similarly, the encoder E_{A}(a) corresponds to \pi_{\{a\}} , adapting the latent space to the specific input action. Finally, the decoder D(z,a) plays the role of \phi_{A=\{a\}} , transforming the resulting latent vector into the output associated with that action. While this comparison is heuristic, we provide numerical evidence for the emergence of minAIR in VAIR in Section 3 . We note that, while Section 1 considered discrete actions, numerical experiments presented below demonstrate that minAIRs can also be achieved for continuous actions (see Section 3.3 ). Figure 2 : Abstract experiment a) Factors of variation in the experiment, indicating the subsets I_{a} associated with each of the considered actions. b) Latent neuron variances \sigma_{i} output by E_{a} for the two actions in the experiment, plotted as a function of training epoch. c) Latent neuron means \mu_{i} output by E_{x} as a function of the hidden factors c_{i} . Colors correspond to the neurons (as in panel b), while line styles indicate different factors. d) Mutual information gap (MIG) computed for two groups of factors: c_{2} , and all remaining factors excluding c_{2} . Boxplots show results from 20 independent training runs per model, each with random dataset generation and weight’s initialization. Why does VAIR approximate minAIRs? We now give a heuristic interpretation on why VAIR approximates minAIRs. In the polarized regime, latent neurons can be idealized to be either noise-free ( \sigma_{i}^{2}\rightarrow 0 ) or noised-out ( \sigma_{i}^{2}\rightarrow 1 ), depending on the choice of A\in P_{\mathbb{A}} . The noise-free ones correspond to our decoders \phi_{A}\circ\pi_{A} acting on the projected subspace. Since the training prefers to make as many neurons noisy as possible, the training will prefer a latent representation in which as few neurons as possible are noiseless. This facilitates latent representations in which all remaining latent neurons are helpful for some task, i.e. d_{Z}=|\bigcup_{A\in P_{\mathbb{A}}}I_{A}| . Furthermore, the cardinality of the I_{A} should be minimal in the latent space, such that each noise-free latent neuron acts as its own free parameter, giving us openness. For the noise-free neurons, there is effectively no sampling anymore, such that the decoder \phi_{A} should receive a deterministic latent input, which it maps to the unique ground truth y_{A} . Adding the minimality pressure acting on the latent space, it is reasonable to expect that the \phi_{A} tend to be invertible while continuity is built-in in artificial neural networks. 3 Numerical experiments Code for reproducing the results of this section can be found at https://github.com/gorkamunoz/air . 3.1 Abstract experiment We start by considering an abstract problem that will allow us to illustrate the various theoretical concepts presented in Section 1 . For that purpose, we define a simple dataset arising from four factors of variation c_{1},\dots,c_{4} . An observation x\in\mathbb{R}^{d_{x}} is then given by some fixed random functions F (see Section D.1 ). The outputs y\in\mathbb{R}^{d_{y}} are also calculated from fixed random functions G,G^{\prime} as y_{a_{1}}=G(c_{1},c_{2}) and y_{a_{2}}=G^{\prime}(c_{2},c_{3},c_{4}) excluding action combinations. Hence, \dim Z=4 , \dim Z_{a_{1}}=2 and \dim Z_{a_{2}}=3 (see Figure 2 a). We now set d_{x}=d_{y}=10 and train a VAIR with 6 latent neurons (see Appendix E ). In Figure 2 we show the evolution of the latent neurons’ variance predicted by E_{A} for each of the two input actions. Already in the early stages of training, the model learns that only four neurons are required to reconstruct the output, setting \log\sigma_{2}^{2},\log\sigma_{4}^{2}\rightarrow 0 for both actions, hence making them passive neurons. Then, the model encodes information needed for action a_{1} in two neurons, namely 1 and 3, as shown by the variance decrease in the top panel, and in three neurons for action a_{2} . That is, we can see that the model retains four non-passive neurons, from which three (1,5,6) are mixed, as they are only active for either action 1 or 2, and only one is active (3). This is in accordance with expected results for \dim Z=4 , \dim Z_{I_{a_{1}}}=2 and \dim Z_{I_{a_{2}}}=3 . The fact that neuron 3 is an active neuron comes from it being in the intersection I_{a_{1}}\bigcap I_{a_{2}} . By Theorem 1 , and given that c_{2} is at the intersection of the underlying experiments ( Figure 2 a), any minAIR must disentangle such factor of variation. To show this, we plot in Figure 2 c the latent neurons’ means as a function of the different factors’ values. For each line, we fix all c_{j\neq i} , and change the value of c_{i} monotonically in its range. As expected, due to z_{2} and z_{4} being passive, both means are constant at zero. Conversely, the rest show a clear dependence w.r.t. different c_{i} . In particular, \mu_{3} is only dependent on c_{2} , demonstrating that this neuron encodes a disentangled representation. On the other hand, \mu_{1} encodes mainly a representation of c_{1} , but also has a slight dependence on c_{2} . We recall that VAIR is not constructed to create a disentangled representation for factors outside any intersection, and that any other disentanglement mostly depends on the training and the inherent bias of VAEs to disentangle ( 1 ) . Indeed, we see that neurons 5 and 6 encode an entangled representation of the factors needed to reconstruct y_{a_{2}} . Comparison with other VAE variants We compare VAIR to four VAE variants: (i) TC-VAE x , a \beta VAE trained without actions using the Total Correlation (TC) loss ( 3 ) ; (ii) VAE x,a , a \beta VAE receiving both state and action as input; (iii) VAE {}_{D_{a}} , which also feeds the action to the decoder, partially mirroring VAIR’s structure; and (iv) TC-VAE {}_{D_{a}} , identical to the previous but trained with the TC-loss. See Appendix F for details. While VAE x,a must encode the action in the latent space, violating the AIR assumptions, VAE {}_{D_{a}} and TC-VAE {}_{D_{a}} do not, and could in principle approximate minAIRs. We then evaluate the learned representations using the mutual information gap (MIG) between the factors of variation c_{i} and the mean of each latent neuron \mu_{k} ( 3 ) , with larger MIG values indicating more disentangled representations. In Figure 2 d, we plot the MIG scores for two subsets of neurons. The blue bars show the MIG corresponding to c_{2} , the factor expected to be disentangled from the experiment’s construction. For TC-VAE x , as no actions are considered, we average over all c , showcasing the inability of VAE to fully disentangle the dataset ( 17 ) . On the other hand, as predicted by Theorem 1 and observed in Figure 2 c, VAIR achieves near-complete disentanglement of this factor. \text{VAE}_{D_{a}} and \text{TC-VAE}_{D_{a}} also shows improved disentanglement of c_{2} compared to the standard VAE, although it still performs significantly worse than VAIR . This is due to their inability to dissociate the action from their encoded representation, which also further hinders the MIG when average over the rest of the variables \left_{j\neq 2} (see Section F.2 for further on this). Similarly, the scores drop substantially even for VAIR , as it is not the objective of the architecture to promote disentanglement of these components. Figure 3 : Classical experiment a) Schematic representation of the experiment: we consider objects of certain mass and charge and two actions: an elastic collision with a larger object with fixed mass ( a_{M} ) or the activation of an electric field via a capacitor ( a_{E} ). We also consider the additional presence of an external magnetic field B of different coupling strengths \alpha_{ext}>0 . b) The central panel shows the learned variances by E_{A} for the two different actions at \alpha_{\text{ext}}=0 . The two lower plots showcase the relationship between each latent neuron’s \mu_{i} , as a function of the input trajectories’ mass and charge. c) Difference between the predicted variances by E_{A} for each action, as a function of the strength \alpha_{\text{ext}} of the external magnetic field B . d) Same as panel b but for \alpha_{\text{ext}}=2 . 3.2 Classical physics experiment Next, we present a realistic experiment based on classical physics and closely related to the running example in Section 1 . We will show the existence of non-trivial minAIRs that differ from our typical physical interpretation of a system and the effect of uncontrolled external sources hindering disentanglement by AIR. We consider an object of mass m and charge q (see Figure 3 a). The experiment consists of two actions: 1) the collision action, referred to as mass action a_{M} , considers the elastic collision of the object with a fixed, chargeless test mass M>m with velocity v_{H} ; 2) the electric action a_{E} consists of the placement of a parallel plate capacitor that generates an electric field E . Instead of action combinations, we will also later consider the presence of an additional noise source in the form of a magnetic field B with varying strengths (see in Section D.2 ). The goal of this experiment, and hence the output of VAIR, is to predict the two dimensional trajectory y_{a_{i}}(t) as the result for a given action. From this setting, one can see that for y_{H}(t) only the mass is relevant while for y_{E}(t) one needs both the mass and charge. For simplicity, we consider the observation to consist of the stacked trajectories resulting from the actions, x=\{y_{H}(t),y_{E}(t)\} . An alternative choice would have been, for example, a previous observation of the object’s dynamics. Crucially, in line with the problem formulation in Section 1 , all factors of variation must be recoverable from such observations. If one were to train on a single trajectory at a time instead, the encoder would be able to infer which action was applied. As a result, VAIR may construct representations that differ from minAIRs . We now train on such a dataset a VAIR with two latent neurons, for simplicity and due to the expected \dim Z=2 . As shown by the final learned variances presented in Figure 3 b, the two neurons are mixed, each remaining active for one action but not the other. While this was expected for the mass action, as a single factor ( m ) is needed for y_{H}(t) , we expected both m and q to be required to predict y_{E}(t) . By inspecting the learned representations (side panels in Figure 3 b), we see that while \mu_{1} encodes m and is independent of q , \mu_{2} encodes a function of both m and q . In particular, it encodes a representation proportional to q/m , the factor defining the trajectory of a massive, charged object in an electric field (see Section D.2 ). Indeed, the learned representation z=(m,q/m) is a minAIR, and the expected c=(m,q) is not, as Z_{I^{(z)}_{a_{E}}}=1