SEDGE: Structural Extrapolated Data Generation Kun Zhang, Jiaqi Sun, Yiqing Li, Ignavier Ng, Namrata Deka, Shaoan Xie Abstract This paper proposes a framework for Structural Extrapolated Data GEneration (SEDGE) based on suitable assumptions on the underlying data generating process. We provide conditions under which data satisfying new specifications can be generated reliably, together with the approximate identifiability of the distribution of such data under certain “conservative” assumptions. On the algorithmic side, we develop practical methods to achieve extrapolated data generation, based on the structure-informed optimization strategy or diffusion posterior sampling, respectively. We verify the extrapolation performance on synthetic data and also consider extrapolated image generation as a real-world scenario to illustrate the validity of the proposed framework. 1 Introduction Generative AI, designed to generate new content in the form of images, text, audio, or videos, has seen major breakthroughs and attracted much attention in recent years. For instance, in the image domain, well-known generative AI tools, including the latent diffusion model ( 21 ) , DALL-E by OpenAI ( 20 ) , Google’s Imagen ( 25 ) , can generate high-quality images in a controllable manner. However, in many real-world problems, we expect generative AI tools to be able to produce novel content and automate creative tasks in a reliable way. That is, they are required not only to interpolate within known regimes, but also to extrapolate beyond the support of available data. For instance, in AI-generated images for creative design, the desired content or visual specifications often lie outside the support of the training data. Similarly, in drug discovery, candidate molecules are optimized toward target properties to address new diseases or previously unseen combinations of diseases, for which no direct training examples exist. In such scenarios, the ability to generate reliable extrapolated data is essential, but as a classical problem in machine learning, especially in deep learning, it still remains a fundamental challenge. Extrapolated data generation differs in theory from standard data augmentation ( 27 ) or interpolation (estimating unknown data points within the range of a given dataset). Interpolation can be achieved, for instance, by making use of suitable smoothness assumptions on the function to be estimated from data; in contrast, extrapolation often requires a deeper understanding of the process behind the given data and applying it to novel scenarios. Otherwise, if one directly uses the learned flexible, deep data-generation model for extrapolation, the value of the input of the model is out of support of the training data, and the output of the model often does not have correctness guarantees. In fact, existing generative models, despite their expressive power, frequently fail when tasked with producing data that lie outside the training distribution ( 10 ; 18 ; 28 ) . Researchers have developed several types of assumptions or constraints to deal with the extrapolated generation issues in machine learning. A typical one is to directly constrain the functional class of the prediction function to be linear (or parametric) or with an additive form. Some other methods exploit compositional properties of generative functions to reduce instability during generation ( 17 ; 29 ; 15 ) . Some work derives extrapolation guarantees from regularities on distribution shifts, for instance, smoothness ( 13 ) . These developments do not provide a general framework for extrapolated data generation given novel specifications when properties of the involved functions are unknown. In this paper, we study extrapolated data generation from a principled structural perspective, without directly imposing rather strong assumptions on the underlying functions. First, we provide an open analysis of two different data-generating processes, represented by graphical models, to formulate the relationship between specifications and features of the data to be generated: modeling specifications as conditioning variables that generate features, as commonly adopted in conditional generation frameworks (as seen in Fig. 1 (a), in which Z_{i} and X_{i} denote specifications and features, respectively), or modeling the process as features generating or satisfying specifications (see Fig. 1 (b)). We show that the latter structure, in which features point to specifications in the graphical representation, offers a more natural explanation for unseen data. We further improve the graphical representation to explain why the novel scenarios do not appear in the training data, which also enables a principled treatment to go beyond such data. Interestingly, this structural view reveals a key source of difficulty in extrapolation: features directly involved in multiple specifications are inherently harder to extrapolate, since each specification imposes additional constraints on the admissible data. In contrast, in scenarios where such shared features are absent, extrapolation becomes substantially more straightforward. The contributions of this work are summarized as follows: 1) We propose a structural view of the extrapolated data generation problem and conduct a systematic investigation to characterize the conditions under which valid extrapolated data can be generated and those that enable the whole distribution of the extrapolated data to be approximately constructed. 2) We develop practical algorithms to optimize for the extrapolated data or sample from the constructed conditional distribution given the novel combinations of specifications. 3) We further demonstrate the effectiveness of the proposed framework on both synthetic extrapolation tasks and extrapolated image generation. 2 Initial Thoughts {X}_{1} {Z}_{1} {X}_{2} {Z}_{2} {X}_{3} (a) Specification generates features. {X}_{1} {Z}_{1} {X}_{2} {Z}_{2} {X}_{3} (b) Features produce specifications. Figure 1 : The two generating processes as initial thoughts. Let us use a simple example to illustrate the setting and our idea. Suppose we are required to generate a sensible image of a peacock eating ice cream, assuming that there is no such image in the training data. For illustrative purposes, let us simply suppose it involves two specfications, “peacock” ( Z_{1}=1 ) and “eating ice cream” ( Z_{2}=1 ). 1 1 1 In practice, this example may involve more specifications, such as “eating” and “ice cream”; in this case, they are merged into a macro specification. As an initial idea, one may adopt the graphical model in Fig. 1 (a), where features \mathbf{X}=(X_{1},X_{2},X_{3}) are generated by specifications \mathbf{Z}=(Z_{1},Z_{2}) . Here, we may think of X_{2} as indicating the presence of a “beak”. Under this model, given a novel combination of \mathbf{Z} values, say, (Z_{1}=1,Z_{2}=1) , which never happened in the training data, unless strong assumptions are enforced on the functions (for instance, a parametrtic model is assumed for p(\mathbf{x}\,|\,z_{1},z_{2})) , the distribution of feature X_{2} , which is adjacent to both Z_{1} and Z_{2} , is undefined, and hence we cannot find the distribution or values of \mathbf{X} to satisfy the novel combination of \mathbf{Z} . In other words, the model does not know how to generate the feature “beak” for the unseen combination “peacock eating ice cream”, since this combination is not present in the training data. With this graphical model, without additional assumptions on the functions mapping \mathbf{Z} to \mathbf{X} , extrapolated data generation for novel specifications is not possible if Z_{i} with novel value combinations are influenced by the same features. In contrast, Fig. 1 (b) seems to provide a more reasonable explanation of the relationship between \mathbf{X} and \mathbf{Z} , where features point to specifications. This structure implies the conditional independence Z_{1}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z_{2}\,|\,X_{2} . For example, upon observing a “beak” ( X_{2}=1 ), inferring whether the object is “eating ice cream” ( Z_{2}=1 ) does not depend on whether it is a “peacock” ( Z_{1}=1 ) or a chicken. Such a conditional independence relationship is not captured by the graphical model in Fig. 1 (a), which instead generally implies conditional dependence between Z_{1} and Z_{2} given X_{2} . However, if one adopts the graphical representation in Fig. 1 (b) for the data-generating process, why is there no such scenario in the training data? How can we go beyond the training data in a principled way? Now we are ready to introduce our framework, which models the different distributions across the training data and the data to be generated, figures out their connections, and allows us to go from training data to the construction of new data following novel combinations of the specifications. 3 SEDGE: Structural Extrapolated Data Generation Let us now demonstrate how the structural properties can be leveraged for data generation under novel specifications. We first formalize the data generating process in Section 3.1 . We then study the case of two specifications in Section 3.2 , which serves to illustrate our key ideas of extrapolated data generation, including how to generate novel samples and recover the novel data distribution. Lastly, we present the theoretical results, together with the involved assumptions, for the general setting with multiple specifications in Section 3.3 . 3.1 Formulation Let \mathbf{X}=(X_{1},\dots,X_{d}) denote the feature variables to be generated, and let \mathbf{Z}=(Z_{1},\dots,Z_{n}) denote the specification variables. Let S be a selection variable indicating whether a sample is contained in the given dataset ( 11 ) . A sample is contained in the given dataset if and only if S=1 ; otherwise, S=0 . That is, the given training data correspond to the distribution p(\mathbf{X}\,|\,S=1) . We assume that the variables \mathbf{X} , \mathbf{Z} , and S form a Bayesian network with respect to a directed acyclic graph (DAG) \mathcal{G} ( 19 ; 12 ) . That is, the Markov property is satisfied: each variable is conditionally independent of its non-descendants in \mathcal{G} given its parents in \mathcal{G} . Consistent with the discussion in Section 2 , we impose the following structural assumptions: (i) there are no edges from specifications \mathbf{Z} to features \mathbf{X} , (ii) there are no edges among the specifications \mathbf{Z} , and (iii) selection variable S has no children, and its parents are restricted to the specifications \mathbf{Z} . These assumptions formalize our idea of feature generating specification and ensure that the specification variables \mathbf{Z} are conditionally independent of one another given the features \mathbf{X} . Given data points observed as samples from distribution p(\mathbf{X}\,|\,S=1) , our goal is to generate samples from p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{z}) , where the combination of the specifications values \mathbf{Z}=\mathbf{z} might be novel, i.e., unseen in the given/training data. Note that if the combination of their values already appears in the training data, then this conditional distribution is already implied by the training data, since we have p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{z})=p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{z},S=1) , implied by the conditional independence between S and \mathbf{X} given \mathbf{Z} according to our structural assumptions discussed above. Our central question is therefore: can we generate new data that satisfy novel combinations of specifications, and, if yes, what conditions make it possible, and under what conditions can the corresponding conditional distributions of \mathbf{X} be constructed? In this section, we assume that both the features \mathbf{X} and the specifications \mathbf{Z} are given. Our framework, however, is general. For instance, the features \mathbf{X} can represent variables from tabular data or latent representations learned from images, while the specifications \mathbf{Z} can represent textual concepts extracted from text. We also assume that the specifications \mathbf{Z} are binary, which is natural in many settings where specifications correspond to different concepts. The framework can be straightforwardly extended to the continuous case. Notations. We use \mathbf{1} to denote an all-ones vector of appropriate dimension. To simplify notation, and without loss of generality, we treat \mathbf{Z}=\mathbf{1} as the novel specification combination to which we aim to extrapolate in this section; the analysis extends straightforwardly to other specification values. We use uppercase letters to denote random variables and lowercase letters to denote their realizations. When the random variables are unambiguous from context, we omit variables in density functions, e.g., writing p(\mathbf{x}) instead of p(\mathbf{X}=\mathbf{x}) . We use p^{\mathcal{D}}(\cdot) to denote the distribution implied by the given/training data, which satisfies S=1 , e.g., p^{\mathcal{D}}(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1})\coloneqq p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1},S=1) . We occasionally abuse notation by treating a vector as the corresponding set of its entries when the meaning is clear from context. We also write [n]\coloneqq\{1,\dots,n\} for integer n and use \propto to denote equality up to a positive multiplicative constant. 3.2 Extrapolation with Two Specifications For illustrative purposes, we start with the case of two specification variables, \mathbf{Z}=(Z_{1},Z_{2}) , to illustrate our key idea of leveraging structural properties to generate data that satisfy novel combinations of specifications. Later in Section 3.3 , we extend this setting to general scenarios with more than two specification variables which can also be continuous. To be more precise about novel specifications, suppose we observe only (Z_{1}=1,Z_{2}=0) and (Z_{1}=0,Z_{2}=1) , and possibly (Z_{1}=0,Z_{2}=0) , in the training data, and aim to generate sensible data \mathbf{X} for (Z_{1}=1,Z_{2}=1) , i.e., p(Z_{1}=1,Z_{2}=1\,|\,S=1)=0 . The novel combination has positive probability (and thus is feasible) in the underlying population, i.e., p(Z_{1}=1,Z_{2}=1)>0 . If each specification value Z_{1}=1 and Z_{2}=1 has never appeared in any form in the given data, i.e., p(Z_{i}=1\,|\,S=1)=0 , it is impossible to generate the corresponding samples, since p(\mathbf{x}\,|\,Z_{i}=1) is completely unknown. It is thus necessary to assume that each specification value has been observed in some combination within the dataset. These conditions are formalized in the following assumption. Assumption 1 (Basic conditions for specifications and selection) . We have \displaystyle P(S=1) \displaystyle>0, \displaystyle P(Z_{1}=1,Z_{2}=1) \displaystyle>0, \displaystyle P(Z_{i}=1\,|\,S=1) \displaystyle>0,\quad i=1,2. (1) {X}_{1} {Z}_{1} {X}_{2} {Z}_{2} {X}_{3} {S} {X}_{4} (a) The generating process where Z_{1} and Z_{2} do not share common features. {X}_{1} {Z}_{1} {X}_{2} {Z}_{2} {X}_{3} {S} (b) The generating process where Z_{1} and Z_{2} share X_{2} as a common parent. Figure 2 : The generating processes where features generate the specification and the observed data satisfy the constraint that the selection variable S=1 . 3.2.1 Identifiable Case: Specifications do not Share Common Features Let us first consider the generating process in Figure 2 (a). in which the specifications Z_{1} and Z_{2} do not share common features. It is worth noting the framework also applies when the features generating the same specification variable, such as X_{1} and X_{2} , which generate Z_{1} , have dependence relations in between. In this case, one can successfully recover the distribution with novel specifications p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) from the given data. Proposition 1 . Suppose that 1 holds and the data generating process follows Figure 2 (a). Then, the novel data distribution p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) is identifiable from the given data. A proof is given in Appendix A.2. The idea is to make use of specific conditional independence relations implied by the structural properties of Figure 2 (a), where the specifications do not share common features, i.e., Z_{1}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z_{2} . In particular, a key property is that, when conditioning on a given specification, the features associated with (i.e., that are parent of) that specification are independent of the other specification. Note that after recovering the novel data distribution p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) , one can generate data points with the novel specifications (Z_{1}=1,Z_{2}=1) , thus achieving extrapolated data generation. 3.2.2 Non-Identifiable Case: Specifications Share Common Features We now consider the scenario in Figure 2 (b), where the specifications Z_{1} and Z_{2} share a common parent X_{2} . This case is more complex than the previous scenario in Figure 2 (a), where specifications do not share common features, because here X_{2} is a direct parent of both Z_{1} and Z_{2} . Non-identifiability of novel data distribution. In contrast to the earlier setting in Figure 2 (a), where specifications do not share common features, we show in the following proposition that the distribution corresponding to p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) is, in general, non-identifiable for the generating process in Figure 2 (b). A full proof is given in Section A.3 . Proposition 2 . Suppose that 1 holds and the data generating process follows Figure 2 (b). Then, the novel data distribution p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) is not identifiable from the given data. Intuitively, this is because p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) involves distributions conditioning on S=0 , which are unknown. Existence of novel data sample. Although the distribution corresponding to novel specifications, p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) , is not identifiable, one may still ask whether it is possible to generate samples that satisfy these specifications. Surprisingly, we show that this is indeed possible under mild conditions. We begin by stating the following condition. Assumption 2 (Common features in separate specifications) . There exists value \tilde{x}_{2} of X_{2} such that p^{\mathcal{D}}(X_{2}=\tilde{x}_{2}\,|\,Z_{i}=1)>0,\quad i=1,2. This assumption requires that there exists at least one value of the common feature X_{2} that has a positive density under each specification individually in the given data. Intuitively, it ensures that the marginal distributions of the specifications overlap in the space of the shared feature, making it possible to generate samples satisfying novel combinations of Z_{1} and Z_{2} . Under this assumption, we obtain the following result, which shows that one can construct data points satisfying the novel specifications from the given data. A full proof deferred to Section A.4 . Proposition 3 . Suppose 1 and 2 hold, and that the data generating process follows Figure 2 (b). Then, there exists a value \tilde{\mathbf{x}} of \mathbf{X} that can be constructed from the given data such that p(\mathbf{X}=\tilde{\mathbf{x}}\,|\,Z_{1}=1,Z_{2}=1)>0. Intuively, it is because \displaystyle p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) \displaystyle= \displaystyle p(X_{2}\,|\,Z_{1}=1,Z_{2}=1) \displaystyle~~~~\cdot p^{\mathcal{D}}(X_{1}\,|\,X_{2},Z_{1}=1)p^{\mathcal{D}}(X_{3}\,|\,X_{2},Z_{2}=1) \displaystyle\propto \displaystyle\frac{p(X_{2}\,|\,Z_{1}=1)p(X_{2}\,|\,Z_{2}=1)}{p(X_{2})} \displaystyle~~~~\cdot p^{\mathcal{D}}(X_{1}\,|\,X_{2},Z_{1}=1)p^{\mathcal{D}}(X_{3}\,|\,X_{2},Z_{2}=1)\ \displaystyle\approx \displaystyle\frac{p^{\mathcal{D}}(X_{2}\,|\,Z_{1}=1)p^{\mathcal{D}}(X_{2}\,|\,Z_{2}=1)}{p^{\mathcal{D}}(X_{2})} \displaystyle~~~~\cdot p^{\mathcal{D}}(X_{1}\,|\,X_{2},Z_{1}=1)p^{\mathcal{D}}(X_{3}\,|\,X_{2},Z_{2}=1), and all the three terms above are positive. The argument is constructive as it provides an explicit procedure for generating data points satisfying novel specification combinations from the observed data, by leveraging the conditional independence relation Z_{1}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z_{2}\,|\,X_{2} implied by the structure. Specifically, given estimates of p^{\mathcal{D}}(X_{3}\,|\,X_{2},Z_{2}=1) , p^{\mathcal{D}}(X_{1}\,|\,X_{2},Z_{1}=1) , and p^{\mathcal{D}}(X_{2}\,|\,Z_{i}=1),i=1,2 , all of which can be learned from the given/training data, one can explicitly construct a data point \tilde{\mathbf{x}} that satisfies the novel specifications. Moreover, if the relevant density functions are further assumed to be continuous, then one can construct a set of data points with nonzero Lebesgue measure satisfying the same inequality in Proposition 3 (or 2 ), since it will hold throughout an open neighborhood of the point \tilde{\mathbf{x}} ( 23 ) . Conservative solutions for non-identifiable distributions. We also propose a way to circumvent the non-identifiability issue with p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) described in Proposition 2 . Let us assume that p(S=0\,|\,Z_{1}=1) and p(S=0) are very small, which means that in the original data generating process, the chance of having novel scenarios, which are not contained in the distribution of the given data, is low. Assumption 3 . We have P(S=0\,|\,Z_{i}=1)\approx 0,\,\,i=1,2\quad\text{and}\quad P(S=0)\approx 0. With the assumption above, the novel data distribution p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) becomes approximately identifiable, in the sense that it is approximately equal to a distribution constructed from certain distribution components implied by the training data, formally stated below. We provide a sketch of the proof below and defer the full proof to Section A.5 . Proposition 4 . Suppose that 1 and 3 hold. Then, the novel data distribution p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) is approximately identifiable from the given data. Sketch of proof. By 3 , one can show \displaystyle p(X_{2}\,|\,Z_{i}=1) \displaystyle\approx p^{\mathcal{D}}(X_{2}\,|\,Z_{i}=1),\quad i=1,2, \displaystyle p(X_{2}) \displaystyle\approx p^{\mathcal{D}}(X_{2}). This yields \displaystyle p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) \displaystyle\approx \displaystyle\frac{p^{\mathcal{D}}(X_{2}\,|\,Z_{1}=1)p^{\mathcal{D}}(X_{2}\,|\,Z_{2}=1)}{p^{\mathcal{D}}(X_{2})}\frac{P(Z_{1}=1)P(Z_{2}=1)}{P(Z_{1}=1,Z_{2}=1)} \displaystyle\quad\cdot p^{\mathcal{D}}(Z_{2}=1\,|\,X_{2})p^{\mathcal{D}}(X_{3}\,|\,X_{2},Z_{2}=1) (2) \displaystyle\propto \displaystyle p^{\mathcal{D}}(X_{1},X_{2}\,|\,Z_{1}=1)p^{\mathcal{D}}(Z_{2}=1\,|\,X_{2}) \displaystyle\quad\cdot p^{\mathcal{D}}(X_{3}\,|\,X_{2},Z_{2}=1). (3) ∎ The proof is constructive and provides a way to recover the distribution p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) under 3 . We call Equation 3 a conservative solution since it does not enforce assumptions on the distribution of the (potential) data that were not in the given dataset. The calculation of p(X_{2}) involves p(X_{2}\,|\,S=0) , which is unknown as S=0 corresponds to (potential) data that were not included in the given dataset. Fortunately, it can be ignored by assuming that such data are actually rare in conservative solutions. 3.3 SEDGE with Multiple Specifications We now extend the results from the previous section to show how structural properties can be exploited for extrapolated data generation in the setting with multiple specifications. Due to page limits, we provide a summary of the theoretical extension here, and readers may refer to the details in Appendix A.6 . We first consider the multi-specification setting in which specifications can be partitioned into disjoint subsets that do not share features. Under the extended basic coverage conditions (Appendix Assumption 4), which are analogous to Assumption 1 in the two-specification setting, we know that every specification subset appears with positive probability under selection. The novel distribution is therefore identifiable by recovering each component separately and recombining them; see Theorem 1 in Appendix A.6 . When specifications share common features, p(\mathbf{X}\mid\mathbf{Z}=\mathbf{1}) is generally not identifiable. Nevertheless, analogous to the two-specification setting, extrapolated data points can still be constructed. Leveraging conditional independence between specification subsets given the shared features, together with a positivity condition on the shared features, one can combine observed samples to form feature vectors satisfying the novel specification combination. The existence of valid novel data points is therefore guaranteed; see Theorem 2 in Appendix A.6 . Finally, analogous to the conservative recovery result in the two-specification case, non-identifiability can be addressed similarly to Proposition 4. Assuming that the probability of unobserved scenarios is low, the novel distribution p(\mathbf{X}\mid\mathbf{Z}=\mathbf{1}) becomes approximately identifiable and can be estimated from the given data, as seen from Theorem 3 in Appendix A.6 . Overview of the theory. To enable extrapolated data generation for novel combinations of specifications, our theory identifies three key conditions. First, specification variables Z_{i} are conditionally independent from each other given the features \mathbf{X} , allowing joint satisfaction to decompose into individual constraints. Second, novelty arises only from unseen specification combinations, but not their separate values: every value of the specification Z_{i} must already appear in the training data. Third, the mapping from features \mathbf{X} to specifications \mathbf{Z} should be sparse, so that as few features as possible are shared across specifications. This reduces potential conflicts among specifications and facilitates sampling from the constructed distribution; in the extreme case of no shared features (Fig. 2 (a)), extrapolation is straightforward. All the results above can be directly generalized to continuous case. In the next section, we will rely on these conditions to design practical algorithms for extrapolated data generation. 4 Model Estimation and Data Generation This section describes how to estimate the conditional distribution p(\mathbf{Z}\,|\,\mathbf{X}) from data as constraint to be required to be satisfied for extrapolated data generation and how to efficiently generate samples under novel combinations of specifications under different data availability settings. 4.1 Model Estimation Specifications and features are given. When both \mathbf{X} and \mathbf{Z} are given (i.e., tabular data), we estimate each conditional distribution term p^{\mathcal{D}}(Z_{i}|\mathbf{V}_{i}) separately. We denote \mathbf{V}_{i}\subseteq\mathbf{X} as the parent set of specification Z_{i} . We assume each specification Z_{i} follows a Gaussian distribution specified by \mathbf{V}_{i} , such that Z_{i}\sim\mathcal{N}(h^{\mu}_{i}(\mathbf{V}_{i}),h^{\sigma}_{i}(\mathbf{V}_{i})) , which yields the closed-form log-likelihood: \log p^{\mathcal{D}}(Z_{i}=z_{i}\,|\,\mathbf{V}_{i}=\mathbf{v}_{i})=-\frac{1}{2}\|\frac{Z_{i}-h^{\mu}_{i}(\mathbf{V}_{i})}{h^{\sigma}_{i}(\mathbf{V}_{i})}\|^{2}_{2}-\frac{1}{2}\log h^{\sigma}_{i}(\mathbf{v}_{i})-\frac{1}{2}\log 2\pi . Both h_{i}^{\mu} and h_{i}^{\sigma} are parameterized by flexible Multi-Layer Perceptrons (MLPs), with \sigma_{i} constrained to be non-negative. When the parent sets \mathbf{V}_{i} are known, the networks h_{i}^{\mu} and h_{i}^{\sigma} are restricted to only take \mathbf{V}_{i} as input, allowing the model to exploit the potential sparsity in the underlying structure. The estimation objective is \displaystyle\mathcal{L}_{g}=\mathbb{E}_{\mathbf{x},\mathbf{z}\sim p^{\mathcal{D}}(\mathbf{X},\mathbf{Z})}\bigg[-\sum_{i\in[n]}\log p^{\mathcal{D}}(Z_{i}=z_{i}\,|\,\mathbf{V}_{i}=\mathbf{v}_{i})\bigg], (4) where the subscript g of \mathcal{L}_{g} indicates that both features and specifications are given. If the parent sets are unknown, we introduce learnable masks \mathbf{M}_{i} to make the links between specification Z_{i} and features as sparse as possible. In this case, we first adapt \mathcal{L}_{g} to encourage sparsity: \displaystyle\mathcal{L}_{m}=\mathbb{E}_{\mathbf{x,z}}\bigg[-\sum_{i\in[n]}\log p^{\mathcal{D}}_{Z_{i}|\mathbf{X}_{i}}(Z_{i}=z_{i}\,|\,\mathbf{V}_{i}=\mathbf{x}\odot\mathbf{M}_{i})\bigg] (5) where the subscript for the summation is simplified (same as follows). The full objective becomes \mathcal{L}_{g^{\prime}}=\mathcal{L}_{m}+\beta\mathcal{L}_{s} , where \beta controls the strength of the L_{1} sparsity constraint \mathcal{L}_{s}=\sum_{i\in[n]}\|\mathbf{M}_{i}\|_{1} . Specifications and features are not given. For modalities such as images or text, where \mathbf{X} and \mathbf{Z} are latent, we estimate them from the respective observations \mathbf{Y_{X}} and \mathbf{Y_{Z}} . We employ variational autoencoders and define a generic VAE loss for an observation \mathbf{a} and its estimated latent \mathbf{b} as \displaystyle\mathcal{L}_{\mathrm{VAE}}(\mathbf{a})= \displaystyle-\mathbb{E}_{q_{\phi}(\mathbf{b}\mid\mathbf{a})}\!\left[\log p_{\theta}(\mathbf{a}\mid\mathbf{b})\right] (6) \displaystyle~~~~~+\mathrm{KL}\!\left(q_{\phi}(\mathbf{b}\mid\mathbf{a})\,\|\,p(\mathbf{b})\right), where q_{\phi}(\mathbf{b}\mid\mathbf{a}) is the encoder, p_{\theta}(\mathbf{a}\mid\mathbf{b}) is the decoder, and p(\mathbf{b}) is a prior over latents. We then define the combined objective as \displaystyle\mathcal{L}_{r}=\mathbb{E}_{\mathbf{Y_{X}},\mathbf{Y_{Z}}}\Big[\mathcal{L}_{\mathrm{VAE}}(\mathbf{Y_{X}})+\mathcal{L}_{\mathrm{VAE}}(\mathbf{Y_{Z}})\Big]. (7) Reconstruction alone does not enforce the conditional independence and sparsity that facilitate the generation of extrapolated data as we conclude in the end of previous section. These properties are imposed by combining the likelihood (as it assumes conditional independence) and sparsity terms: \mathcal{L}_{ng}=\mathcal{L}_{r}+\alpha\mathcal{L}_{m}+\beta\mathcal{L}_{s} , where \alpha adjusts the strength for the negative log-likelihood term and \beta controls the weight for the sparsity constraint. One may wonder whether the solutions for \mathbf{X} and \mathbf{Z} are asymptotically unique and, under our conditions of extrapolation, whether the underlying true representations, if they exist, are identifiable. We defer a formal discussion of identifiability and related conditions to Appendix B.1 . Figure 3 : Synthetic experiment results with given \mathbf{X} and \mathbf{Z} . Panels (a–b) illustrate the data split in a two-dimensional view over the X_{1} and X_{3} axes. The novel specifications induce a previously unseen joint distribution over (X_{1},X_{3}) , ensuring that successful performance requires true extrapolation rather than interpolation. Panels (c–h) present the optimization-based generation (OPT) results for all five models. Panel (h) shows the three-dimensional view of the Oracle \mathbf{X} . Panels (i–p) visualize the three-dimensional generated \mathbf{X} using OPT and diffusion posterior sampling (DPS) for models A, B, and C, and OPT for models D and E, with the corresponding MMD values to the Oracle \mathbf{X} reported. SANA Aligner SD3.5-L QwenImage Z-Image SEDGE GPT5.2 a peacock is eating ice cream a suitcase with bird wings a camera with insect antennae Figure 4 : Comparison of image generation results under differnet compositional prompts. Baseline models include SANA ( 32 ) , Aligner ( 34 ) , Stable Diffusion 3.5-Large ( 9 ) , QwenImage ( 30 ) , Z-Image ( 1 ) , and GPT5.2. Each row displays generated image for a prompt, and each column presents results from either our SEDGE or baseline models. Despite its relatively small size of 1.6B parameters, our method achieves performance comparable to GPT5.2 and outperforms the 20B QwenImage model, highlighting the effectiveness of our analysis for extrapolation. 4.2 Structure-Informed Extrapolated Data Generation Having estimated the required conditional distribution p(\mathbf{Z}\,|\,\mathbf{X}) (called likelihood) under different data availability settings, we now describe how to generate samples to satisfy novel combinations of specifications. Optimization-based generation. When the numbers of \mathbf{X} and \mathbf{Z} are relatively small, we can directly generate extrapolated samples via optimization. Usually in these cases the specifications and features are given. We initialize from the empirical prior, i.e., \mathbf{X}_{init}\sim p^{\mathcal{D}}_{\mathbf{X}} , and perform gradient ascent to maximize the conditional probability p(\mathbf{Z}\,|\,\mathbf{X}) to achieve extrapolation on novel specification combinations. The sparsity and conditional independence structure implied by the model significantly stabilizes optimization, as each feature is influenced by only a limited subset of specifications, reducing gradient conflicts. Diffusion-based generation. In more general settings with high-dimensional feature and specification spaces, optimization-based generation can be sensitive to initialization, as the region of attraction for high-likelihood solutions may be small. To address this issue, we adopt the diffusion posterior sampling framework of ( 6 ) to guide the step-wise sampling process to satisfy the novel combination of specification characterized by the structured p(\mathbf{Z}\,|\,\mathbf{X}) . To align with the algorithm, we train a diffusion model on \mathbf{X} (if the features are not given, we train the diffusion model on the estimated features \hat{\mathbf{X}}=f(\mathbf{Y_{X}}) ). We then generate data points from Gaussian noise and gradually apply the constraint of the likelihood in the reversed denoising process, enabling extrapolated data generation under novel combinations of specifications. Complete algorithms for both generation methods are given in Appendix C . 5 Experiments We are now ready to report simulation results with our proposed SEDGE method, followed by the results of extrapolated imagae generation. 5.1 Synthetic Experiment We verify the extrapolation conditions proposed in the previous sections. To make the synthetic setting informative, we design the data generation process and the baselines as follows. Data: We follow the structure in Fig. 1 (b) to sample \mathbf{X} and \mathbf{Z} , where X_{1},X_{3}\sim\mathcal{U}(0,1) , X_{2}\sim\mathcal{U}(0.75,0.8) , Z_{1}=0.8X_{1}+0.6Z_{2}+\epsilon_{1} , and Z_{2}=0.6X_{3}+0.8Z_{2}+\epsilon_{2} , with \epsilon_{1},\epsilon_{2}\sim\mathcal{U}(0,0.2) . We then split the dataset into training and test sets, corresponding to S=1 and S=0 , respectively. Since the selection variable S depends on \mathbf{Z} , the Oracle version of the data to be generated is defined by the condition (Z_{1}<0.8)\cap(Z_{2}>1.0) , and the remaining samples constitute Seen X (training set). This construction ensures that the evaluation genuinely tests extrapolation ability. Our model and baselines: We compare five models, denoted as models A, B, C, D, and E. The first three models adopt the “features pointing to specifications” structure shown in Fig. 2 , while the last two adopt the “specifications pointing to features” structure shown in Fig. 1 (a). Model A (ours) exploits the sparsity of the links and conditional independence among Z_{i} by estimating the forward likelihood using the exact form of Eq. 4 . Model B ignores the sparse structure and models the likelihood by conditioning each Z_{i} on all of \mathbf{X} . Model C further drops the conditional independence assumption and jointly models p(\mathbf{Z}\mid\mathbf{X}) . Model D reverses the causal direction and learns p(\mathbf{X}\mid\mathbf{Z}) , as assumed in many conditional generation models. Model E also reverses the direction but assumes each feature X_{i} is conditionally independent given \mathbf{Z} , modeling p(X_{i}\mid\mathbf{Z}) , which reflects text-conditioned image generation models ( 34 ) . For models A, B, and C, we apply both optimization-based inference and diffusion posterior sampling for extrapolated generation. For models C and E, we directly generate samples given novel specifications, as no inverse problem is involved. Metric: We evaluate the Maximum Mean Discrepancy (MMD) between the generated samples and the test-set features \mathbf{X} . We use a Gaussian kernel for MMD, for which the discrepancy is zero if and only if the two distributions are identical. Fig. 3 (a–b) illustrates the data split. As seen from the data distributions, the Oracle data (to be generated) are clearly out of support of the Seen data. Across all models, our proposed model achieves the smallest MMD, indicating the closest match between the generated data and the oracal data. The improved performance over models B and C verifies that our proposed extrapolation conditions, namely conditional independence among specifications and sparse structure, are effective and essential. For models D and E, where specifications point to features, extrapolation ability is further limited, as the generated manifolds are more constrained. Due to page limitations, we refer readers to Appendix D.1.1 for a more complete two-dimensional visualization of the generation results. For scenarios in which the specifications and features are not given but have to be learned from the observations, the only change is in their estimation, and the extrapolated data generation process stays the same once the mapping from the observations to the features or the specifications is learned. We refer readers to Appendix B.1 for a discussion of the recoverability of \mathbf{X} and \mathbf{Z} , even though this is not the main focus of this work. 5.2 Extrapolated Text-to-Image Generation To further validate our theory, we conduct extrapolation experiments on text-to-image generation to evaluate the model’s ability to synthesize unseen compositional concepts, e.g. “a peacock eating ice cream.” As an initial implementation of SEDGE, we leverage concepts from Aligner ( 34 ) to learn a graphical model with links from features (image concepts) to specifications (text concepts) and a likelihood predictor. We use diffusion-based generation since images are of high dimensions. Our generative backbone is based on SANA ( 32 ) . For further implementation details please refer to Appendix D.2 . We compare our model against several state-of-the-art baselines, including SANA, Aligner, Stable Diffusion 3.5-Large ( 9 ) , QwenImage ( 30 ) , Z-Image ( 1 ) , and GPT5.2 (Fig. 4 ). Baselines often exhibit concept leakage or include unspecified or miss specified components, such as a peacock merely touching ice cream or the addition of redundant wings and birds. Our method produces semantically coherent images without missing attributes. Notably, our 1.6B parameter model achieves performance better than the 20B QwenImage and comparable to GPT5.2, demonstrating its extrapolation ability and efficiency even with limited training data. 6 Conclusion For generative AI to reliably produce novel content, the ability to extrapolate beyond the training data is essential. Unlike traditional approaches that often rely on assumptions about the functional forms of the underlying relationships, we investigate this problem from a structural perspective, by employing a graphical model to represent how features and property specifications are related, as well as how the training data are limited. This framework enables us to characterize which components in the decomposition of the joint distribution of the training data can be leveraged to inform extrapolated data generation, and how this can be achieved in practice. We further provide conditions under which novel data can be generated reliably, as well as conditions under which the distribution of the novel data can be approximately constructed. As future work, we plan to explore how these ideas can address important problems such as drug discovery and novel protein structure prediction. Impact Statement This paper presents work whose goal is to advance the field of machine learning, especially for the purpose of achieving robustness and trustworthiness in extrapolated data generation. There are many potential societal consequences, none of which we feel must be specifically highlighted here. Appendix A Appendix A.1 Related Work Extrapolation Existing approaches for extrapolation largely fall into two categories, distinguished by where the assumption are imposed. The first category make assumptions on the data distribution, typically assuming the target distribution shares the same support as the source. It includes most work on the topic of out-of-distribution generalization, which focuses on invariance or reweighting rather than extrapolation on novel specifications ( 36 ) . There is also work in regression and statistical inference studying prediction or uncertainty guarantees outside the observed support ( 8 ; 26 ) without addressing novel combinations of specifications. The second category introduce functional assumptions on the generation process to regularize the behavior ouside training support. Special restrictions are introduced to data generating process including linear, additive, slot-wise, compositional generators, or other specific parametric relations ( 17 ; 15 ; 29 ; 24 ) . Other work derives extrapolation guarantees from regularities on distribution shifts, for instance, smoothness ( 13 ) . These assumptions are hard to be verified from observed data and do not provide a general framework for extrapolated data generation on novel specifications. In contrast, we provide a general structural view to the problem that does not assume a particular functional form of the generating mechanism or smoothness of shifts. Text-to-Image Generation In recent years, diffusion-based approaches have become the dominant paradigm for text-to-image generation due to their strong performance ( 22 ; 9 ; 1 ; 30 ; 14 ) . One line of research focuses on improving the efficiency of training and sampling ( 4 ; 2 ; 32 ; 33 ; 3 ; 5 ) . Another line of work aims to enhance controllability by incorporating additional conditioning signals, such as structural or semantic guidance ( 38 ; 37 ; 39 ; 35 ) . A further line of research studies disentanglement in text-to-image generation, seeking to separate and manipulate semantic factors within the generative process ( 31 ; 16 ; 34 ; 7 ) . In this paper, we consider text-to-image generation as a real problem to address, but in the context of extrapolation. A.2 Proof of Proposition 1 See 1 Proof. By Bayes Rule, we have \displaystyle p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) \displaystyle=p(X_{1},X_{2}\,|\,Z_{1}=1,Z_{2}=1) \displaystyle\qquad\cdot p(X_{3},X_{4}\,|\,X_{1},X_{2},Z_{1}=1,Z_{2}=1) \displaystyle=p(X_{1},X_{2}\,|\,Z_{1}=1)p(X_{3},X_{4}\,|\,Z_{2}=1) (8) \displaystyle=p^{\mathcal{D}}(X_{1},X_{2}\,|\,Z_{1}=1)p^{\mathcal{D}}(X_{3},X_{4},\,|\,Z_{2}=1), (9) where Equation 8 follows from \{X_{1},X_{2}\}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z_{2}\,|\,Z_{1} and \{X_{3},X_{4}\}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\{X_{1},X_{2},Z_{1}\}\,|\,Z_{2} . Equation 9 follows from \{X_{1},X_{2}\}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}S\,|\,Z_{1} , \{X_{3},X_{4}\}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}S\,|\,Z_{2} , and Equation 1 in 1 . ∎ A.3 Proof of Proposition 2 See 2 Proof. We begin with deriving the data distribution following the novel specification, p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) . \displaystyle p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) (10) \displaystyle= \displaystyle p(X_{1},X_{2}\,|\,Z_{1}=1,Z_{2}=1)p(X_{3}\,|\,X_{1},X_{2},Z_{1}=1,Z_{2}=1) (11) \displaystyle= \displaystyle p(X_{2}\,|\,Z_{1}=1,Z_{2}=1)p(X_{1}\,|\,X_{2},Z_{1}=1,Z_{2}=1)p(X_{3}\,|\,X_{1},X_{2},Z_{1}=1,Z_{2}=1) (12) \displaystyle= \displaystyle p(X_{2}\,|\,Z_{1}=1,Z_{2}=1)p(X_{1}\,|\,X_{2},Z_{1}=1,)p(X_{3}\,|\,X_{2},Z_{2}=1) (13) \displaystyle= \displaystyle p(X_{2}\,|\,Z_{1}=1,Z_{2}=1)p(X_{1}\,|\,X_{2},Z_{1}=1,S=1)p(X_{3}\,|\,X_{2},Z_{2}=1,S=1) (14) \displaystyle= \displaystyle p(X_{2}\,|\,Z_{1}=1,Z_{2}=1)p^{\mathcal{D}}(X_{1}\,|\,X_{2},Z_{1}=1)p^{\mathcal{D}}(X_{3}\,|\,X_{2},Z_{2}=1), (15) Equation 13 holds because X_{1}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z_{2}\,|\,\{X_{2},Z_{1}\} and X_{3}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\{Z_{1},X_{1}\}\,|\,\{X_{2},Z_{2}\} . Equation 15 is the case because X_{1}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}S\,|\,\{X_{2},Z_{1}\} and X_{3}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}S\,|\,\{X_{2},Z_{2}\} . Conditioning on S=1 means that they are distributions implies by the given data. Note that although X_{2}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}S\,|\,\{Z_{1},Z_{2}\} implies p(X_{2}\,|\,Z_{1}=1,Z_{2}=1)=p(X_{2}\,|\,Z_{1}=1,Z_{2}=1,S=1), the quantity on the right-hand side is not defined since for the conditioning variables, p(Z_{1}=1,Z_{2}=1,S=1)=p(Z_{1}=1,Z_{2}=1\,|\,S=1)p(S=1)=0 , as the given data do not satisfy Z_{1}=1,Z_{2}=1 . Deriving p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) then reduces to the estimation of p(X_{2}\,|\,Z_{1}=1,Z_{2}=1) . \displaystyle p(X_{2}\,|\,Z_{1}=1,Z_{2}=1) (16) \displaystyle= \displaystyle p(Z_{1}=1,Z_{2}=1\,|\,X_{2})\cdot\frac{p(X_{2})}{P(Z_{1}=1,Z_{2}=1)} (17) \displaystyle\propto \displaystyle p(Z_{1}=1,Z_{2}=1\,|\,X_{2})p(X_{2}) (18) \displaystyle= \displaystyle p(Z_{1}=1\,|\,X_{2})p(Z_{2}=1\,|\,X_{2})p(X_{2}) (19) \displaystyle\propto \displaystyle p(X_{2}\,|\,Z_{1}=1)p(X_{2}\,|\,Z_{2}=1)\cdot\frac{1}{p(X_{2})}. (20) The problem is that none of the three involved quantities, p(X_{2}\,|\,Z_{1}=1) , p(X_{2}\,|\,Z_{1}=1) , and \cdot\frac{1}{p(X_{2})} is known since we only have access to the distribution corresponding to the given data, for which S=1 . In fact, p(X_{2}\,|\,Z_{1}=1)=p(X_{2}\,|\,Z_{1}=1,S=1)P(S=1\,|\,Z_{1}=1)+p(X_{2}\,|\,Z_{1}=1,S=0)P(S=0\,|\,Z_{1}=1). (21) We do not have information of the data with S=0 , which are not in the given dataset. As a consequence, neither of p(X_{2}\,|\,Z_{1}=1,S=0) and P(S=0\,|\,Z_{1}=1) is known. Hence, p(X_{2}\,|\,Z_{1}=1) is not identifiable from the given data, and it also happens to p(X_{2}\,|\,Z_{1}=1) and {p(X_{2})} . Therefore, p(X_{2}\,|\,Z_{1}=1,Z_{2}=1) is not identifiable , and hence according to equation 15 , p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) is not identifiable given the distribution of the observed data. ∎ A.4 Proof of Proposition 3 See 3 Proof. First, note that p(S=1\,|\,Z_{1}=1)>0 as for the given data, S=1 , and bear in mind that p(X_{2}=\tilde{x}_{2}\,|\,Z_{1}=1,S=1)>0, according to 2 . Then according to equation 21 , we have p(X_{2}=\tilde{x}_{2}\,|\,Z_{1}=1)>0. Similarly, we have p(X_{2}=\tilde{x}_{2}\,|\,Z_{2}=1)>0. Then according to equation 19 , p(X_{2}=\tilde{x}_{2}\,|\,Z_{1}=1,Z_{2}=1)>0 . We further know that there exist \tilde{x}_{1} , as values of X_{1} , such that p^{\mathcal{D}}(X_{1}=\tilde{x}_{1}\,|\,X_{2}=\tilde{x}_{2},Z_{1}=1)>0; this is because otherwise, p^{\mathcal{D}}(X_{1}=\tilde{x}_{1}\,|\,X_{2}=\tilde{x}_{2},Z_{1}=1)=0 implies p^{\mathcal{D}}(X_{1},X_{2}=\tilde{x}_{2}\,|\,Z_{1}=1)=0 and hence p^{\mathcal{D}}(X_{2}=\tilde{x}_{2}\,|\,Z_{1}=1)=0 , contradicting 2 . Similarly, there exist values of X_{3} , denoted by \tilde{x}_{3} , such that p^{\mathcal{D}}(X_{3}=\tilde{x}_{3}\,|\,X_{2}=\tilde{x}_{2},Z_{2}=1)>0. Finally, according to equation 15 , we obtain p(X_{2}=\tilde{x}_{2},X_{1}=\tilde{x}_{1},X_{3}=\tilde{x}_{3}\,|\,Z_{1}=1,Z_{2}=1)>0. ∎ A.5 Proof of Proposition 4 See 4 Proof. By 3 , one can show \displaystyle p(X_{2}\,|\,Z_{i}=1) \displaystyle\approx p^{\mathcal{D}}(X_{2}\,|\,Z_{i}=1),\quad i=1,2, \displaystyle p(X_{2}) \displaystyle\approx p^{\mathcal{D}}(X_{2}). Plugging these into Equations 15 and 20 yields \displaystyle p(\mathbf{X}\,|\,Z_{1}=1,Z_{2}=1) \displaystyle\approx \displaystyle\frac{p^{\mathcal{D}}(X_{2}\,|\,Z_{1}=1)p^{\mathcal{D}}(X_{2}\,|\,Z_{2}=1)}{p^{\mathcal{D}}(X_{2})}\frac{P(Z_{1}=1)P(Z_{2}=1)}{P(Z_{1}=1,Z_{2}=1)} \displaystyle\quad\cdot p^{\mathcal{D}}(Z_{2}=1\,|\,X_{2})p^{\mathcal{D}}(X_{3}\,|\,X_{2},Z_{2}=1) (22) \displaystyle\propto \displaystyle p^{\mathcal{D}}(X_{1},X_{2}\,|\,Z_{1}=1)p^{\mathcal{D}}(Z_{2}=1\,|\,X_{2}) \displaystyle\quad\cdot p^{\mathcal{D}}(X_{3}\,|\,X_{2},Z_{2}=1). (23) ∎ A.6 SEDGE with Multiple Specifications We now extend the results from the previous subsection to show how structural properties can be exploited to enable extrapolated data generation in the general setting with multiple specifications. Analogous to the two-specification setting in 1 , we aim to extrapolate to the novel combination of specifications \mathbf{Z}=1 that is unseen in the given data due to selection, i.e., p(\mathbf{Z}=1\,|\,S=1)=0 . We introduce the following basic conditions. Assumption 4 (Basic conditions for specifications and selection) . We have \displaystyle P(S=1) \displaystyle>0, (24) \displaystyle P(\mathbf{Z}=\mathbf{1}) \displaystyle>0, (25) \displaystyle P(Z_{i}=1\,|\,S=1) \displaystyle>0,\quad i\in[k]. (26) Specifically, Equation 24 ensures that the given dataset is nonempty. Equation 25 guarantees that the target specification combination is feasible in the underlying population, while Equation 26 ensures that each individual specification value appears in the given/training data in some context, providing the necessary marginal coverage to potentially extrapolate to the novel combinations. A.6.1 Identifiable Case: Specifications Do Not Share Common Features Recall that Figure 2 (a) and Proposition 4 consider the setting in which specifications do not share common features, by making use of the independence between Z_{1} and Z_{2} , i.e., Z_{1}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z_{2} . Building on this intuition, we now extend it to multiple specifications. We present the following result, which shows that when specifications do not share common features, the distribution corresponding to novel specifications, p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1}) , is identifiable from the observed data. Theorem 1 . Suppose that there exists a partition of the specifications \mathbf{Z} into disjoint specification subsets \mathbf{Z}_{1},\dots,\mathbf{Z}_{k} such that no feature is shared between any two different subsets and that features across different subsets are not adjacent. Suppose further that 4 holds and P(\mathbf{Z}_{i}=1\,|\,S=1)>0,\quad i\in[k]. Then, the novel data distribution p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1}) is identifiable from the given data. The proof is given in Section A.7 . It is worth noting that, unlike the setting in Figure 2 (a) that requires all specifications not to share any feature, the result above further allows the specifications to be partitioned into subset of specifications. That is, while no features are shared across different subsets, arbitrary feature sharing and dependence relations are permitted within each subset. A.6.2 Non-Identifiable Case: Specifications Share Common Features Previously, Figure 2 (b) and Proposition 3 study the setting in which two specifications share common features, by leveraging the conditional independence between Z_{1} and Z_{2} given their common child X_{2} , i.e., Z_{1}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}Z_{2}\,|\,X_{2} . Analogous to this case, we now consider a more general setting with multiple specifications that share common features. As shown in Proposition 2 , the distribution satisfying novel specifications is generally not identifiable. In the following, we extend Proposition 3 from the two-specification setting to show that generating novel data samples remains possible with multiple specifications, and generalize Proposition 4 to show how the non-identifiability issue with multiple specifications can be addressed by learning conservative solutions. Existence of novel data sample. The following result shows that, under a generalization of 2 which requires that the common features have a positive density under each set of specifications individually in the given data, one can construct data points satisfying the novel specifications \mathbf{Z}=\mathbf{1} from the given data. Theorem 2 . Suppose that there exists a partition of the specifications \mathbf{Z} into disjoint specification subsets \mathbf{Z}_{1},\dots,\mathbf{Z}_{k} such that they are conditionally independent given a subset of features \mathbf{X}_{c}\subset\mathbf{X} . Moreover, suppose that, given \mathbf{X}_{c} , each feature X_{j}\notin\mathbf{X}_{c} is conditionally independent of all other features that are not associated with the same specification subset as X_{j} . 2 2 2 Here, a feature is said to be associated with a specification set if it is a parent of at least one specification variable in that set. Suppose further that 4 holds, P(\mathbf{Z}_{i}=1\,|\,S=1)>0,\quad i\in[k], and there exists value \tilde{\mathbf{x}}_{c} of \mathbf{X}_{c} such that p^{\mathcal{D}}(\mathbf{X}_{c}=\tilde{\mathbf{x}}_{c}\,|\,\mathbf{Z}_{i}=1)>0,\quad i\in[k]. Then, there exists a value \tilde{\mathbf{X}=\mathbf{x}} of \mathbf{X} that can be constructed from the given data such that p(\mathbf{X}=\tilde{\mathbf{x}}\,|\,\mathbf{Z}=\mathbf{1})>0. The proof is provided in Section A.8 . The theorem is rather general and accommodates complex structures, allowing one to exploit conditional independencies implied by the structural properties to generate data points satisfying novel specifications in a broad range of settings. Conservative solutions for non-identifiable distributions. Similar to Proposition 4 with two specifications, the non-identifiability issue of p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1}) with multiple specifications can be resolved by assuming that the chance of having novel scenarios in the data generating process, which are not contained in the given data, is low. This is formally stated in the following result, with a proof given in Section A.9 . Theorem 3 . Suppose that there exists a partition of the specifications \mathbf{Z} into disjoint specification subsets \mathbf{Z}_{1},\dots,\mathbf{Z}_{k} such that they are conditionally independent given a subset of features \mathbf{X}_{c}\subset\mathbf{X} . Moreover, suppose that, given \mathbf{X}_{c} , each feature X_{j}\notin\mathbf{X}_{c} is conditionally independent of all other features that are not associated with the same specification subset as X_{j} . Suppose further that 4 holds and \displaystyle P(\mathbf{Z}_{i}=1\,|\,S=1) \displaystyle>0,\quad i\in[k], \displaystyle P(S=0\,|\,\mathbf{Z}_{i}=1) \displaystyle\approx 0,\quad i\in[k], \displaystyle P(S=0) \displaystyle\approx 0. Then, the novel data distribution p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1}) is approximately identifiable from the given data. A.7 Proof of Theorem 1 See 1 Proof. With a slight abuse of notation, let \mathbf{V}_{\mathbf{Z}_{i}}\subseteq\mathbf{X} be the parent set of specification subset \mathbf{Z}_{i} . By assumption, no feature is shared between the specification subsets \mathbf{Z}_{1},\dots,\mathbf{Z}_{k} , and features across different subsets are not adjacent. This implies that (1) \mathbf{V}_{\mathbf{Z}_{1}},\dots,\mathbf{V}_{\mathbf{Z}_{k}} are conditionally independent of each other given \mathbf{Z}_{1},\dots,\mathbf{Z}_{k} and (2) \mathbf{V}_{\mathbf{Z}_{i}}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\mathbf{Z}_{j}\,|\,\mathbf{Z}_{i} for i\neq j . By Bayes Rule, we have \displaystyle p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1}) \displaystyle=\prod_{i=1}^{k}p(\mathbf{V}_{\mathbf{Z}_{i}}\,|\,\mathbf{Z}=\mathbf{1}) \displaystyle=\prod_{i=1}^{k}p(\mathbf{V}_{\mathbf{Z}_{i}}\,|\,\mathbf{Z}_{i}=\mathbf{1}) \displaystyle=\prod_{i=1}^{k}p(\mathbf{V}_{\mathbf{Z}_{i}}\,|\,\mathbf{Z}_{i}=\mathbf{1},S=1) \displaystyle=\prod_{i=1}^{k}p^{\mathcal{D}}(\mathbf{V}_{\mathbf{Z}_{i}}\,|\,\mathbf{Z}_{i}=\mathbf{1}). Here, the second line follows from the fact that \mathbf{V}_{\mathbf{Z}_{1}},\dots,\mathbf{V}_{\mathbf{Z}_{k}} are conditionally independent of each other given \mathbf{Z}_{1},\dots,\mathbf{Z}_{k} . The third line follows from \mathbf{V}_{\mathbf{Z}_{i}}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}\mathbf{Z}_{j}\,|\,\mathbf{Z}_{i} for i\neq j and the assumption of P(\mathbf{Z}_{i}=1\,|\,S=1)>0 . The fourth line follows from \mathbf{V}_{\mathbf{Z}_{i}}\mathchoice{\mathrel{\hbox to0.0pt{$\displaystyle\perp$\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{$\textstyle\perp$\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptstyle\perp$\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{$\scriptscriptstyle\perp$\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}S\,|\,\mathbf{Z}_{i} . ∎ A.8 Proof of Theorem 2 See 2 Proof. With a slight abuse of notation, let \mathbf{V}_{\mathbf{Z}_{i}}\subseteq\mathbf{X} be the parent set of specification subset \mathbf{Z}_{i} . By assumption, the disjoint specification subsets \mathbf{Z}_{1},\dots,\mathbf{Z}_{k} are conditionally independent given \mathbf{X}_{c} . Also, each feature X_{j}\notin\mathbf{X}_{c} is conditionally independent of all other features that are not associated with the same specification subset as X_{j} . By Bayes rule, We have \displaystyle p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1}) \displaystyle= \displaystyle p(\mathbf{X}_{c}\,|\,\mathbf{Z}=\mathbf{1})p(\mathbf{X}\setminus\mathbf{X}_{c}\,|\,\mathbf{X}_{c},\mathbf{Z}=\mathbf{1}) \displaystyle= \displaystyle p(\mathbf{X}_{c}\,|\,\mathbf{Z}=\mathbf{1})\prod_{i=1}^{k}p(\mathbf{V}_{\mathbf{Z}_{i}}\setminus\mathbf{X}_{c}\,|\,\mathbf{X}_{c},\mathbf{Z}=\mathbf{1}) \displaystyle= \displaystyle p(\mathbf{X}_{c}\,|\,\mathbf{Z}=\mathbf{1})\prod_{i=1}^{k}p(\mathbf{V}_{\mathbf{Z}_{i}}\setminus\mathbf{X}_{c}\,|\,\mathbf{X}_{c},\mathbf{Z}_{i}=\mathbf{1}) \displaystyle= \displaystyle p(\mathbf{X}_{c}\,|\,\mathbf{Z}=\mathbf{1})\prod_{i=1}^{k}p(\mathbf{V}_{\mathbf{Z}_{i}}\setminus\mathbf{X}_{c}\,|\,\mathbf{X}_{c},\mathbf{Z}_{i}=\mathbf{1},S=1) \displaystyle= \displaystyle p(\mathbf{X}_{c}\,|\,\mathbf{Z}=\mathbf{1})\prod_{i=1}^{k}p^{\mathcal{D}}(\mathbf{V}_{\mathbf{Z}_{i}}\setminus\mathbf{X}_{c}\,|\,\mathbf{X}_{c},\mathbf{Z}_{i}=\mathbf{1}) (27) Here, the third line follows from the fact that, given \mathbf{X}_{c} , each feature X_{j}\notin\mathbf{X}_{c} is conditionally independent of all other features that are not associated with the same specification subset as X_{j} . The fourth line follows from the fact that each X_{j}\in\mathbf{V}_{\mathbf{Z}_{i}}\setminus\mathbf{X}_{c} is conditionally independent with \mathbf{Z}_{l},l\neq i given \mathbf{X}_{c} . The fifth line follows from the assumption of P(\mathbf{Z}_{i}=1\,|\,S=1)>0 and the fact that each X_{j}\in\mathbf{V}_{\mathbf{Z}_{i}}\setminus\mathbf{X}_{c} is conditionally independent with S given \mathbf{X}_{c} . Deriving p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1}) then reduces to the estimation of p(\mathbf{X}_{c}\,|\,\mathbf{Z}=\mathbf{1}) . We have \displaystyle p(\mathbf{X}_{c}\,|\,\mathbf{Z}=\mathbf{1}) \displaystyle= \displaystyle p(\mathbf{Z}=\mathbf{1}\,|\,\mathbf{X}_{c})\cdot\frac{p(\mathbf{X}_{c})}{P(\mathbf{Z}=\mathbf{1})} \displaystyle\propto \displaystyle p(\mathbf{Z}=\mathbf{1}\,|\,\mathbf{X}_{c})p(\mathbf{X}_{c}) \displaystyle= \displaystyle\prod_{i=1}^{k}p(\mathbf{Z}_{i}=\mathbf{1}\,|\,\mathbf{X}_{c})p(\mathbf{X}_{c}) (28) \displaystyle\propto \displaystyle\prod_{i=1}^{k}p(\mathbf{X}_{c}\,|\,\mathbf{Z}_{i}=\mathbf{1})\cdot\frac{1}{p(\mathbf{X}_{c})}. (29) By assumption, there exists value \tilde{\mathbf{x}}_{c} of \mathbf{X}_{c} such that p^{\mathcal{D}}(\mathbf{X}_{c}=\tilde{\mathbf{x}}_{c}\,|\,\mathbf{Z}_{i}=1)>0,\quad i\in[k], This implies p(\mathbf{X}_{c}=\tilde{\mathbf{x}}_{c}\,|\,\mathbf{Z}_{i}=1)>0. Then according to Equation 29 , p(\mathbf{X}_{c}=\tilde{\mathbf{x}}_{c}\,|\,\mathbf{Z}=\mathbf{1})>0 . We further know that there exist \tilde{\mathbf{v}}_{\mathbf{Z}_{i}}\setminus\tilde{\mathbf{x}}_{c} , as values of \mathbf{V}_{\mathbf{Z}_{i}}\setminus\mathbf{X}_{c} , such that p^{\mathcal{D}}(\mathbf{V}_{\mathbf{Z}_{i}}\setminus\mathbf{X}_{c}=\tilde{\mathbf{v}}_{\mathbf{Z}_{i}}\setminus\tilde{\mathbf{x}}_{c}\,|\,\mathbf{X}_{c}=\mathbf{x}_{c},\mathbf{Z}_{i}=\mathbf{1})>0; Finally, according to equation Equation 27 , we can construct the point such that p(\mathbf{X}=\tilde{x}\,|\,\mathbf{Z}=\mathbf{1})>0. ∎ A.9 Proof of Theorem 3 See 3 Proof. By assumption, one can show \displaystyle p(\mathbf{X}_{c}\,|\,\mathbf{Z}_{i}=\mathbf{1}) \displaystyle\approx p^{\mathcal{D}}(\mathbf{X}_{c}\,|\,\mathbf{Z}_{i}=\mathbf{1}),\quad i\in[k], \displaystyle p(\mathbf{X}_{c}) \displaystyle\approx p^{\mathcal{D}}(\mathbf{X}_{c}). Plugging these into Equations 29 and 27 yields p(\mathbf{X}\,|\,\mathbf{Z}=\mathbf{1})\propto\frac{\prod_{i=1}^{k}p^{\mathcal{D}}(\mathbf{X}_{c}\,|\,\mathbf{Z}_{i}=1)}{p^{\mathcal{D}}(\mathbf{X}_{c})}\prod_{i=1}^{k}p^{\mathcal{D}}(\mathbf{V}_{\mathbf{Z}_{i}}\setminus\mathbf{X}_{c}\,|\,\mathbf{X}_{c},\mathbf{Z}_{i}=\mathbf{1}). ∎ Appendix B Appendix on Derivations B.1 Discussion of the Identification of the Specifications and Features ( 34 ) gives conditions for identifiability for both the features \mathbf{X} and the specifications \mathbf{Z} . In our work, we share the part where the specifications are required to be identifiable in order to decompose the constraint of a novel combination of specifications into individual ones, and the individual constraints are already defined as seen in the data (recall that there is no novel value for any specification Z_{i} ). The component-wise identifiability of \mathbf{Z} is mostly built on the conditional independence of each Z_{i} given \mathbf{X} (see Condition 4.2 and Theorem 4.4 in their work). However, Condition 4.3 for deriving component-wise identifiability of \mathbf{X} is rather strong in our setting, and our extrapolated data generation does not require each X_{i} to be identifiable. Instead, to guarantee a sufficient and clean estimate of the conditional distribution p(Z_{i}\mid\mathbf{V}_{i}) , we need to locate the subspace of \mathbf{V}_{i} . Such a property should be facilitated by the sparsity constraint of the functions from \mathbf{X} to Z_{i} . At the same time, features that connect to multiple specifications should be easier to identify. In this work, we provide preliminary results on the identification of \mathbf{X} and \mathbf{Z} with sparsity constraints. The experimental setting is straightforward: we follow the same structure as Fig. 2 (b) to generate \mathbf{X} and \mathbf{Z} , assuming linear gaussian model for the latent \mathbf{X} and \mathbf{Z} . Then, we apply linear mixture to the generated \mathbf{X} and \mathbf{Z} to produce the actual observations \mathbf{Y_{X}} and \mathbf{Y_{Z}} , respectively. The parameters for the latent model and the data split are the same as what we used in the synthetic experiment in the main content. To generate the generated data \mathbf{Y}_{X} and \mathbf{Y}_{Z} , of which both the dimensions are set to 20 , the linear mixing transformations for is uniformly sampled from \mathcal{U}(0,1) . Finally, we follow the estimation procedure in Section 4 and obtain the estimated \mathbf{X} and \mathbf{Z} . We demonstrate the results by enforcing a sparsity strength of \beta=5e^{-4} and a likelihood constraint with weight \alpha=0.05 . See the scatter plots in Figure 5 and Figure 6 . Figure 5 : Identification of Specifications ( Z[0] and Z[1] corresponds to Z1 and Z2 , respectively.) Figure 6 : Identification of Features ( X[0] , X[1] , and X[2] correspond to X1 , X2 , and X3 , respectively.) The identifiability of both features and specifications is more universally verified by averaging the Mean Correlation Coefficient (MCC) between the estimated variables and the ground truth variables on multiple runs. In the table below, we collect the averaged MMC scores for Z_{i};i=1,2 and {}_{i};i=1,2,3 from 10 runs. See the results in Table 1 Table 1 : MCC scores of features and specifications for verification of identifiability X_{1} X_{2} X_{3} Z_{1} Z_{2} 0.4346 0.9017 0.7097 0.9088 0.9393 Additionally, in Table 2 , we show that by using the recovered features and specifications, we can achieve comparable MMD scores as the scenario where both are given. Table 2 : MMD scores of extrapolated generation by two proposed methods MMD (OPT) MMD (DPS) 0.0332\pm 0.0024 0.0346\pm 0.0029 Appendix C Appendix of Structured Extrapolated Data Generation Algorithms Algorithm 1 Structure-Informed Extrapolated Sampling (Optimization-Based) Input: target specifications \mathbf{z}\in\mathbb{R}^{n} ; forward models \{h_{i}:\mathbf{V}_{i}\to Z_{i}\}_{i=1}^{n} where \mathbf{V}_{i}\subseteq\mathbf{X} and h_{i} consist h_{i}^{\mu} and h_{i}^{\sigma} ; prior diffusion model p_{\mathbf{X}} trained on \mathbf{X} ; normalization statistics \mu_{\mathbf{X}},\sigma_{\mathbf{X}},\mu_{\mathbf{Z}},\sigma_{\mathbf{Z}} ; learning rate \eta>0 and optimization step T_{\mathrm{opt}} . Output: extrapolated sample \mathbf{x}\in\mathbb{R}^{d} . 1. Preprocessing: Normalize target: \hat{\mathbf{z}}\leftarrow(\mathbf{z}-\mu_{\mathbf{Z}})/\sigma_{\mathbf{Z}} . 2. Method 1: Optimization-based posterior sampling (a) Initialize from training data: \mathbf{x}^{(0)}\leftarrow\mathbf{X}_{\text{train}}[\text{random\_idx}] (normalized). (b) For t=1,\ldots,T_{\text{opt}} (e.g., T_{\text{opt}}=1000 ): i. Compute forward predictions: each dimension of \hat{\mathbf{z}} is computed by \hat{\mathbf{z}}_{i}^{(t)}\leftarrow f_{i}(\mathbf{V}_{i}^{(t)}) with gradients enabled. ii. Compute likelihood loss normalized space: \mathcal{L}_{g}^{(t)}=\left\|\hat{\mathbf{z}}^{(t)}-\mathbf{z}\right\|_{2}^{2} (batch iteration is ignored). iii. Compute loss in the normalized space: \mathcal{L}_{g}^{(t)}=\frac{1}{n}\sum_{i=1}^{n}\left\|\hat{\mathbf{z}}_{i}^{(t)}-\mathbf{z}\right\|_{2}^{2} . iv. Compute gradient: \mathbf{g}^{(t)}\leftarrow\nabla_{\mathbf{x}}-\mathcal{L}_{g}^{(t)} . v. Update with Adam optimizer: \mathbf{x}^{(t)}\leftarrow\text{Adam}(\mathbf{x}^{(t-1)},\mathbf{g}^{(t)},\eta) . (c) Denormalize: \mathbf{x}\leftarrow\sigma_{\mathbf{X}}\cdot\mathbf{x}^{(T_{\text{opt}})}+\mu_{\mathbf{X}} . Algorithm 2 Structure-Informed Extrapolated Sampling (Diffusion Posterior Sampling) ( 6 ) Input: target specifications \mathbf{z}\in\mathbb{R}^{n} ; forward models \{h_{i}:\mathbf{V}_{i}\to Z_{i}\}_{i=1}^{n} where \mathbf{V}_{i}\subseteq\mathbf{X} ; diffusion model p_{\mathbf{X}} trained on \mathbf{X} ; normalization statistics \mu_{\mathbf{X}},\sigma_{\mathbf{X}},\mu_{\mathbf{Z}},\sigma_{\mathbf{Z}} ; guidance scale \lambda>0 and diffusion steps T (DPS). Output: extrapolated sample \mathbf{x}\in\mathbb{R}^{d} . 1. Preprocessing: Normalize target: \hat{\mathbf{z}}\leftarrow(\mathbf{z}-\mu_{\mathbf{Z}})/\sigma_{\mathbf{Z}} . 2. Method 2: Diffusion posterior sampling (DPS) (a) Initialize from noise: \mathbf{x}^{(T)}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) . (b) For t=T,T-1,\ldots,1 : i. Prior denoising step: A. Predict noise: \hat{\boldsymbol{\epsilon}}^{(t)}\leftarrow\epsilon_{\theta}(\mathbf{x}^{(t)},t) using prior model. B. Compute predicted mean: \boldsymbol{\mu}^{(t)}\leftarrow\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}^{(t)}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\hat{\boldsymbol{\epsilon}}^{(t)}\right) ii. Likelihood guidance: A. Compute forward predictions: each dimension of \hat{\mathbf{z}} is computed by \hat{\mathbf{z}}_{i}^{(t)}\leftarrow h_{i}(\mathbf{V}_{i}^{(t)}) with gradients enabled. B. Compute likelihood loss normalized space: \mathcal{L}_{g}^{(t)}=\left\|\hat{\mathbf{z}}^{(t)}-\mathbf{z}\right\|_{2}^{2} (batch iteration is ignored). C. Compute guidance gradient: \mathbf{g}_{\text{like}}^{(t)}\leftarrow\nabla_{\mathbf{x}}\mathcal{L}_{\text{like}}^{(t)} . D. Apply guidance: \boldsymbol{\mu}^{(t)}\leftarrow\boldsymbol{\mu}^{(t)}-\lambda\cdot\mathbf{g}_{\text{like}}^{(t)} . iii. Add noise and update: \mathbf{x}^{(t-1)}\leftarrow\begin{cases}\boldsymbol{\mu}^{(t)}+\sqrt{\beta_{t}}\cdot\boldsymbol{\epsilon},&\text{if }t>1,\quad\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\\ \boldsymbol{\mu}^{(t)},&\text{if }t=1\end{cases} (c) Denormalize: \mathbf{x}\leftarrow\sigma_{\mathbf{X}}\cdot\mathbf{x}^{(0)}+\mu_{\mathbf{X}} . Appendix D Appendix on Experiment Details and Additional Results D.1 Synthetic Experiments The split of the data set given the distribution of \mathbf{Z} is given in Fig. 7 , where the marginal distribution of each Z_{i} is complete, but only some parts of the joint distribution are unseen in the given dataset. And out data generation setting ensures that to satisfy such novel combinations of specifications, the ability to do extrapolated generation is necessary, as shown in Fig 3 (a-b). Figure 7 : Data Split Based on Specifications D.1.1 More Results on Given Features and Specifications Fig. 8 and Fig. 9 show more 2-d views of the data points generated by our method (model A) and other baselines. From both diagrams, it is clear to see that the samples generated by our method are the most close to the oracle \mathbf{X} and achieves extrapolation on novel combination of specifications. Figure 8 : Complete 2-d view of the extrapolated data points compared with baselines (model B, C, D, E) on optimization-based generation method Figure 9 : Complete 2-d view of the extrapolated data points compared with baselines (model B, C) on diffusion posterior sampling method D.2 Extrapolated Text-to-Image Generation In this section, we first describe the experimental setup, including implementation details and datasets. D.2.1 Experiment Setup As mentioend in the main paper, we use Diffusion-based generation (i.e. diffusion posterior sampling, DPS) here since images (even image concepts) are of high dimensions. Therefore, we need three components: a likelihood model, a causal mask, and a prior model. To give an initial implementation of SEDGE method in text-to-image generation task, we use a simple model for this qualitative experiment. We want to do extrapolation for image, so we define the image concepts ( z^{I} ) as the feature variables and text concepts ( z^{T} ) as the specification ones. We assume the concepts learned by Aligner ( 34 ) are the ground-truth concepts, so the task fall into Scenario 1 in Synthetic experiment, i.e. specifications and features are given. However, we do not have the true causal graph from features to specifications, since the original causal relations learned by Aligner is inverse to our setting. So we design a model to train a likelihood predictor while leanring the causal relations from image concepts to text concepts. We use N_{I} and N_{T} as the number of image and text concepts, respectively. Likelihood Predictor. To model the conditional dependency between image and text concepts, we use a concept-wise conditional flow model to estimate p^{\mathcal{D}}(\mathbf{Z}\,|\,\mathbf{X})=p(\mathbf{z}^{T}\mid\mathbf{z}^{I}). For each target text concept z^{T}_{i} , the model employs a learnable soft-mask \mathbf{M}_{i}\in[0,1]^{N_{I}} to aggregate its parent image concepts z^{I} (denoted as \mathbf{V}_{i} in the main paper). This masked context is then encoded via an MLP to parameterize a conditional normalizing flow, which computes the log-likelihood of z^{T}_{i} , \log p(z^{T}_{i}\mid z^{I})=\log p_{u}(f_{i}(z^{T}_{i};c_{i}))+\log\left|\det\frac{\partial f_{i}}{\partial z^{T}_{i}}\right|, where c_{i}=\operatorname{MLP}_{i}(z^{I}\odot\mathbf{M}_{i}) , p_{u} represents a standard Gaussian distribution \mathcal{N}(0,\mathbf{I}) . The entire predictor is optimized by minimizing the total negative log-likelihood (NLL) across all N_{T} text concepts. Causal Mask. As mentioned above, we construct a learnable mask \mathbf{M}\in\mathbb{R}^{N_{T}\times N_{I}} , where \mathbf{M}_{ij}=\sigma(w_{ij}) , \sigma(\cdot) means sigmoid function. We apply the sparsity constraint to learn this mask, i.e. \mathcal{L}_{s}=\sum_{ij}\|\mathbf{M}_{ij}\|_{1} . Prior Model. We employ an unconditional Diffusion Transformer (DiT) to model the prior distribution of image concepts p^{\mathcal{D}}(z^{I}) . This model treats the concept set \mathbf{Z}^{I}\in\mathbb{R}^{N_{I}\times D_{I}} as a sequence of tokens ( D_{I} is image concepts dimension). Backbone. For the generation model from image concepts z^{I} to true images, we implement our method based on SANA ( 32 ) , similar to Aligner ( 34 ) . Datasets. We use the dataset similar to Aligner, since we want to keep the leanred latnet concepts meaningful. We exclude the EMU-Edit dataset. D.2.2 More Experiment Results We show some more results of our SEDGE method on image generation, compared to SANA and Aligner, as shown in Fig. 10 . Also, to show the generation consistency of our model, we also include Fig. 11 for analysis. a bird is eating ice cream a crocodile with a lion’s mane a dolphin with eagle wings a shopping cart with tank treads SANA Aligner SEDGE Figure 10 : More qualitative comparisons under different prompts. Aligner Our results across different guidance values ( \eta\in\{0.1,0.3,0.5,0.7,1.0,1.5\} ) Figure 11 : Consistency of our generation. Given prompt a camera with insect atennae , we generated figures using both ours and concept aligner models acorss 8 different seeds. As we can see in the figure, the generation of concept aligner has mixed up concepts like legs and bodies. Our generation results (especially with larger guidance values) makes consistent generation.