Incorporating Inductive Biases to Energy-based Generative Models
Yukun Li [email protected], Li-Ping Liu [email protected] · 2025
Read Incorporating Inductive Biases to Energy-based Generative Models by Yukun Li [email protected], Li-Ping Liu [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01111.txt.
Introduction
Yukun Li and Li-Ping Liu add built-in bias to energy-based generative models. Their 2025 paper joins a fresh wave of interest in such models, trained by score matching and sampled by Langevin steps. Neural energy functions are flexible but start from zero knowledge. Their hybrid bolts on a fixed statistic function, borrowed from exponential family models.
Training then pushes the model's statistics toward the data's, even without exact likelihood heights. That match lets users place direct constraints on what the model must respect. Tests show the hybrid fits and generates better when the added statistics truly inform the data. The authors offer a way to teach generative models what matters before they learn.
Catalog record
| CREATOR | Yukun Li [email protected], Li-Ping Liu [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01111
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Incorporating Inductive Biases to Energy-based Generative Models Yukun Li [email protected], Li-Ping Liu [email protected] Abstract With the advent of score-matching techniques for model training and Langevin dynamics for sample generation, energy-based models (EBMs) have gained renewed interest as generative models. Recent EBMs usually use neural networks to define their energy functions. In this work, we introduce a novel hybrid approach that combines an EBM with an exponential family model to incorporate inductive bias into data modeling. Specifically, we augment the energy term with a parameter-free statistic function to help the model capture key data statistics. Like an exponential family model, the hybrid model aims to align the distribution statistics with data statistics during model training, even when it only approximately maximizes the data likelihood. This property enables us to impose constraints on the hybrid model. Our empirical study validates the hybrid model’s ability to match statistics. Furthermore, experimental results show that data fitting and generation improve when suitable informative statistics are incorporated into the hybrid model. Reviewed on OpenReview: https://openreview.net/forum?id=k98ZDblyhN 1 Introduction Energy-based models (EBMs) ( Murphy, 2012 ; LeCun et al., 2006 ; Ngiam et al., 2011 ) , which provide a flexible way of parameterization, are widely used in data modeling such as sensitive estimation ( Nguyen & Reiter, 2015 ; Wenliang et al., 2019 ; Jiang & Xiao, 2021 ) , structured prediction ( Belanger & McCallum, 2016 ; He & Jiang, 2020 ; Pan et al., 2020 ) , and anomaly detection ( Zhai et al., 2016 ; Liu et al., 2020 ) . They recently have achieved successes as generative models for data of different modalities, such as images ( Du & Mordatch, 2019 ; Vahdat & Kautz, 2020 ) , texts ( Deng et al., 2020 ; Yu et al., 2022 ) , graph ( Liu et al., 2021 ; Chen et al., 2022 ; Xu et al., 2022 ) and point cloud ( Xie et al., 2020 ; Xie et al., 2021 ; Luo & Hu, 2021 ) , thanks to the advent of new training methods of score matching ( Song & Ermon, 2019 ; Song et al., 2020b ; Ho et al., 2020 ) . Previous EBMs often have linear energy functions in model parameters, so they fall into the category of exponential-family models ( Wainwright et al., 2008 ) . An exponential family model has a few nice properties. It matches data statics when it maximizes the likelihood of the training data. Among all models, it also has the largest entropy with the constraint of matching statistics. When constructing an EBM, statistic functions are often used to capture key statistics of the data. However, in recent years, EBMs have used neural networks as their energy functions. Since their energy functions are not linear in trainable parameters, they are not exponential-family models anymore. In our construction of EBMs, we propose to include linear terms defined with special statistic functions in the energy function to retrieve some good properties of exponential-family models. Even with complex data types such as point clouds and graphs, we still often have domain knowledge to specify statistic functions so that the model can capture desired properties. Therefore, the linear term with special statistic functions allows us to inject inductive biases in model fitting. This framework applies to the domains where prior information can be written in statistic functions. The proposed method is very useful for data fitting in multiple iterations. Note that data fitting is often an iterative procedure ( Gelman et al., 1995 ) , and the goodness of fit is often monitored by the discrepancy between statistics computed respectively from model samples and real data. With our method, we can improve the model by adding corresponding terms to narrow the discrepancy. Taking molecule generation as an example, when we observe molecule samples violate the valency constraint, we can incorporate the knowledge of valency rules to improve the model. We evaluate the effectiveness of this new approach on three tasks: modeling molecular data, fitting handwritten digits, and modeling point clouds. The results demonstrate the effectiveness of the proposed approach in a wide range of data-fitting domains. 2 Related Work The need to incorporate inductive bias into a generative model has drawn researchers’ attention for a long time. Zhu et al. (1997) proposes the general principle for incorporating prior knowledge into a generative model. Khalifa et al. (2020) introduces a novel method for text generation using EBMs to enforce desired statistical constraints. Their framework employs the EBM model to approximate a pre-trained language model instead of directly fitting the data. Korbak et al. (2021) extends this approach to code generation, imposing a constraint that generated sequences are compilable. Qin et al. (2022) applies energy constraints to a transformer decoder to control text semantics and style. Lafon et al. (2023) learns the hybrid model combined with the Gaussian Mixture Model (GMM) and EBMs for out-of-distribution detection. In contrast, our work focuses on improving data fitting with inductive bias. Another thread of research changes a generative model’s behavior during the sampling stage ( Dhariwal & Nichol, 2021 ) . For example, Chung et al. (2022) proposes to use manifold constraint to guide a pre-trained diffusion model’s sampling path. Kim et al. (2022) ; Zhao et al. (2022) ; Song et al. (2022) ; Yu et al. (2023) ; Liu et al. (2023) ; Bansal et al. (2023) ; Li & Liu (2024) propose different guidance for different specific problems to boost the performance of each task. In contrast, our method aims to fit the original data distribution more accurately by adding the statistics term. Notably, the classifier-guidance method operates during inference with manually tuned weights, whereas our model learns the inductive bias strength corresponding to data distribution statistics. 3 Background 3.1 Score-matching for energy-based models An energy-based model is defined with an energy function E_{\mathbold{\theta}} parameterized by \mathbold{\theta} , \displaystyle p_{\mathbold{\theta}}(\mathbf{x})=\frac{\exp(-E_{\mathbold{\theta}}(\mathbf{x}))}{Z_{\mathbold{\theta}}}. (1) Here, \mathbf{x}\in\mathbf{R}^{d} , d is the feature dimension. Z_{\mathbold{\theta}} is the partition function: Z_{\mathbold{\theta}}=\int\exp(-E_{\mathbold{\theta}}(\mathbf{x}))d\mathbf{x} , which is usually intractable. Various methods exist to train an energy-based model. Maximum likelihood training with Markov Chain Monte Carlo (MCMC) sampling ( Hinton, 2002 ) is one standard method to train the Energy-based model, but MCMC is typically computationally expensive. A modern EBM often devises the energy function E{\theta}(\mathbf{x}) with a neural network, and \mathbold{\theta} represents learnable parameters of the neural network ( Song & Kingma, 2021 ) . These models are usually fit by score matching ( Hyvärinen & Dayan, 2005 ; Kingma & Cun, 2010 ) , which avoids the explicit calculation of the Z_{\mathbold{\theta}} . Score matching minimizes the mean square error between the model score function \mathbf{s}_{\mathbold{\theta}}(\mathbf{x})=\nabla_{\mathbf{x}}\log p_{\mathbold{\theta}}(\mathbf{x}) and the data score function \nabla_{\mathbf{x}}\log p_{data}(\mathbf{x}) : D_{F}(p_{data}(\mathbf{x})||p_{\mathbold{\theta}}(\mathbf{x}))=E_{p_{data}(\mathbf{x})}\left[\frac{1}{2}||\nabla_{\mathbf{x}}\log p_{data}(\mathbf{x})-\nabla_{\mathbf{x}}\log p_{\mathbold{\theta}}(\mathbf{x})||_{2}^{2}\right]. (2) where p_{data}(\mathbf{x}) is the true data density function, p_{\mathbold{\theta}}(\mathbf{x}) is the model density function. To bypass the intractable of the first term, we often use integration by parts to rewrite it as : D_{F}(p_{data}(\mathbf{x})||p_{\mathbold{\theta}}(\mathbf{x}))=E_{p_{data}(\mathbf{x})}\left[\frac{1}{2}\sum_{i=1}^{d}\left(\frac{\partial E_{\mathbold{\theta}}(\mathbf{x})}{\partial x^{i}}\right)^{2}+\frac{\partial^{2}E{\theta}(\mathbf{x})}{(\partial x^{i})^{2}}\right]+const. (3) Several efficient variants of the score matching have been proposed. One is Sliced Score Matching ( Song et al., 2020a ) , which randomly samples a projection vector \mathbf{v} , takes the inner product between \mathbf{v} and the two scores,