Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation

Zhijun Zeng, Yixuan Jiang, Pipi Hu, Zuoqiang Shi · 2026

Read Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation by Zhijun Zeng, Yixuan Jiang, Pipi Hu, Zuoqiang Shi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09085.txt.

Introduction

Zhijun Zeng and three co-authors seek one method for estimating probability patterns across very different kinds of data. Continuous values, discrete choices, and mixes of the two are usually handled by separate tools, even though the underlying task is the same.

Their framework, called Constant-Target Energy Matching, learns an energy score for each possible state. Instead of chasing density ratios, which can blow up for rare outcomes, it trains toward a fixed target of one using a bounded transformation. The learned score recovers the log-probability without extra normalization steps. Across tests on continuous, discrete, and mixed data, the method beat strong rivals at density estimation and produced better samples under standard sampling routines.

Catalog record

CREATORZhijun Zeng, Yixuan Jiang, Pipi Hu, Zuoqiang Shi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09085

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Constant-Target Energy Matching: A Unified Framework for Continuous and Discrete Density Estimation Zhijun Zeng, Yixuan Jiang, Pipi Hu, Zuoqiang Shi Abstract Density estimation is a central primitive in probabilistic modeling, yet continuous, discrete, and mixed-variable domains are often treated by separate objectives, limiting the ability to exploit a common statistical structure across data types. Continuous score-based methods rely on log-density gradients, while discrete extensions typically use concrete score whose unbounded targets become unstable near low-probability states. We introduce Constant-Target Energy Matching (CTEM), a unified energy-based framework for density estimation on general state spaces. CTEM replaces ordinary density-ratio regression with a bounded energy-difference transform and derives from it a sample-only training objective with the constant target 1 1 . The learned scalar potential recovers log ⁡ ρ \log\rho without partition-function estimation or explicit unbounded ratio regression. Across continuous, discrete, and mixed-variable benchmarks, CTEM substantially improves density estimation over competitive baselines and yields higher-quality samples under standard sampling procedures. 1 Introduction Density estimation is a fundamental primitive in probabilistic modeling, sampling, inference, and uncertainty quantification. Classical kernel estimators ( 25 ; 29 ) , energy-based models ( 15 ; 5 ) , normalizing flows ( 24 ; 4 ; 23 ; 14 ; 9 ) , and score-based generative models ( 11 ; 34 ; 32 ; 31 ; 10 ; 33 ; 13 ) provide complementary ways to represent and learn probability distributions. Despite this progress, most modern objectives remain closely tied to the type of state space on which the distribution is defined: continuous methods exploit differential structure through scores, while discrete methods typically rely on finite differences, probability ratios, or denoising surrogates over neighboring states. This separation is especially visible in score-based modeling. In continuous spaces, score matching avoids partition-function estimation by learning \nabla\log\rho instead of the normalized density itself ( 11 ; 34 ; 32 ) . This idea underlies modern diffusion and score-based generative models ( 31 ; 10 ; 33 ) . However, the score \nabla\log\rho is not defined on general discrete spaces. Recent discrete score and diffusion methods therefore replace gradients by finite-difference or ratio-based quantities over structured neighborhoods ( 12 ; 1 ; 3 ; 21 ; 19 ; 26 ; 28 ; 8 ; 27 ) . These approaches have enabled substantial progress, but their regression targets often involve ordinary density ratios such as \rho(\tilde{z})/\rho(z) , or conditional surrogates of such ratios. Ordinary ratios are statistically natural but numerically brittle: they are unbounded and become poorly conditioned near low-probability states. This issue is particularly acute on sparse discrete supports and mixed spaces, where many local comparisons involve states with highly unequal probability mass. Existing discrete generative models address this instability through score-entropy objectives ( 19 ) , absorbing-state or masked parameterizations ( 26 ; 28 ; 22 ) , flow-based formulations ( 8 ) , or noise-process designs that connect discrete and Gaussian diffusion ( 27 ) . While effective in their intended settings, these remedies are usually coupled to a particular state-space geometry, corruption process, or reverse-process parameterization. They therefore do not directly provide a single density-estimation objective that applies unchanged across continuous, discrete, and mixed-variable domains. We introduce Constant-Target Energy Matching (CTEM), a unified energy-based framework for density estimation on general state spaces. CTEM learns a scalar potential f_{\theta}\approx\log\rho+C by replacing the ordinary density-ratio target with the bounded modified ratio \frac{\rho(z)-\rho(\tilde{z})}{\rho(z)+\rho(\tilde{z})}=\tanh\!\left(\frac{\log\rho(z)-\log\rho(\tilde{z})}{2}\right). Starting from a density-weighted matching objective over pairs of states, we show that a symmetry argument removes the unknown density terms and yields a sample-only loss with the constant regression target 1 . The same objective applies to continuous variables through local directional or Gaussian comparisons, to discrete variables through graph-based neighborhood comparisons, and to mixed product spaces by combining the two. At the population level, the learned potential recovers \log\rho up to an additive constant on each connected component induced by the comparison structure. Empirically, CTEM performs well across continuous, discrete, and mixed-variable settings. On continuous density-estimation benchmarks, CTEM improves density recovery over kernel estimators and normalizing-flow baselines ( 29 ; 2 ; 4 ; 6 ; 7 ) . On sparse quantized distributions, CTEM reduces total variation and KL divergence relative to concrete score matching and SEDD ( 21 ; 19 ) . On binary MNIST, it produces recognizable samples using a noise-conditional scalar energy and Metropolis–Hastings sampling. On mixed continuous–discrete product spaces, it recovers a globally aligned energy over both Euclidean and categorical coordinates. Contributions. 1. We propose Constant-Target Energy Matching (CTEM), a unified density-estimation framework for continuous, discrete, and mixed state spaces. CTEM uses a single scalar-potential parameterization and a common training principle across heterogeneous variables, providing a direct route to multimodal probabilistic modeling beyond purely continuous or purely discrete domains. 2. We derive a contrastive density-estimation loss from a modified density-ratio identity. The resulting objective is sample-based, applies to both continuous and discrete variables, and avoids explicit regression to unbounded density ratios that become unstable near low-density states. Unlike small-noise score-matching objectives, the population target is not tied to an infinitesimal perturbation limit, so the comparison scale can be chosen for training stability and coverage. 3. We validate CTEM across continuous, discrete, and mixed-variable density estimation tasks. CTEM improves density estimation over competitive kernel, flow, and discrete score-matching baselines; the induced score yields high-quality samples under standard sampling procedures; and the same objective remains effective on mixed continuous–discrete product spaces. 2 Preliminaries Let (\mathcal{Z},\nu) be a measurable space with a \sigma -finite reference measure, covering continuous variables, discrete variables, and their products by choosing \nu as the corresponding Lebesgue, counting, or product reference measure. Let P be the data distribution and assume that P\ll\nu with strictly positive density \rho(z)=\frac{dP}{d\nu}(z). (1) We learn a scalar energy f_{\theta}:\mathcal{Z}\to\mathbb{R} such that f_{\theta}(z)=\log\rho(z)+C, (2) where the additive constant is unidentifiable. In discrete spaces, unless otherwise specified, we take \nu to be the counting measure, so that \rho(z_{i})=P(z_{i}) and ( 2 ) recovers the log-probability mass up to an additive constant. 3 Method We introduce Constant-Target Energy Matching (CTEM), an energy-based framework for density estimation on general state spaces. Existing ratio-based methods, including ratio matching and concrete score matching, often target ordinary density ratios such as \rho(\tilde{z})/\rho(z) . These ratios encode relative probabilities, but they are unbounded and can be numerically unstable when the denominator is small. CTEM instead uses the bounded modified density ratio r_{\rho}(z,\tilde{z}):=\frac{\rho(z)-\rho(\tilde{z})}{\rho(z)+\rho(\tilde{z})}, (3) which is linked to log-density differences by the following identity. Lemma 1 (Bounded modified-ratio identity) . For any z,\tilde{z}\in\mathcal{Z} with \rho(z),\rho(\tilde{z})>0 , \tanh\!\left(\frac{\log\rho(z)-\log\rho(\tilde{z})}{2}\right)=\frac{\rho(z)-\rho(\tilde{z})}{\rho(z)+\rho(\tilde{z})}. (4) Lemma 1 suggests learning f_{\theta} by matching the model-side bounded difference \tanh\!\left(\frac{f_{\theta}(z)-f_{\theta}(\tilde{z})}{2}\right) to r_{\rho}(z,\tilde{z}) . The challenge is that r_{\rho} still depends on the unknown density. CTEM removes this unknown target algebraically and yields a sample-based objective with a constant regression target. 3.1 Constant-target energy matching To specify which pairs of states are compared, let \omega:\mathcal{Z}\times\mathcal{Z}\to\mathbb{R}_{\geq 0} be a non-negative comparison weight. Throughout the main text, we assume that \omega is symmetric, \omega(z,\tilde{z})=\omega(\tilde{z},z) . The weight \omega controls the comparison geometry, such as local perturbations in continuous spaces or neighborhood edges in discrete spaces. We begin with the density-weighted matching objective \mathcal{J}_{\omega}(\theta):=\iint\left[\tanh\!\left(\frac{f_{\theta}(z)-f_{\theta}(\tilde{z})}{2}\right)-\frac{\rho(z)-\rho(\tilde{z})}{\rho(z)+\rho(\tilde{z})}\right]^{2}\bigl(\rho(z)+\rho(\tilde{z})\bigr)\omega(z,\tilde{z})\,\nu(dz)\nu(d\tilde{z}). (5) This objective is not directly computable from samples, since it contains the unknown density \rho both

Read the complete text