Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Models
Jisung Hwang, Jaihoon Kim, Minhyuk Sung · 2025
Read Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Models by Jisung Hwang, Jaihoon Kim, Minhyuk Sung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07027.txt.
Introduction
Jisung Hwang, Jaihoon Kim, and Minhyuk Sung propose a new regularization for text-to-image models in a 2025 paper, based on Gaussianity of moments and power spectra. Their loss pushes samples toward a standard Gaussian shape. When samples follow that shape, later work in the model's latent space, where images are adjusted, becomes easier.
The method treats each entry of a high-dimensional sample as if it were a one-dimensional standard Gaussian variable. Its combined loss pairs a moment-based term, acting in the spatial domain, with a power-spectrum term in the spectral domain. The target values for both parts can be worked out in advance, so the loss rewards matching them. Inputs are shuffled at random so order does not matter. The authors also show that earlier Gaussianity methods sit inside their single unified framework.
Catalog record
| CREATOR | Jisung Hwang, Jaihoon Kim, Minhyuk Sung |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07027
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Models Jisung Hwang, Jaihoon Kim, Minhyuk Sung Abstract We propose a novel regularization loss that enforces standard Gaussianity, encouraging samples to align with a standard Gaussian distribution. This facilitates a range of downstream tasks involving optimization in the latent space of text-to-image models. We treat elements of a high-dimensional sample as one-dimensional standard Gaussian variables and define a composite loss that combines moment-based regularization in the spatial domain with power spectrum-based regularization in the spectral domain. Since the expected values of moments and power spectrum distributions are analytically known, the loss promotes conformity to these properties. To ensure permutation invariance, the losses are applied to randomly permuted inputs. Notably, existing Gaussianity-based regularizations fall within our unified framework: some correspond to moment losses of specific orders, while the previous covariance-matching loss is equivalent to our spectral loss but incurs higher time complexity due to its spatial-domain computation. We showcase the application of our regularization in generative modeling for test-time reward alignment with a text-to-image model, specifically to enhance aesthetics and text alignment. Our regularization outperforms previous Gaussianity regularization, effectively prevents reward hacking and accelerates convergence. 1 Introduction The Gaussian distribution plays a central role in numerous machine learning applications, widely used to model measurement noise, uncertainty, and the mean of independent samples. Its mathematical simplicity and analytical tractability have made it a default modeling choice in many contexts. In particular, within generative modeling , the standard Gaussian is commonly used as the latent distribution mapped to complex data distributions. Given the pervasive presence of Gaussianity not only in machine learning but also across the broader sciences, quantifying how closely a data point follows—or deviates from—a Gaussian distribution, i.e., measuring Gaussianity , has become a fundamental technique an1933sulla ; smirnov1948table ; anderson1954test ; cramar1928composition ; lilliefors1967kolmogorov ; shapiro1965analysis ; gnanadesikan1968probability ; shapiro1972approximate ; jarque1980efficient ; d1990suggestion ; anscombe1983distribution . In generative modeling in particular, measuring Gaussianity can facilitate optimization by identifying the latent Gaussian variable that best maps to a desired data point, thereby enabling more precise and controllable generation. For example, in the context of widely used text-to-image models flux2024 ; esser-sd3 ; chen2024pixartsigma ; liu2024instaflow ; podell2023sdxl , prior work has shown that Gaussianity-based regularization can improve downstream tasks such as aesthetic image generation tang2024tuning and text-image alignment eyring2024reno . In this work, we study the structural properties of the standard Gaussian distribution and propose a novel regularization loss that unifies various existing approaches under a unified theoretical framework. Leveraging the identity covariance of the standard Gaussian, we treat a high-dimensional sample as a collection of i.i.d. scalar standard Gaussian variables—that is, as D samples drawn from a one-dimensional standard Gaussian. With a specific ordering, we analyze the distribution of these scalar values in both the one-dimensional spatial and spectral domains. In the spatial domain, we utilize the moments of the samples—the expected values of their powers—and define a moment-based regularization loss. Notably, previously proposed losses such as norm-based samuel2023norm ; eyring2024reno , kurtosis-based chmiel2020robust , and KL-divergence-based kingma2013auto regularization can be interpreted as, or shown to be asymptotically equivalent to, specific instances of this moment-based formulation. Spatial-domain regularization alone is often insufficient in generative modeling, as it can leave residual patterns that lead to mappings to unrealistic data points (Figure 1 (c)). To address this, we further analyze the samples in the spectral domain and introduce an additional regularization loss based on the fact that the power spectrum of i.i.d. Gaussian samples follows a chi-square distribution. This corresponds to fitting the sample covariance to the identity matrix in the spatial domain, aligning with prior work tang2024tuning . However, by operating in the spectral domain, we reduce the computational complexity from \mathcal{O}(D^{2}) to \mathcal{O}(D\log D) , thereby eliminating the need for dimension-wise sampling. Crucially, because both spatial and spectral regularization should hold regardless of the ordering of elements in a high-dimensional Gaussian sample, we enforce permutation invariance by applying our losses to randomly permuted versions of the input. As applications of Gaussianity regularization, inspired by Eyring et al. eyring2024reno , we showcase test-time reward alignments using a one-step text-to-image model flux2024 , specifically for enhancing image aesthetics and text alignment. Reward alignment refers to the task of optimizing the latent sample of a generative model so that the resulting output maximizes a given reward function, such as aesthetic quality or textual alignment. However, directly optimizing the latent can lead to overfitting to the reward signal—known as reward hacking—which often degrades output quality (e.g., reduced image realism). Our Gaussianity regularization effectively mitigates this issue by encouraging the optimized latents to remain close to the original Gaussian prior. Across both aesthetic and text alignment tasks, our method outperforms existing regularization approaches, achieving the highest scores across all metrics while preventing reward hacking and accelerating convergence. 2 Related Work 2.1 Gaussianity Testing and Regularization The Gaussian distribution is a fundamental component in statistics, and numerous methods have been developed to assess whether a set of samples conforms to it. Classical tests such as K-S, A-D, and CvM an1933sulla ; smirnov1948table ; anderson1954test ; cramar1928composition ; lilliefors1967kolmogorov measure discrepancies between empirical and theoretical cumulative distribution functions. Other approaches assess Gaussianity using order statistics shapiro1965analysis or quantile alignment gnanadesikan1968probability ; shapiro1972approximate . These methods, however, are based on non-differentiable functions and are therefore limited in their applicability to optimization-based techniques in machine learning. There have also been differentiable approaches applied as regularization across a range of machine learning applications. A classical example is the KL divergence used in variational autoencoders (VAEs) kingma2013auto to measure the difference between the approximate posterior and a standard Gaussian prior—though this operates on distributional parameters rather than directly on samples. Chmiel et al. chmiel2020robust introduced a kurtosis-based regularizer that penalizes heavy-tailed behavior in model weights to improve training stability. More recently, norm-based regularization samuel2023norm ; eyring2024reno ; Benhamu:2024DFlow has been used to constrain latent vectors to lie on a hyperspherical shell, matching the expected norm of a unit Gaussian. However, these methods only address marginal statistics and do not account for inter-component dependencies. We propose a unified approach that incorporates these approaches as components, with the relationships detailed in Section 3.1.3 . Probability-Regularized Noise Optimization (PRNO) tang2024tuning is a notable example that aligns the empirical covariance matrix of latent samples with the identity, thereby capturing inter-component dependencies. While effective, this method incurs quadratic memory and time complexity, making it less scalable in high-dimensional settings. In our unified framework, we propose a more efficient regularization that achieves the same objective but in the spectral domain, with details provided in Section 3.2.2 and Appendix C . 2.2 Reward Alignment via Latent Noise Optimization Reward alignment refers to the process of steering generative models to produce outputs that maximize a given reward function. While previous approaches using direct fine-tuning Prabhudesai2023:AlignProp ; Clark2024:DRaFT or reinforcement learning Black2024:DDPO ; Wallace:2024DiffusionDPO ; Yang2024:D3PO have been widely studied, these methods are computationally expensive and require retraining the model for each new reward—posing significant scalability challenges. In contrast, noise optimization offers an efficient alternative by directly optimizing the initial Gaussian noise at inference time, improving the outputs of pretrained generative models without additional training. Due to its effectiveness, noise optimization has