Couple to Control: Joint Initial Noise Design in Diffusion Models

Jing Jia, Liyue Shen, Guanyang Wang · 2026

Read Couple to Control: Joint Initial Noise Design in Diffusion Models by Jing Jia, Liyue Shen, Guanyang Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11311.txt.

Introduction

Jing Jia, Liyue Shen and Guanyang Wang argue that diffusion image models need not start each picture from independent noise. Each noise can stay marginally standard Gaussian, so a trained model sees the same single-picture input, while dependence across pictures is designed. Coupled noise improves gallery diversity on its own at no extra sampling cost. Repulsive Gaussian coupling raises diversity on three Stable Diffusion versions while largely preserving prompt fit and quality.

Catalog record

CREATORJing Jia, Liyue Shen, Guanyang Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74850
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11311

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Couple to Control: Joint Initial Noise Design in Diffusion Models Jing Jia, Liyue Shen, Guanyang Wang Abstract Diffusion models typically generate image batches from independent Gaussian initial noises. We argue that this independence assumption is only one choice within a broader class of valid joint noise designs. Instead, one can specify a coupling of the initial noises: each noise remains marginally standard Gaussian, so the pretrained diffusion model receives the same single-sample input distribution, while the dependence across samples is chosen by design. This reframes initial-noise control from selecting or optimizing individual seeds to designing the dependence structure of a multi-sample gallery. This view gives a general framework for initial-noise design, covering several existing methods as special cases and leading naturally to new coupled-noise constructions. Coupled noise can improve generation on its own without adding sampling cost, and it is flexible enough to serve as a structured initialization for optimization-based pipelines when additional computation is available. Empirically, repulsive Gaussian coupling improves gallery diversity on SD1.5, SDXL, and SD3 while largely preserving prompt alignment and image quality. It matches or outperforms recent test-time noise-optimization baselines on several diversity metrics at the same sampling cost as independent generation. Subspace couplings also support fixed-object background generation, producing diverse, natural backgrounds compared with specialized inpainting baselines, with a tunable trade-off in foreground fidelity. . 1 Introduction Modern diffusion models have achieved remarkable photorealism, but their practical value increasingly depends on two goals: quality , producing realistic and prompt-faithful images, and control , following user-specified structure across generated outputs. The quality objective has been extensively studied and engineered. Guidance mechanisms such as classifier guidance ( 7 ) , classifier-free guidance ( 13 ) , and CLIP guidance ( 32 ) provide effective ways to steer samples toward high-quality, semantically aligned outputs. Control is more subtle, however, since it can reflect different user intentions. In a standard text-to-image (T2I) generation task, a user may provide a single prompt and want a diverse set of plausible images; in this case, the desired control is over the diversity of the generated gallery. In other settings, such as e-commerce or product editing, the user may instead want to preserve a specific object while varying its surrounding context. Here, the desired control is to keep the product faithful to the input image while generating natural backgrounds that blend smoothly with the preserved object. A natural way for controlling the output of a diffusion model is to control its initial noise. Recent studies suggest that initial noise affects generated images in structured and partly predictable ways. 39 show that different initial noises can systematically affect output quality, layout, and object position. 2 identify “trigger patches” in initial noise that help predict object locations in generated images. 34 report a highly linear relationship between perturbations to the initial noise and changes in the output. 15 show that opposite noise pairs yield significantly negatively correlated outputs. In parallel, a growing body of inference-time methods optimizes or selects initial noises to improve alignment, aesthetics, diversity, or inverse problem performance. These works show that initial noise is not just a source of randomness; it is an effective test-time control variable. However, existing methods treat the noises in a generated batch as separate objects. Each noise may be selected, ranked, or optimized independently. This leaves open a basic question: before any per-sample optimization is performed, can we design the dependence structure among the initial noises themselves? We answer this question through a coupling framework for initial-noise design. A coupling specifies a joint distribution over a batch of random variables while preserving their marginals. In the diffusion setting, this means constructing a batch of noises (Z_{1},\ldots,Z_{K}) such that each Z_{i} remains standard Gaussian, but the noises need not be independent. The pretrained diffusion model therefore receives the same single-sample input distribution as usual, while the user can choose how samples are related. This view turns coupling into a flexible control layer underneath existing initial-noise methods. It naturally supports three modes of use. First, coupled noise can be used directly as a plug-in that replaces independent noise at the start of sampling, with no change to the pretrained model, prompt, sampler, guidance rule, or denoising trajectory. Second, coupled noise can serve as an initialization for direct noise optimization since our framework allows the same optimizer to start from a structured coupling instead of from independent initialization. Third, one may perform amortized optimization of the coupling , where the coupling is represented by a matrix in a Gaussian parameterization and optimized under marginal-preserving constraints. Once learned, such a coupling can be reused for new prompts by simply sampling from it, adding no extra denoising or test-time optimization cost. Our framework naturally generalizes the antithetic sampling method of ( 15 ) , which corresponds to our specific case for K=2 and (Z_{1},Z_{2})=(Z,-Z) . This broader perspective allows us to scale to arbitrary K and unlocks new usages, yielding (i) it extends to arbitrary K , introducing a K -sample repulsive coupling that is minimax optimal for worst-pair correlation; (ii) it enables new applications via subspace and learned couplings for structured generative tasks (e.g., fixed-object backgrounds); and (iii) it naturally fits into noise optimization pipelines as an initialization step, which is unaddressed by 15 . Our contributions are as follows. First, we introduce coupled initial-noise design as a marginal-preserving batch-level control layer for diffusion generation. Unlike standard seed selection or noise optimization, which acts on realized noises, our framework designs the joint distribution from which the noises are drawn. This makes coupling compatible with three common use cases: zero-cost sampling, structured initialization for test-time noise optimization, and amortized learning of reusable coupling matrices. Second, we study Gaussian couplings in this framework. We identify useful special cases, including identical, independent, antithetic, and repulsive couplings; give a minimax characterization of the repulsive coupling; and develop a matrix parameterization for constructing and learning couplings under marginal-preserving constraints. Third, we show that this design choice has practical value. On diverse text-to-image generation with SD1.5, SDXL, and SD3, repulsive coupling matches or exceeds recent test-time optimization baselines on several diversity metrics with no extra denoising or test-time optimization cost, while largely preserving prompt alignment and image quality. When combined with contrastive noise optimization, it often gives further gains. For fixed-object background generation, subspace coupling supports a different form of structured control, preserving the object while varying the background. 2 Related Work Initial noise: As the input to a diffusion model, initial noise has been studied both as a lens for understanding the input–output mechanism and as a lever for improving generation. Besides the mechanism studies mentioned in Section 1 , optimizing initial noise has recently attracted interest for its potential to support inference-time scaling ( 25 ) . These techniques have been shown to improve performance across many applications, including text-image alignment, aesthetics, diversity, and inverse problem solving; see ( 11 ; 8 ; 31 ; 3 ; 36 ; 42 ; 5 ; 38 ; 37 ; 16 ) . Coupling constructs random variables on the same probability space with prescribed marginal distributions, while allowing them to be dependent. It has played an important role in probability and statistics, especially in the study of Markov

Read the complete text