Cosine-Similarity Methods for Efficient Training and Sampling in High-Dimensional Latent Spaces

Xu Duan, Dongmei Chen · 2025

Read Cosine-Similarity Methods for Efficient Training and Sampling in High-Dimensional Latent Spaces by Xu Duan, Dongmei Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00684.txt.

Introduction

Xu Duan and Dongmei Chen wrote a 2025 machine learning paper on latent spaces. Generative models often lift images into high-dimensional feature spaces, where traits like shape or style become easier to separate. Newer designs, such as representation autoencoders, lean on spaces that line up with meaning.

Duan and Chen argue that these spaces hold patterns that older methods miss. In particular, the directions between features carry useful structure. Their cosine-similarity methods use the angles between feature vectors for training and for sampling new outputs. Using direction well could make training cheaper and samples better in very high dimensions.

Catalog record

CREATORXu Duan, Dongmei Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00684

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cosine-Similarity Methods for Efficient Training and Sampling in High-Dimensional Latent Spaces Xu Duan, Dongmei Chen Abstract Latent generative models are increasingly shifting from traditional VAEs toward representation autoencoders and semantically aligned latent spaces, which lift images into higher-dimensional feature domains where semantic factors become more separable. Yet these spaces also contain geometric regularities that existing methods do not fully exploit—particularly in the directional relationships between features. We introduce a cosine-similarity–based mechanism that improves both training and sampling by selecting couplings that produce cleaner, less entangled velocity fields. This simple alignment reduces gradient noise, accelerates convergence, and improves sample fidelity. Building on this idea, we develop cosine-similarity–based fine-tuning and time-scheduling strategies that reduce the FID of an 800-epoch RAE from 11.99 to 8.60. Furthermore, by formulating an optimal-transport coupling using a cosine cost, a single-epoch fine-tuning step at the 20-epoch checkpoint reaches 3.30 FID—matching the performance of the 80-epoch baseline. 1 Introduction Pixel-space diffusion models remain computationally demanding, motivating the shift toward compact latent-space generation. Models such as Latent Diffusion Models (LDMs) [ 23 ] and Diffusion/Flow Transformers (DiT/SiT) [ 21 , 19 ] demonstrate that learning and sampling in a low-dimensional latent space yields higher visual fidelity and efficiency compared to pixel-space diffusion. Concurrently, advances in self-supervised representation learning—including DINO, DINOv2, MAE, and MOCOV3 [ 1 , 20 , 9 , 4 ] —produce encoders whose feature spaces exhibit robust semantic structure and strong generalization. These learned representations naturally raise the question of whether generative models should incorporate such semantics into their latent spaces. Recent work increasingly shows that generation benefits from semantically constrained latent variables. Yao et al. [ 39 ] highlight that generation in unconstrained VAE latents leads to semantic drift, while REPA-style methods [ 42 , 14 ] explicitly align generative outputs with self-supervised visual features to improve consistency and reduce distortions. Other approaches adapt the generative noise itself. DNA-Edit [ 36 ] optimizes noise vectors along semantic directions to improve realism. Tong et al. [ 31 ] further formalize noise design as an optimal-transport problem, aligning data with structured Gaussian priors to produce smoother training dynamics. Li et al. propose Immiscible Diffusion [ 16 ] , arguing that fully mixing all images across the noise space complicates denoising. Their assignment-then-diffusion strategy pairs each image with nearby noise before diffusion, reducing noise–data entanglement. This simple modification preserves Gaussian structure while significantly accelerating training and improving fidelity. Despite recent progress, high-dimensional representation spaces still contain rich geometric structure that generative models rarely capitalize on. We develop a cosine-similarity framework that exploits this structure by coupling training targets and sampling trajectories according to their directional agreement in feature space. This alignment not only accelerates convergence but also improves sample fidelity. Additionally, we introduce cosine-driven scheduling strategies—applicable during both training and ODE-based generation—that adapt step sizes based on semantic proximity. Finally, we construct an optimal-transport formulation using a cosine cost, enabling a one-epoch fine-tuning procedure that pushes a low-epoch checkpoint to near-saturated performance. Together, these techniques illustrate that carefully leveraging semantic geometry can yield substantial gains in generative modeling—without modifying architectures or increasing inference complexity. Figure 1 : Scratch of High Dimension Representation Space Generation 2 Related Work Generative models for image generation. Classical diffusion models such as DDPM [ 11 ] and DDIM [ 26 ] perform progressive noise removal over hundreds or thousands of steps, resulting in substantial computational cost during both training and inference. Latent diffusion models (LDMs) [ 23 ] mitigate this by operating in a compressed latent space using pretrained autoencoders, while transformer-based architectures—including DiT [ 21 ] and SiT [ 19 ] —enhance global context modeling via self-attention mechanisms. Despite these architectural advances, diffusion training remains slow due to the need to learn denoising scores across an entire noise schedule. A large body of work seeks to accelerate diffusion through improved sampling or revised training procedures. Sampling-side acceleration includes higher-order solvers [ 18 , 17 , 44 ] , consistency distillation [ 27 ] , and one-step or few-step student models [ 8 , 41 ] , though these often require expensive teacher models or can introduce instability. Training-side acceleration employs masked or partial-timestep training [ 13 , 46 ] , multi-resolution optimization [ 40 ] , or architectural modifications, but these typically increase implementation complexity. Other approaches compress the latent space through quantization or semantic bottlenecks [ 37 ] , sometimes at the cost of generation fidelity. Image generation with semantic representations. Recent work explores coupling generative models with pretrained representation encoders. VA-VAE [ 39 ] aligns VAE latents with semantic features, and MAETok [ 2 ] , DC-AE 1.5 [ 3 ] , and l-DeTok [ 38 ] incorporate MAE- or DAE-style objectives [ 32 ] into VAE training. While alignment improves both reconstruction and sample quality, the reliance on heavily compressed, low-dimensional latents limits fidelity and the richness of learned representations. In contrast, we reconstruct directly from representation-encoder features without additional compression, and we show that a simple ViT-based decoder on top of frozen features achieves reconstruction performance comparable to or exceeding SD-VAE [ 23 ] while preserving significantly stronger representations. Representation for generation. A parallel line of work investigates using semantic representations to improve generative modeling itself. REPA [ 42 ] accelerates DiT convergence by aligning its middle block with representation-encoder features; REPA-E [ 14 ] extends this idea by also unfreezing the VAE. DDT [ 34 ] further improves convergence by decoupling DiT into an encoder–decoder structure and applying REPA loss to the encoder output. REG [ 35 ] introduces a learnable token into the DiT sequence and aligns it with semantic features, while ReDi [ 12 ] jointly models VAE latents and PCA-compressed DINOv2 features within a diffusion framework. Orthogonally, [ 33 ] leverage contrastive-learning-based loss designs to improve image quality in a plug-and-play fashion without modifying the network architecture. 3 Background Our work builds on RAE [ 45 ] . We begin by introducing the relevant preliminaries. Flow and diffusion models both leverage stochastic processes to gradually transform Gaussian noise \epsilon\sim\mathcal{N}(0,I) into data samples \mathbf{x}_{*} . This process can be unified as \mathbf{x}_{t}=\alpha_{t}\mathbf{x}_{*}+\sigma_{t}\epsilon, (1) where \alpha_{t} is a decreasing and \sigma_{t} an increasing function of time t . Flow-based models typically interpolate between noise and data over a finite interval, while diffusion models define a forward stochastic differential equation (SDE) that converges to a Gaussian distribution as t\rightarrow\infty . Sampling from these models can be achieved via either a reverse-time SDE or a probability flow ordinary differential equation (ODE), both of which yield the same marginal distributions for \mathbf{x}_{t} . The probability flow ODE is: \dot{\mathbf{x}}_{t}=\mathbf{v}(\mathbf{x}_{t},t), (2) where the velocity field \mathbf{v}(\mathbf{x},t) can be formulated by the conditional expectation: \mathbf{v}(\mathbf{x},t)=\mathbb{E}[\dot{\mathbf{x}}_{t}\mid\mathbf{x}_{t}=\mathbf{x}]=\dot{\alpha}_{t}\mathbb{E}[\mathbf{x}_{*}\mid\mathbf{x}_{t}=\mathbf{x}]+\dot{\sigma}_{t}\mathbb{E}[\epsilon\mid\mathbf{x}_{t}=\mathbf{x}]. (3) To synthesize data, we can integrate Eqn. (3) in reverse time, initializing from \mathbf{X}_{T}=\epsilon where \epsilon\sim\mathcal{N}(0,I) . This process yields samples from p_{0}(\mathbf{x}) , serving as an approximation to the true data distribution p(\mathbf{x}) . This velocity can be estimated by a model \mathbf{v}_{\theta}(\mathbf{x}_{t},t) , which is trained to minimize the following loss function: \mathcal{L}_{v}(\theta)=\int_{0}^{T}\mathbb{E}\!\left[\left\|\mathbf{v}_{\theta}(\mathbf{x}_{t},t)-\dot{\alpha}_{t}\mathbf{x}_{*}-\dot{\sigma}_{t}\epsilon\right\|^{2}\right]dt. (4) The reverse-time SDE can describe the probability distribution p_{t}(\mathbf{x}) of \mathbf{x}_{t} at time t , which can be expressed as: d\mathbf{x}_{t}=\mathbf{v}(\mathbf{x}_{t},t)dt-\frac{1}{2}w_{t}\mathbf{s}(\mathbf{x}_{t},t)dt+\sqrt{w_{t}}\,d\overline{\mathbf{W}}_{t}, (5) with \mathbf{s}(\mathbf{x},t) denoting the score that can be

Read the complete text