Hyper-Transforming Latent Diffusion Models

Ignacio Peis, Batuhan Koyuncu, Isabel Valera, Jes Frellsen · 2025

Read Hyper-Transforming Latent Diffusion Models by Ignacio Peis, Batuhan Koyuncu, Isabel Valera, Jes Frellsen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16580.txt.

Introduction

Ignacio Peis and co-authors introduce hyper-transforming latent diffusion models in a 2025 paper on generating functions, not just images. Their frame joins implicit neural representations, which encode shapes as functions, with transformer hypernetworks.

Older hypernetworks use plain multilayer nets that scale poorly. A transformer decoder here generates the function parameters from latent codes, lifting both capacity and speed. A trick called hyper-transforming tunes only the decoder while freezing a pretrained latent space. That adapts strong models to function outputs without full retraining. Tests across media show better scale, expressiveness, and generalization than past function generators. The work unifies function learning under one flexible frame.

Catalog record

CREATORIgnacio Peis, Batuhan Koyuncu, Isabel Valera, Jes Frellsen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16580

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hyper-Transforming Latent Diffusion Models Ignacio Peis, Batuhan Koyuncu, Isabel Valera, Jes Frellsen Abstract We introduce a novel generative framework for functions by integrating Implicit Neural Representations (INRs) and Transformer-based hypernetworks into latent variable models. Unlike prior approaches that rely on MLP-based hypernetworks with scalability limitations, our method employs a Transformer-based decoder to generate INR parameters from latent variables, addressing both representation capacity and computational efficiency. Our framework extends latent diffusion models (LDMs) to INR generation by replacing standard decoders with a Transformer-based hypernetwork, which can be trained either from scratch or via hyper-transforming —a strategy that fine-tunes only the decoder while freezing the pre-trained latent space. This enables efficient adaptation of existing generative models to INR-based representations without requiring full retraining. We validate our approach across multiple modalities, demonstrating improved scalability, expressiveness, and generalization over existing INR-based generative models. Our findings establish a unified and flexible framework for learning structured function representations. 1 Introduction Generative modelling has seen remarkable advances in recent years, with diffusion models achieving state-of-the-art performance across multiple domains, including images, videos, and 3D synthesis ( Ho et al., 2020 ; Dhariwal & Nichol, 2021 ; Rombach et al., 2022 ) . A key limitation of existing generative frameworks, however, is their reliance on structured output representations, such as pixel grids, which constrain resolution and generalisation across data modalities. In contrast, Implicit Neural Representations (INRs) have emerged as a powerful alternative that parametrises signals as continuous functions ( Sitzmann et al., 2020 ; Mildenhall et al., 2021 ) . By leveraging INRs, generative models can represent complex data distributions at arbitrary resolutions, yet learning expressive distributions over INR parameters remains a fundamental challenge. \times 0.125 \times 0.25 \times 0.5 \times 1 \times 2 \times 4 Figure 1 : Samples from LDMI at multiple resolutions. A common approach to modelling distributions over INRs is to use hypernetworks ( Ha et al., 2017 ) , which generate the weights and biases of an INR conditioned on a latent code ( Dupont et al., 2022b ; Koyuncu et al., 2023 ) . However, MLP-based hypernetworks suffer from scalability bottlenecks, particularly when generating high-dimensional INRs, as direct parameter regression constrains flexibility and expressiveness. More recently, Transformer-based hypernetworks have been proposed to alleviate these issues, introducing attention mechanisms to efficiently predict INR parameters ( Chen & Wang, 2022 ; Zhmoginov et al., 2022 ) . Nonetheless, existing approaches such as Trans-INR ( Chen & Wang, 2022 ) remain deterministic, limiting their applicability in probabilistic frameworks. In this work, we introduce a novel framework for INR generation, named Latent Diffusion Models of INRs ( LDMI ), that incorporates our proposed Hyper-Transformer Decoder ( HD ), a probabilistic Transformer-based decoder for learning distributions over INR parameters. Our approach combines the strengths of hypernetworks with recent advances in meta-learning for INRs while integrating into latent diffusion-based generative frameworks. Unlike previous works, our HD decoder employs a full Transformer architecture, where a Transformer Encoder processes latent variables, and a Transformer Decoder generates INR parameters via cross-attention. This design enables flexible, probabilistic generation of neural representations across diverse data modalities, surpassing the deterministic limitations of prior Transformer-based hypernetworks ( Chen & Wang, 2022 ; Zhmoginov et al., 2022 ) . Our proposed framework, LDMI , supports two training paradigms: (i) full training, where the model is trained from scratch alongside a latent diffusion model ( Rombach et al., 2022 , LDM;) , and (ii) hyper-transforming, where a pre-trained LDM is adapted by replacing its decoder with the HD , allowing for efficient transfer learning without retraining the entire generative pipeline. This flexibility enables LDMI to scale effectively while leveraging existing pre-trained diffusion models. Contributions Our key contributions can be summarized as follows: • We introduce the HD decoder, a full Transformer-based probabilistic decoder for learning distributions over INR parameters. • We integrate our method into Latent Diffusion Models ( Rombach et al., 2022 ) , with support both full training and hyper-transforming , enabling efficient adaptation of pre-trained models. We refer to this approach as LDMI . • The HD decoder achieves scalability and generalisation across data modalities, overcoming the bottlenecks of MLP-based hypernetworks and extending beyond deterministic Transformer-based methods. • We demonstrate the effectiveness of our approach on various generative tasks, showcasing its superiority in modelling high-resolution, structured data. By integrating probabilistic INR modelling within diffusion-based generative frameworks and efficient latent-to-parameters generation, our work establishes a new direction for flexible and scalable generative modelling with unconstrained resolution. The following sections provide a detailed description of the background (Section 2 ), the proposed methodology (Section 3 ), and empirical results (Section 4 ). 2 Background and Related Work 2.1 Latent Diffusion Models Latent Diffusion Models ( Rombach et al., 2022 , LDM;) are a class of generative models that learn efficient representations by applying diffusion processes in a compressed latent space. Unlike traditional diffusion models that operate directly in high-dimensional data spaces {\bm{y}}\in\mathbb{R}^{D} to approximate p({\bm{y}}) , LDMs first encode data into a lower-dimensional latent space {\bm{z}}\in\mathbb{R}^{d} , where d\ll D , using a probabilistic encoder \mathcal{E}_{\psi}(\cdot) to produce rich latent representations q_{\psi}({\bm{z}}|{\bm{y}}) . These are decoded via \mathcal{D}_{\lambda}(\cdot) to accurately recover data samples by modeling the conditional distribution p_{\lambda}({\bm{y}}|{\bm{z}}) . A \beta -VAE ( Higgins et al., 2017 ) with low \beta , a Vector-Quantized VAE ( van den Oord et al., 2017 , VQ-VAE;) , or VQGAN ( Esser et al., 2021 ) are suitable choices for learning the structured latent space. Additionally, for image data, perceptual losses can be incorporated to further enhance generation quality ( Larsen et al., 2016 ; Johnson et al., 2016 ; Hou et al., 2017 ; Dosovitskiy & Brox, 2016 ; Hou et al., 2019 ) . In the second stage, LDMs approximate the aggregate posterior q_{\psi}({\bm{z}})=\int q_{\psi}({\bm{z}}|{\bm{y}})p_{\text{data}}({\bm{y}})\dif{\bm{y}} with a diffusion model p_{\theta}({\bm{z}}) , shifting the generative modeling task from data space to latent space and significantly reducing computational cost. While both continuous-time SDE-based approaches ( Song et al., 2021b ) and discrete Markov chain formulations ( Ho et al., 2020 ; Song et al., 2021a ) are viable, we follow the DDPM-based approach as adopted in the original LDM implementation ( Rombach et al., 2022 ) . DDPMs ( Ho et al., 2020 ) train using an objective derived from a variational lower bound on the negative log-likelihood, reparameterized for efficiency as a denoising task. This results in a simplified loss that closely resembles denoising score matching ( Song et al., 2021b ) , and is widely used in state-of-the-art models ( Saharia et al., 2022 ; Dhariwal & Nichol, 2021 ) . For latent diffusion, the loss is applied to {\bm{z}}\sim q_{\psi}({\bm{z}}|{\bm{y}}) , yielding: L_{\text{DDPM}}=\mathbb{E}_{{\bm{y}},{\bm{z}},t,\epsilon}\left[\|\epsilon-\epsilon_{\theta}({\bm{z}}_{t},t)\|^{2}\right], (1) where {\bm{y}}\sim p_{\text{data}}({\bm{y}}) , {\bm{z}}\sim q_{\psi}({\bm{z}}|{\bm{y}}) , t\sim\mathcal{U}(1,T) , and \epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}) . This objective corresponds to predicting the added noise in a fixed forward process, effectively denoising {\bm{z}}_{t} toward the original latent {\bm{z}} . Sampling In DDPM, the reverse posterior density is no longer Markovian and coincides with the inference model proposed later in DDIM ( Song et al., 2021a ) . In DDIM, it is demonstrated that faster sampling can be achieved without retraining, simply by using the posterior approximation for the estimation \hat{{\bm{z}}} , which re-defines the generative process

Read the complete text