A Non-Adversarial Approach to Idempotent Generative Modelling

Mohammed Al-Jaff, Giovanni Luca Marchetti, Michael C Welle, Jens Lundell, Mats G. Gustafsson, Gustav Eje Henter, Hossein Azizpour, Danica Kragic · 2025

Read A Non-Adversarial Approach to Idempotent Generative Modelling by Mohammed Al-Jaff, Giovanni Luca Marchetti, Michael C Welle, Jens Lundell, Mats G. Gustafsson, Gustav Eje Henter, Hossein Azizpour, Danica Kragic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02614.txt.

Introduction

Mohammed Al-Jaff and seven co-authors fix a family of generative models that kept collapsing. Idempotent networks are trained to act as the identity on real data and to project anything else back onto the data manifold. One network both generates samples and repairs corrupted inputs. In practice their training borrowed adversarial ingredients, and with them came the familiar diseases of such training: mode collapse, dropped modes, and instability.

The replacement, NAIGN, removes the adversarial part entirely. Its loss combines plain repair with the non-adversarial objective of implicit maximum likelihood estimation. The new models restored corrupted data better and produced samples matching the data distribution more closely than the original design. As a bonus, the trained networks turn out to encode the distance to the data manifold, and with it an energy-based model of the data.

Catalog record

CREATORMohammed Al-Jaff, Giovanni Luca Marchetti, Michael C Welle, Jens Lundell, Mats G. Gustafsson, Gustav Eje Henter, Hossein Azizpour, Danica Kragic
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-02614

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Non-Adversarial Approach to Idempotent Generative Modelling Mohammed Al-Jaff, Giovanni Luca Marchetti, Michael C Welle, Jens Lundell, Mats G. Gustafsson, Gustav Eje Henter, Hossein Azizpour, Danica Kragic Abstract Idempotent Generative Networks (IGNs) are deep generative models that also function as local data manifold projectors, mapping arbitrary inputs back onto the manifold. They are trained to act as identity operators on the data and as idempotent operators off the data manifold. However, IGNs suffer from mode collapse, mode dropping, and training instability due to their objectives, which contain adversarial components and can cause the model to cover the data manifold only partially – an issue shared with generative adversarial networks. We introduce Non-Adversarial Idempotent Generative Networks (NAIGNs) to address these issues. Our loss function combines reconstruction with the non-adversarial generative objective of Implicit Maximum Likelihood Estimation (IMLE). This improves on IGN’s ability to restore corrupted data and generate new samples that closely match the data distribution. We moreover demonstrate that NAIGNs implicitly learn the distance field to the data manifold, as well as an energy-based model. † † paperid: 1171 1 Introduction and Related Work Generative modeling is one of the fundamental tasks in modern machine learning ( Tomczak 2024 ) . Contemporary deep generative models are characterized by their ability to transform random noise into data-like outputs, enabling the synthesis of new, realistic samples across various domains, e.g., image generation ( Zhang et al. 2023 ) , text synthesis ( Li et al. 2024a ) , and 3D object modeling ( Li et al. 2024b ) . Models such as Generative Adversarial Networks (GANs; ( Goodfellow et al. 2020 ) ), Variational Autoencoders (VAEs; ( Kingma and Welling 2014 ; Rezende et al. 2014 ) ), and diffusion models ( Ho et al. 2020 ) have demonstrated remarkable success in capturing complex data distributions and producing high-fidelity samples. Figure 1 : NAIGNs f are trained to fix points ( f(\mathbf{x})~=~\mathbf{x} ) on the data manifold \mathcal{M} and to map arbitrary points to \mathcal{M} ( f(\mathbf{x})\in\mathcal{M} ) via IMLE, which in turn imply idempotency ( f(f(\mathbf{x}))=f(\mathbf{x}) ). Another fundamental task in machine learning is manifold learning and projection ( Izenman 2012 ) , consisting of mapping corrupted or out-of-distribution data points back onto the data manifold. The latter includes tasks related to image-to-image translation and domain adaptation ( Zhu et al. 2020 ; Mütze et al. 2022 ) . The former represents the support of the underlying data distribution. This is essential in applications requiring reconstruction, restoration, and robustness, including inverse problems such as denoising, deblurring, data imputation, and adversarial purification ( Nie et al. 2022 ; Peng et al. 2023 ; Zhu et al. 2023 ) . Figure 2 : Comparison between NAIGN (top row) and IGN (bottom row) trained on a simple tri-modal one-dimensional distribution. Our proposed method, NAIGN, is better at mitigating mode collapse and mode dropping issues that IGN is susceptible to. For reference, the light gray histograms in the three rightmost columns are the target distribution from the first column. While generative modeling and manifold projection have traditionally been treated as separate tasks, recent advancements have sought to unify these capabilities within a single framework ( Sun et al. 2025 ) . Models that can both generate realistic samples and project onto the data manifold hold significant promise for tasks that simultaneously require synthesis and manifold distance estimation. Idempotent Generative Networks (IGNs; ( Shocher et al. 2023 ) ) represent a step in this direction, aiming to integrate these functionalities. Formally, let f:\mathbb{R}^{D}\to\mathbb{R}^{D} denote a neural network, let \mathbf{z}\in\mathbb{R}^{D} be an arbitrary point in ambient space, and let \mathbf{x}\in\mathcal{M}\subset\mathbb{R}^{D} denote a point on the data manifold. IGNs are defined by their ability to act as identity operators on manifold points, i.e., f(\mathbf{x})=\mathbf{x} when \mathbf{x}\in\mathcal{M} , and as idempotent operators on the ambient space, satisfying f(f(\mathbf{z}))=f(\mathbf{z}) . This dual property enables generation via projection of noise onto the data manifold. In the case of IGNs, the generation process happens within a single (-or few) model forward passes. This is in contrast to diffusion-based models, where multiple passes are performed to incrementally map noise samples to the data manifold. However, as we will show, IGNs face challenges in mode collapse, mode dropping, and training instability due to adversarial components in their objective. In this work, we aim to improve and enhance IGNs by addressing their issues. We propose Non-Adversarial Idempotent Generative Networks (NAIGN), introducing an alternative loss formulation that eliminates adversarial components. Our objective consists of a reconstruction loss term and an Implicit Maximum Likelihood Estimation (IMLE) term ( Li and Malik 2018 ) , which together act as a generative and fixed-point loss – see Figure 1 . Our approach not only stabilizes training dynamics, but also enhances the model’s ability to approximate the local distance field of the data manifold. We empirically validate NAIGN and compare it with IGN on low-dimensional synthetic datasets, as well as on the MNIST image dataset. We investigate idempotency, stability, as well as generation and restoration capabilities. Furthermore, we showcase the performance of NAIGN as a manifold projector with applications to density estimation. The Python code for NAIGN and all supplementary material available at https://github.com/MohammedAlJaff/naign ( Al-Jaff 2025 ) 2 Background 2.1 Notation Let p_{\text{data}} be an unknown distribution over an ambient space \mathbb{R}^{D} . Data is sampled i.i.d. from this distribution. In order to formalize the data manifold , we assume that p_{\text{data}} is concentrated around a manifold \mathcal{M}\subset\mathbb{R}^{D} . Let p_{Z} be a source distribution, e.g., standard normal, from which we sample \mathbf{z}\in\mathbb{R}^{D} . Consider a neural network f_{\theta}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{D} with parameters \theta . We define the drift of the network as the distance between the input and the output: \delta_{\theta}(\mathbf{z}):=d(\mathbf{z},f_{\theta}(\mathbf{z})) , \mathbf{z}\in\mathbb{R}^{D} . Here, d is some distance function, such as the Euclidean metric. To quantify how far any point \mathbf{z}\in\mathbb{R}^{D} is from the data manifold, we define the manifold distance field as d_{\mathcal{M}}(\mathbf{z})=\min_{\mathbf{m}\in\mathcal{M}}d\left(\mathbf{z},\mathbf{m}\right). (1) This function measures the shortest distance from \mathbf{z} to any point on \mathcal{M} , capturing the notion of proximity to the data manifold. 2.2 Idempotent Generative Networks This section reviews IGNs, which were introduced by Shocher et al. 2023 . Two specific properties characterize these deep generative models. First, an ideal IGN fixes points on the data manifold. Formally, for \mathbf{x}\in\mathcal{M}\subset\mathbb{R}^{D} we have: f_{\theta}(\mathbf{x})=\mathbf{x}. (2) Second, IGNs should be idempotent , meaning that applying the model twice to any point in the ambient space is equivalent to applying it once. Formally, for \mathbf{z}\in\mathbb{R}^{D} we have: f_{\theta}(f_{\theta}(\mathbf{z}))=f_{\theta}(\mathbf{z}). (3) While idempotency and reconstruction are desirable properties, they do not ensure full coverage of \mathcal{M} , and even allow for trivial solutions, such as the identity function f_{\theta}(\mathbf{z})=\mathbf{z} . This raises the need for careful loss design. Shocher et al. 2023 propose to circumvent these issues via a combination of three specific weighted loss functions: \mathcal{L}_{\text{IGN}}(\theta)=w_{\text{rec}}\mathcal{L}_{\text{rec}}(\theta)+w_{\text{idem}}\mathcal{L}_{\text{idem}}(\theta)+w_{\text{tight}}\mathcal{L}_{\text{tight}}(\theta). (4) The above loss terms are defined as follows. • Reconstruction Loss : \mathcal{L}_{\text{rec}}(\theta)=\mathbb{E}_{\mathbf{x}\sim p_{\text{data}}}\left[d\left(\mathbf{x},f_{\theta}(\mathbf{x})\right)\right], (5) where d(\cdot,\cdot) is a distance function, e.g., Euclidean distance. This loss ensures that the IGN acts as the identity function on data manifold points. • Idempotency Loss : \mathcal{L}_{\text{idem}}(\theta)=\mathbb{E}_{\mathbf{z}\sim p_{Z}}\left[d\left(f_{\theta}(\mathbf{z}),f_{\theta_{\bot}}\left(f_{\theta}(\mathbf{z})\right)\right)\right], (6) where \theta_{\bot} indicates that gradients are not propagated through f_{\theta_{\bot}} (i.e., stop-gradient operation is

Read the complete text