GMem: A Modular Approach for Ultra-Efficient Generative Models
Yi Tang, Peng Sun, Zhenglin Cheng, Tao Lin · 2024
Read GMem: A Modular Approach for Ultra-Efficient Generative Models by Yi Tang, Peng Sun, Zhenglin Cheng, Tao Lin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08781.txt.
Introduction
Yi Tang, Peng Sun, Zhenglin Cheng, and Tao Lin redesign how image-generating diffusion models store what they learn. These models turn noise into pictures stepwise, and studies show the process memorizes meaning from the training data. Bigger data patterns seem to demand bigger networks, which makes training slow and costly. The authors split memory out of the network into a separate fixed memory set that holds the meaning of the data.
With memory handled outside, the network can be smaller and training runs faster. On an image benchmark, their system trained about fifty times faster than a leading rival while reaching good quality within hours. With longer training it set a top score among models of its class, beating a rival that needed far more time. Sampling pictures also sped up, and output variety held up. The authors argue modular memory is a path to cheaper generative models.
Catalog record
| CREATOR | Yi Tang, Peng Sun, Zhenglin Cheng, Tao Lin |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08781
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GMem: A Modular Approach for Ultra-Efficient Generative Models Yi Tang, Peng Sun, Zhenglin Cheng, Tao Lin Abstract Recent studies indicate that the denoising process in deep generative diffusion models implicitly learns and memorizes semantic information from the data distribution. These findings suggest that capturing more complex data distributions requires larger neural networks, leading to a substantial increase in computational demands, which in turn become the primary bottleneck in both training and inference of diffusion models. To this end, we introduce GMem: A Modular Approach for Ultra-Efficient Generative Models. Our approach GMem decouples the memory capacity from model and implements it as a separate, immutable memory set that preserves the essential semantic information in the data. The results are significant: GMem enhances both training, sampling efficiency, and diversity generation. This design on one hand reduces the reliance on network for memorize complex data distribution and thus enhancing both training and sampling efficiency. On ImageNet at 256 × 256 256\times 256 resolution, GMem achieves a 50 × 50\times training speedup compared to SiT, reaching FID = 7.66 =7.66 in fewer than 28 28 epochs ( ∼ \sim 4 4 hours training time) 1 1 1 All training time measurements are obtained on an 8 × 8\times H800 GPU cluster. , while SiT requires 1400 1400 epochs. Without classifier-free guidance, GMem achieves state-of-the-art (SoTA) performance FID = 1.53 =1.53 in 160 160 epochs with only ∼ 20 \sim 20 hours of training, outperforming LightningDiT which requires 800 800 epochs and ∼ \sim 95 95 hours to attain FID = 2.17 =2.17 . Additionally, the memory bank supports training-free adaptation to new images not present in the training set. Our code is available at https://github.com/LINs-lab/GMem . † † affiliationnotice: Equal contribution Figure 1 : GMem Significantly enhances training and sampling efficiency of diffusion models on ImageNet 256\times 256 . We propose decoupling memorization capabilities from the model by implementing a separate, immutable memory bank that preserves essential data information. Sub-figure (a) highlights the core concept of GMem, where \boldsymbol{\epsilon} denotes input noise and \mathbf{x}_{0} represents generated samples. In GMem, we disentangle generalization and memorization capabilities, assigning memorization to an external memory bank \mathbf{M} . This decoupling reduces computational and capacity overhead, thus accelerating the process. Sub-figure (b) demonstrates the training efficiency of GMem on ImageNet 256\times 256 . At an FID =4.86 , GMem achieves over 25\times speedup compared to REPA ( Yu et al., 2024 ) . At an FID =7.66 , it achieves over 50\times speedup relative to SiT ( Ma et al., 2024 ) . Sub-figure (c) illustrates sampling efficiency. At the same FID target, GMem requires 5\times fewer NFEs compared to REPA and 10\times fewer NFEs compared to SiT. 1 Introduction Deep generative models based on denoising ( Yang & Wang, 2023 ; Ho et al., 2020 ; Song et al., 2020a ; Nichol & Dhariwal, 2021 ; Choi et al., 2021 ; Li et al., 2024 ; Chen et al., 2024c ; Li et al., 2023c ) —which prioritize the generation of high-quality, realistic data—have achieved notable success within the deep learning community. These methods demonstrate exceptional performance in complex tasks such as zero-shot text-to-image and video generation ( Podell et al., 2023 ; Saharia et al., 2022 ; Esser et al., 2024 ; Polyak et al., 2024 ; Brooks et al., 2024 ) . However, training and sampling diffusion models suffer from the issue of high computational burden ( Karras et al., 2022 ; Karras et al., 2024 ) , due to the necessity of using neural networks with larger capacities for better empirical performance ( Karras et al., 2022 ; Karras et al., 2024 ) . In the meanwhile, a line of research examines the diffusion models from the perspective of representation learning ( Li et al., 2023a ; Xiang et al., 2023 ; Chen et al., 2024c ; Mukhopadhyay et al., 2021 ) , where diffusion models can capture semantic features within their hidden layers, with advanced models producing even stronger representations ( Xiang et al., 2023 ) . Yang & Wang (2023) illustrate that diffusion models strike a balance between learning meaningful features and regularizing the model capacity. Kadkhodaie et al. (2023) further interpret the generalization behavior in diffusion models, indicating that diffusion models aim to learn and encapsulate semantic information from the training data, subsequently generalizing to the true data distribution. In this paper, we build upon the insights from prior works ( Yang & Wang, 2023 ; Kadkhodaie et al., 2023 ) and propose a novel conjecture: the functionality of diffusion models can be implicitly decomposed into two distinct components: (i) memorization of semantic information and (ii) generalization to the true data distribution. We further posit that the intrinsic generation process of diffusion models can be interpreted as a transformation of noisy input samples into the model’s internal semantic representation, which is subsequently “decoded” into the target data distribution. To elaborate, (a) During the training stage, diffusion models rely on substantial neural network capacity and computational resources to extract and memorize semantic information from noisy samples; (b) During the sampling/inference stage, the models typically remap Gaussian noise to the internal semantic distribution before reconstructing the data distribution, rather than directly transforming noise into data. Building upon the aforementioned analyses, the efficiency challenges of training and sampling diffusion models stem from the need to learn and memorize internal semantic information. This process inherently incurs high computational costs and imposes significant demands on model capacity. As a remedy, we propose GMem, a new paradigm that decouples the memorization capability from the neural network by storing semantic information in an external memory bank. This separation eliminates the need for the neural network itself to memorize complex data distributions, enabling significant improvements: (1) accelerated training and sampling by avoiding the direct fitting of intricate semantic distributions; (2) flexible adjustment of memorization capacity in a training-free manner; (3) enhanced generalization, allowing the generation of images beyond the training set; and (4) reduced storage overhead by offloading memorization to the memory bank. Together, GMem provides a scalable and efficient framework for generative tasks. Extensive experiments across diverse diffusion backbones ( Ma et al., 2024 ; Yao & Wang, 2025 ) and image tokenizers ( Yao & Wang, 2025 ; Chen et al., 2024a ) validate the effectiveness and efficiency of GMem. On ImageNet 256\times 256 , GMem achieves SoTA performance FID =1.53 in 160 epochs with only \sim 20 hours training time without classifier-free guidance. Under the same number of training epochs as 80 , GMem enhances the sampling efficiency by 10\times , where GMem requires only 25 sampling steps to achieve an FID =12.3 while the vanilla SiT needs 250 sampling steps, yielding 10\times sampling time savings . We outline the main contributions of this paper below: (a) We propose a novel conjecture that interprets the functionality of diffusion models into two distinct components: (i) memorization of semantic information and (ii) generalization to the true data distribution. (b) We introduce GMem, a simple yet highly effective approach that decouples the memorization capability of diffusion models into an external memory bank. This separation alleviates the memorization burden on the neural networks and enhances generalization flexibility. (c) GMem significantly enhances