HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations
Marco Federici, Riccardo Del Chiaro, Boris van Breugel, Paul Whatmough, Markus Nagel · 2025
Read HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations by Marco Federici, Riccardo Del Chiaro, Boris van Breugel, Paul Whatmough, Markus Nagel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09932.txt.
Introduction
Marco Federici and four colleagues propose HadaNorm, a way to shrink diffusion transformers, in a paper from twenty twenty-five. Diffusion models lead in making images, but they need much memory and compute. That cost blocks their use on small devices with few resources. Post-training quantization cuts the bit width of the matrix math. That shrinks the models after they are trained.
Plain methods of this kind cope poorly with outliers in the data. Stronger cuts call for first reshaping the weights and the activations. HadaNorm is a linear change that does both jobs. It centers channel activations on their mean, and it applies Hadamard transforms. The mix tames outliers and lets the activations be cut much harder. Across the parts of transformer blocks, the authors report lower error than before in each case. Their method beats the best current methods in their tests.
Catalog record
| CREATOR | Marco Federici, Riccardo Del Chiaro, Boris van Breugel, Paul Whatmough, Markus Nagel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-09932
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations Marco Federici, Riccardo Del Chiaro, Boris van Breugel, Paul Whatmough, Markus Nagel Abstract Diffusion models represent the cutting edge in image generation, but their high memory and computational demands hinder deployment on resource-constrained devices. Post-Training Quantization (PTQ) offers a promising solution by reducing the bitwidth of matrix operations. However, standard PTQ methods struggle with outliers, and achieving higher compression often requires transforming model weights and activations before quantization. In this work, we propose HadaNorm, a novel linear transformation that extends existing approaches and effectively mitigates outliers by normalizing activations feature channels before applying Hadamard transformations, enabling more aggressive activation quantization. We demonstrate that HadaNorm consistently reduces quantization error across the various components of transformer blocks, achieving superior efficiency-performance trade-offs when compared to state-of-the-art methods. † † affiliationnotice: Equal contribution 1 Introduction Diffusion models have emerged as the leading technique in deep learning for image generation, offering unparalleled visual realism. However, this advancement comes at a significant computational cost, primarily due to the large model sizes and the iterative denoising procedures required for each image generation. As the demand for scalable and efficient deployment of these models grows both on the cloud and on the edge where computational resources are scarce, optimizing inference efficiency has become a critical area of research. Post-Training Quantization (PTQ) presents a promising solution for enhancing inference efficiency by quantizing weights and activations, especially for high-power demanding operations such as linear layers and large matrix multiplications. Despite its potential, PTQ faces substantial challenges, particularly when pushing activation bitwidth to 8-bits (A8) or weight to 4-bits (W4). Outliers are a major hurdle for low-bitwidth quantization: they can only be represented when a large quantization grid is used, leading to large bins and loss of precision for the majority of data. One approach to reduce outliers and consequently improve quantization is to apply invertible transformations to weights and activations, which do not alter the overall model output, but which do allow scaling outlier channels ( Xiao et al. 2023 ) . Follow-up works ( Ashkboos et al. 2024 ; Liu et al. 2024 ; Ma et al. 2024 ; Zhao et al. 2025 ) use fast Hadamard transforms to mix channels, which effectively whitens the distribution and reduces outliers. Figure 1: HadaNorm reduces quantization error. We take an illustrative setting of four channels with different distributions (top-left). Normalization (bottom-left) improves quantization, but it does not mix channels and hence cannot get rid of heavy tails. Hadamard transform (HT) (top-right) suffers when the channels have different means. HadaNorm (bottom-right) achieves better whitening, by first normalizing and then applying the HT. The more Gaussian is easier to quantize. This paper extends previous work by introducing a simple yet effective dynamic centering transformation, which can be combined with existing approaches to further reduce quantization error. Contributions. Our contributions are as follows: 1. We argue why the efficacy of the Hadamard transform’s mixing is significantly reduced due to mean and scale differences across channels (Figure 1 ). 2. We introduce HadaNorm, a simple transformation that applies centering and rescaling of channels prior to a Hadamard transform to further improve performance of activation quantization. 3. Empirically, we show that Hadanorm significantly improves quantization performance in image diffusion transformers (DiT). In particular, we show it reduces each quantizer’s error consistently, and achieves a state-of-the-art CLIP score of 32.01 on PixArt-Sigma at W4A4. 2 Related Work Transformations for better quantization are most explored in the Large Language Models (LLM) space. Xiao et al. 2023 find that activation quantization is harder than weight quantization, and propose per-channel scales to move outliers from activations to weights. Ma et al. 2024 ; Shao et al. 2024 propose an affine transformation, that effectively adds a static bias term for LLM quantization. Ashkboos et al. 2024 propose to use Hadamard transforms, which are cheap at inference time, yet allow spreading of outliers across channels. ( Zhao et al. 2025 ) combines channel scaling and mixing to achieve higher compression on DiT architectures, although mixed precision is required to address quantization-sensitive components in the architecture. Li et al. 2024 further extends previous work on channel scaling Xiao et al. 2023 by introducing high-precision low-rank matrices to absorb the weight outliers prior to weight quantization, achieving higher compression levels at the cost of a small overhead. Shao et al. 2025 defines time-dependent transformation to address changes in the activation distribution during subsequent denoising steps, by applying transformation tailored to the activation distributions based on Lin et al. 2024 , at the cost of additional overhead at inference time. Our method further extends the literature on transformation applied to DiT models by introducing a simple centering strategy that does not require additional calibration and introduces minimal overhead, yet consistently improves upon existing strategies. 3 Method 3.1 Distributional differences between channels reduces Hadamard transform mixing Some of the most related works use Hadamard transforms to mix channels, thereby spreading outliers over multiple channels. This is motivated by the central limit theorem: the Hadamard transform (HT) effectively adds up different channels (with signs flipped at times), which means that the distribution of each channel should become more Gaussian. We make the simple observation that the convergence towards a Gaussian distribution is slower when the variables added have significantly different moments. This is often the case in transformer models, as outliers are often constraint to a few problematic channels. This then implies that the HT is less effective at mixing and whitening different channel distributions, then if these channels would have more similar distributions (e.g. if we would normalize them before applying the HT). We illustrate this using a toy example. In Figure 1 top-left, we plot the distribution of four channels. If we naively choose a quantization grid based on all four channels, we would see that the quantization error would be large, as many channels would collapse to just one or two values. When we apply a HT (top-right), we find that indeed each channel is closer to a normal distribution. However, due to non-zero mean of the initial channels, we observe large difference in the means across channels after applying the HT. For example, channel 2 corresponds to the row [+1,-1,+1,-1] of the HT, which due to the large positive mean of the initial channel 2, and large negative mean of the initial channel 3, leads to a large negative mean after the HT. The solution is to aim and make the distributions more similar before applying the Hadamard transform. The simplest approach is normalizing channels. In the bottom-left figure, we show the effect of normalizing the channels. Although the SQNR is significantly better than before, channels are not mixed—if there would have been a channel with more significant heavy-tails, just normalizing the channel will not be able to reduce the heavy tails. We propose HadaNorm, which combines the dynamic normalization with the HT. Though simple, we get the best of both worlds: the HT mixes the channel distributions to reduce heavy tails, and because distributions are normalized the convergence is faster. Let us consider how this can be applied to a transformer model. Figure 2: HadaNorm reduces quantization error of all quantizers. Effect of activation quantization