Optimizing Inference in Transformer-Based Models
Daniel Schlabig, Nguyen Duong, Prasad Ganesan, Siu Hang Ho · 2025
Read Optimizing Inference in Transformer-Based Models by Daniel Schlabig, Nguyen Duong, Prasad Ganesan, Siu Hang Ho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17894.txt.
Introduction
Siu Hang Ho and 3 co-authors test ways to make a diffusion model run faster and cheaper. Their 2025 bench study uses the fast-DiT model, which makes images step by step. Bigger models tend to score better but cost more time, memory, and compute.
The team tries pruning, low-bit packing, teaching a small copy, and leaner attention steps. They also try a mix-of-experts plan, where just part of the net runs per input. Each trick is checked for how much score it keeps as cost falls. The results map which cuts are safe and which hurt output.
Catalog record
| CREATOR | Daniel Schlabig, Nguyen Duong, Prasad Ganesan, Siu Hang Ho |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17894
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing Inference in Transformer-Based Models Daniel Schlabig, Nguyen Duong, Prasad Ganesan, Siu Hang Ho Abstract Efficient inference is a critical challenge in deep generative modeling, particularly as diffusion models grow in capacity and complexity. While increased complexity often improves accuracy, it raises compute costs, latency, and memory requirements. This work investigates techniques such as pruning, quantization, knowledge distillation, and simplified attention to reduce computational overhead without impacting performance. The study also explores the Mixture of Experts (MoE) approach to further enhance efficiency. These experiments provide insights into optimizing inference for state-of-the-art Fast Diffusion Transformer (fast-DiT) model. 1 Introduction / Background / Motivation Although diffusion transformers peebles2023scalablediffusionmodelstransformers deliver high accuracy and image quality, their heavy computational demands impede real-time and low-resource deployment. This paper studied methods to reduce the computational cost of deep generative models during inference by measuring changes in compute cost and quality compared to a selected base model. Faster inference is important for diffusion models to make high-quality image generation faster and less expensive, lowering the barriers to deploying advanced image generation in a wide range of applications and industries, from mobile devices to real-time systems. Applications include real-time, interactive media and personalized services, scalable, automated workflows in industry, and more accessible state-of-the-art models for hobbyists and researchers, democratizing access across professional and personal environments. Our research explores various techniques for model and inference optimization, examining their impact on compute cost and image quality. Prior work for each approach is briefly discussed below. Pruning. Pruning reduces memory requirements and computational complexity of deep learning networks by removing less critical layers or increasing parameter sparsity. Unstructured L2 norm pruning molchanov2017pruning removes weights with the smallest L2 norms, adding sparsity without altering model architecture, making it ideal for fine-grained compression of large models. Quantization. Quantization reduces the precision of model weights and activations (typically from 32-bit floating-point to 8-bit integers), significantly reducing model size and memory usage while aiming to preserve performance. Recent work shows that quantization techniques can be successfully extended to large-scale transformer and generative models dettmers2022llmint88bitmatrixmultiplication . Common approaches include post-training quantization (PTQ), which requires no retraining, and quantization-aware training (QAT), which incorporates quantization during model training jacob2017quantizationtrainingneuralnetworks ; han2016deepcompressioncompressingdeep . Our work focuses on post-training quantization as a zero-training-cost approach to model compression, making it particularly attractive for large diffusion models where retraining is expensive. Attention. In transformer models, scaled dot-product attention complexity scales quadratically with token sequence length N and linearly in hidden layer size C ( \mathcal{O}(N^{2}C) ) vaswani2023attentionneed . Reducing or eliminating the quadratic dependence is the focus of much prior work. Methods include neighborhood or window-based approaches to ’localize’ most attention hassani2023neighborhoodattentiontransformer liu2021swintransformerhierarchicalvision , and ”linearizing” methods han2023flattentransformervisiontransformer pu2024efficientdiffusiontransformerstepwise that seek to avoid multiplying QK^{T} , eliminating the N^{2} dependency by calculating K^{T}V first at \mathcal{O}(N^{2}C) . Others have applied multiple queries to the same keys and values ainslie2023gqatraininggeneralizedmultiquery described as grouped-query attention or multi-query attention (when a single K-V head is used). These methods aim to reduce memory bandwidth in cases where memory is the inference bottleneck. These methods may be less relevant to smaller models applied to low resolution images, leading us to explore a range of attention simplification approaches. Knowledge Distillation. Knowledge distillation transfers knowledge from a larger ”teacher” model to a smaller ”student” model. The student learns to mimic the teacher’s behavior while using fewer parameters and computational resources. This technique has been effective in various domains, including image classification hinton2015distillingknowledgeneuralnetwork , natural language processing sanh2020distilbertdistilledversionbert , and more recently in diffusion models salimans2022progressivedistillationfastsampling . By learning from soft predictions instead of only hard labels, the student gains richer supervision during training. This can improve generalization and enable the student to reach accuracy close to that of the larger teacher model. Mixture of Experts. Mixture of Experts (MoE) is an architectural optimization technique where for each input, only a subset of specialized expert subnetworks is activated based on token routing. Instead of evaluating the entire model for every input, only a few experts process each token, significantly reducing computational cost while preserving or even increasing model capacity and expressiveness. When applied to Diffusion Transformers, the MoE mechanism typically replaces the standard MLP layers inside the Transformer blocks with sparse MoE layers. Each token is routed by a lightweight gating network to a small number of experts, enabling the model to scale up in total parameters without proportionally increasing FLOPs or memory usage. This allows larger, more powerful diffusion models. For example, DiT-MoE fei2024scalingdiffusiontransformers16 scales to 16.5 billion parameters while only activating 3.1 billion per forward pass. Dataset . ImageNet ImageNet256 is a large-scale dataset of images organized in a hierarchical ontology. We used a pre-processed subset of 540,000 256x256 images grouped into 1000 classes from Kaggle kaggle . These images had been cropped to square and organized into a flat folder structure by class. For our experiments, we randomly sampled 200 classes (350-650 images per class) for approximately 110,000 total images, each with color or grayscale content and a class label. The data source we used was not split into train-validation-test groups. However, since our experiments were generative rather than discriminative (i.e. not classification), we did not split the dataset when training our images. 2 Approach Base model selection . We used the fast-DiT jin2024fast re-implementation of DiT as our base diffusion transformer. Its improvements over the original DiT include saving the latent features of the variational autoencoder (VAE) rather than re-computing at each iteration, and quantizing the model parameters as fp16 without changing the original architecture. We compared qualitative performance and training speed of published model parameters B/2, B/4, S/2, and S/4 (each with depth=12, hidden_size=768, num_heads=12 for B, hidden_size=384, num_heads=6 for S, patch_size=2 or 4 based on name). We also trialed a smaller model size we named XS with depth=6, hidden_size=256, num_heads=4. We selected S/2 as based model based on it producing reasonable quality images with tractable training times for this project. Also, S/2 has a more balanced token sequence length ( N =256) and hidden dimension size ( C =384), which is closer to the ratio seen with high resolution image generation using the DiT XL/2 model. Pruning. We applied L2 norm pruning to reduce the number of attention heads in the S/2 model. For each attention layer, we computed the L2 norm of each head’s Q, K, and V weight slices, then summed them to calculate a score for each attention head. We then kept the top- k scoring heads and masked the rest by setting their weights and biases to zero. k was varied between [2,5] . Quantization. PyTorch’s native quantization tools only support CPU inference pytorch_quantization_doc , so we needed an alternative approach for fair GPU-based comparisons with our baseline fast-DiT models. To address this, we implemented TorchAO’s Int8WeightOnlyConfig() torchao , which provides experimental support for static int8 weight-only quantization on CUDA through quantized tensor subclasses. Only the model weights were quantized; activations remained in float precision during inference. The quantization function applies an affine transformation: q=\text{round}(x/S)+Z (1) where S is a scaling factor and Z is a zero-point, to map real-valued weights into 8-bit integers. While TorchAO enabled successful quantization, its CUDA backend remains experimental and unoptimized, which likely explains the