Calibri: Enhancing Diffusion Transformers via Parameter-Efficient Calibration

Danil TokhchukovMSUAysel MirzoevaMSUAndrey KuznetsovFusionBrain Lab, AXXX, Konstantin SobolevFusionBrain Lab, AXXXMSU · 2026

Read Calibri: Enhancing Diffusion Transformers via Parameter-Efficient Calibration by Danil TokhchukovMSUAysel MirzoevaMSUAndrey KuznetsovFusionBrain Lab, AXXX, Konstantin SobolevFusionBrain Lab, AXXXMSU (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24800.txt.

Introduction

Tokhchukov and co-authors present Calibri, a parameter-efficient way to calibrate diffusion transformers for image generation. A close look at denoising shows one learned scaling parameter per block can lift performance sharply. Calibri frames calibration as a black-box reward optimisation problem, solved with an evolutionary algorithm. It adjusts only about a hundred parameters in total. Across several text-to-image models, quality improves consistently. Calibri also cuts the number of inference steps needed while keeping output quality high, making it a cheap upgrade to existing models.

Catalog record

CREATORDanil TokhchukovMSUAysel MirzoevaMSUAndrey KuznetsovFusionBrain Lab, AXXX, Konstantin SobolevFusionBrain Lab, AXXXMSU
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33638
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.24800

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Calibri: Enhancing Diffusion Transformers via Parameter-Efficient Calibration Danil TokhchukovMSUAysel MirzoevaMSUAndrey KuznetsovFusionBrain Lab, AXXX, Konstantin SobolevFusionBrain Lab, AXXXMSU Abstract In this paper, we uncover the hidden potential of Diffusion Transformers (DiTs) to significantly enhance generative tasks. Through an in-depth analysis of the denoising process, we demonstrate that introducing a single learned scaling parameter can significantly improve the performance of DiT blocks. Building on this insight, we propose Calibri , a parameter-efficient approach that optimally calibrates DiT components to elevate generative quality. Calibri frames DiT calibration as a black-box reward optimization problem, which is efficiently solved using an evolutionary algorithm and modifies just ∼ 10 2 \sim 10^{2} parameters. Experimental results reveal that despite its lightweight design, Calibri consistently improves performance across various text-to-image models. Notably, Calibri also reduces the inference steps required for image generation, all while maintaining high-quality outputs. * * footnotetext: The authors contributed equally. $\dagger$ $\dagger$ footnotetext: Project lead, correspondence: [email protected] $\ddagger$ $\ddagger$ footnotetext: Project: https://v-gen-ai.github.io/Calibri-page/ 1 Introduction In recent years, the field of visual content generation has experienced significant advancements, largely fueled by the development of diffusion models 14 ; 27 . Cutting-edge models like Stable Diffusion 3 8 and FLUX 22 have redefined the landscape of modern generative frameworks. These models represent a shift from the traditional UNet architecture 28 to the more advanced Diffusion Transformer (DiT) 26 , while also incorporating innovative techniques such as flow matching 23 to enhance their capabilities. This powerful combination of a DiT backbone and flow matching has become the new de facto standard, extending far beyond text-to-image synthesis to power diverse tasks such as instruction-guided image editing 21 ; 37 and video generation 34 . Diffusion transformers are built from a sequence of identical blocks, each containing attention and MLP layers. Despite this uniform architecture, recent work suggests their functional contributions are highly uneven. For instance, Stable Flow 1 identified ”vital layers” within the transformer, whose exclusion from generation process significantly alters the model’s output. This finding implies that not all layers contribute equally to the final generation. Building on this insight, we analyze the contribution of individual DiT blocks and uncover two surprising results. First, we find that selectively disabling certain blocks can actually improve generation quality, suggesting some may introduce detrimental artifacts. Second, we discover that a simple re-weighting of each block’s output – by multiplying it with a single learned scalar – consistently enhances the model’s performance over the original. These observations lead us to our central hypothesis: The standard DiT architecture is sub-optimally weighted, and its performance can be significantly improved through a simple post-hoc calibration of its blocks. Motivated by this hypothesis, we propose Calibri , a parameter-efficient approach designed to calibrate the contributions of DiT’s architectural components and improve generation quality (Figure ). Specifically, we frame the process of determining calibration coefficients as a black-box optimization problem with only \sim 10^{2} parameters. The objective is to maximize the quality of model outputs, as measured by a reward model 25 ; 35 . To solve this optimization problem, we leverage the gradient-free evolutionary strategy CMA-ES 11 ; 10 , which effectively identifies optimal scaling coefficients. Furthermore, we introduce Calibri Ensemble , which integrates multiple calibrated models to further boost generative performance. Notably, Calibri also reduces the number of inference steps required for image generation, significantly improving both efficiency and quality. Extensive experiments across diverse baseline models validate the effectiveness of Calibri in achieving consistent performance gains without computational overhead. 2 Related Work Diffusion Models Backbones. Early diffusion models predominantly utilized U-Net 28 backbones with residual blocks 12 , pixelwise self-attention 32 , and cross-attention layers for text-image conditioning 15 ; 27 ; 16 ; 3 . Recently, the field has shifted towards Diffusion Transformer (DiT)-based architecture 26 , which received significant attention due to the scalability of transformer models 32 . One notable development is PixArt-alpha 4 , which effectively applied DiT for text-conditional generation while preserving the conventional cross-attention mechanism for text-based conditioning. using a conventional cross-attention mechanism for text conditioning. A key milestone in this evolution is the introduction of the Multimodal Diffusion Transformer (MM-DiT) 8 , which employs distinct transformers to process textual and visual inputs, subsequently combining their sequences through unified attention operations. Diffusion Model Backbone Interpretability. Recent research has significantly advanced the understanding of diffusion model architecture, enabling novel applications. Early studies showed that cross-attention maps between text prompts and visual tokens produce high-quality saliency maps to predict spatial locations of textual concepts 31 , applied in tasks like image editing 13 and layout control 5 ; 7 . Other works explored diffusion model components: Free-U 29 highlighted the U-Net backbone’s denoising role and skip connections’ contribution of high-frequency features, improving its denoising efficacy. Additionally, methods like Stable Flow 1 and FreeFlux 36 analyzed Diffusion Transformer (DiT) blocks, identifying critical layers for image formation and differentiating positional versus content-focused layers, leading to training-free image editing techniques that leverage interpretability in diffusion models. Visual Generative Model Alignment. Aligning generative models, including diffusion models and rectified flows, with human feedback has significantly improved their performance. Conventional methods rely on reward models to capture human preferences 25 ; 39 ; 20 ; 35 and often use RLHF-inspired techniques like reward backpropagation 6 ; 39 , Direct Preference Optimization (DPO) 33 , Differentiable Diffusion Preference Optimization (DDPO) 2 , and Group Relative Policy Optimization (GRPO) 24 , which typically require full model fine-tuning, making them computationally expensive. 3 Method 3.1 Preliminaries Diffusion transformer architecture The Diffusion Transformer (DiT) architecture comprises sequential DiT blocks that transform input tokens into output tokens. Two main types of DiT blocks have been introduced: Stantdard DiT block 26 consists of Multi-Head Self-Attention (MHSA) layers and feed-forward layers, as illustrated in Figure 2a . Both layers apply LayerNorm to the incoming data and are modulated by a time embedding. The modulation is achieved using vectors \alpha,\beta,\gamma , which are generated by a distinct Multi-Layer Perceptron (MLP). Output of layers can be described by this formula: \begin{split}&x_{l}=x_{l-1}+\gamma_{1}MHSA(\alpha_{1}LN(x_{l-1})+\beta_{1}),\\ &x_{l+1}=x_{l}+\gamma_{2}FF(\alpha_{2}LN(x_{l})+\beta_{2}),\end{split} (1) where x_{l-1} denotes the input token sequence, and x_{l} and x_{l+1} represent the intermediate and final outputs. MM-DiT block 8 builds upon the structure of the Standard DiT block while introducing functionality for multimodal data processing. Specifically, MM-DiT combines textual and visual tokens via concatenation and processes them in parallel. Inter-modal communication is restricted to the MultiModal Attention Layer, enabling effective interaction between the two modalities. Separate modulation vectors are employed for each modality, denoted as \alpha^{v},\beta^{v},\gamma^{v} for visual tokens and \alpha^{t},\beta^{t},\gamma^{t} for textual tokens. Figure 2b provides a visual representation of the MM-DiT block structure, and the forward pass can be expressed as: \begin{split}&x_{l}^{v}=x_{l-1}^{v}+\gamma_{1}^{v}MHSA(\alpha_{1}^{v}LN(x_{l-1})+\beta_{1}^{v}),\\ &x_{l}^{t}=x_{l-1}^{t}+\gamma_{1}^{t}MHSA(\alpha_{1}^{t}LN(x_{l-1})+\beta_{1}^{t}),\\ &x_{l+1}^{v}=x_{l}^{v}+\gamma_{2}^{v}FF(\alpha_{2}LN(x_{l}^{v})+\beta_{2}^{v}),\\ &x_{l+1}^{t}=x_{l}^{t}+\gamma_{2}^{t}FF(\alpha_{2}LN(x_{l}^{v})+\beta_{2}^{t}),\end{split} (2) where x_{l}^{v} and x_{l}^{t} correspond to the transformed tokens for the visual and textual modalities, respectively. (a) DiT block scheme (b) MM-DiT block scheme Figure 2 : Illustration of DiT architectural components. (a) DiT block ablation (b) DiT block scaling Figure 3 : Motivational Experiment: Our findings reveal that the contribution of DiT blocks is not fully optimized. We demonstrate that their performance can be enhanced through a straightforward output scaling using a scalar multiplier. Figure 4 : Illustration of calibration parameter search procedure. 3.2 Motivation Previous works 1 ; 36 have

Read the complete text