MADI: Masking-Augmented Diffusion with Inference-Time Scaling for Visual Editing

Shreya Kadambi, Risheek GarrepalliShubhankar BorseMunawar Hyatt Fatih Porikli Qualcomm AI Research {skadambi, rgarrepa, sborse, mhayat, fporikli}@qti.qualcomm.com · 2025

Read MADI: Masking-Augmented Diffusion with Inference-Time Scaling for Visual Editing by Shreya Kadambi, Risheek GarrepalliShubhankar BorseMunawar Hyatt Fatih Porikli Qualcomm AI Research {skadambi, rgarrepa, sborse, mhayat, fporikli}@qti.qualcomm.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13401.txt.

Introduction

Shreya Kadambi and co-authors present MADI in 2025, a framework that improves diffusion models for visual editing. Diffusion models excel at generating images from text, but precise, grounded edits remain hard. Control over parts and composition is what real editing tools need.

MADI trains with a dual corruption process that mixes standard denoising with masked reconstruction. That teaches the model sharper, part-aware representations, enabling local edits that respect structure. Inference-time scaling adds further gains without retraining. Tests in the paper show better editability and composition than standard training. Designers may one day steer image models with the same ease as text models.

Catalog record

CREATORShreya Kadambi, Risheek GarrepalliShubhankar BorseMunawar Hyatt Fatih Porikli Qualcomm AI Research {skadambi, rgarrepa, sborse, mhayat, fporikli}@qti.qualcomm.com
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13401

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MADI: Masking-Augmented Diffusion with Inference-Time Scaling for Visual Editing Shreya Kadambi, Risheek GarrepalliShubhankar BorseMunawar Hyatt Fatih Porikli Qualcomm AI Research {skadambi, rgarrepa, sborse, mhayat, fporikli}@qti.qualcomm.com Abstract Despite the remarkable success of diffusion models in text-to-image generation, their effectiveness in grounded visual editing and compositional control remains challenging. Motivated by advances in self-supervised learning and in-context generative modeling, we propose a series of simple yet powerful design choices that significantly enhance diffusion model’s capacity for structured, controllable generation and editing. We introduce Masking-Augmented Diffusion with Inference-Time Scaling (MADI) , a framework that improves the editability, compositionality and controllability of diffusion models through two core innovations. First, we introduce Masking-Augmented gaussian Diffusion (MAgD) , a novel training strategy with dual corruption process which combines standard denoising score matching and masked reconstruction by masking noisy input from forward process. MAgD encourages the model to learn discriminative and compositional visual representations, thus enabling localized and structure-aware editing. Second, we introduce an inference-time capacity scaling mechanism based on Pause Tokens , which act as special placeholders inserted into the prompt for increasing computational capacity at inference time. Our findings show that adopting expressive and dense prompts during training further enhances performance, particularly for MAgD. Together, these contributions in MADI substantially enhance the editability of diffusion models, paving the way toward their integration into more general-purpose, in-context generative diffusion architectures . 1 Introduction Diffusion-based generative models ho2020denoising ; song2020scorebased ; rombach2022high have demonstrated impressive capabilities in synthesizing photorealistic images from text descriptions. Yet their effectiveness in precise, grounded image editing remains limited zhao2024ultraedit ; sheynin2024emu . Unlike text to image generation, editing is an inherently constrained and ill-posed inverse problem that demands fine-grained control over semantics, spatial composition, and alignment with both the input prompt and reference image. In this work, we dissect the architectural and training limitations of contemporary generative models for editing and identify three key desiderata essentials: (1) Compositional and Discriminative Visual Representations: The ability to understand scenes as an arrangement of distinct, modifiable local parts rather than a holistic entity, enabling targeted localized edits. (2) Fine-Grained Vision-Language Grounding: A precise alignment between visual features and text to accurately translate textual instructions into visual manipulations. (3) Sufficient Inference-Time Capacity: The computational power to navigate complex solution spaces in challenging inverse problems such as visual editing. To achieve these desirable characteristics in diffusion models for editing, we develop new strategies for both training (dual corruption training via masking) and inference (test-time capacity scaling via pause tokens). We hypothesize that current diffusion models, despite their success in generation rombach2022high , struggle with precise editing due to limitations in their learned representations. While these models can capture global scene structure well, they lack the granular discriminative features zhang2023tale vital for localized manipulation and robust grounding. Features learned via Masked reconstruction oquab2023dinov2 ; he2022masked complement gaussian diffusion with better local understanding zhang2023tale . In our work, we combine the benefits of both gaussian diffusion rombach2022high ; xiao2024omnigen and masked reconstruction xie2024show ; hu2024mask . Dual-Corruption Training via Masking-Augmented Gaussian Diffusion (MAgD) : We develop a novel training method with dual forward process by first adding gaussian noise and then masking out input tokens of noisy-image, which is trained with standard denoised score matching objective. This dual-corruption strategy compels the model to develop highly discriminative, localized features through the imperative of contextual infilling from the masking task, thereby fostering compositional visual representations (Desideratum 1) while retaining the capacity for high-fidelity, globally coherent synthesis from the diffusion process. The resultant representations are further refined by training with rich and descriptive prompts (synthetically generated) resulting in improved fine-grained vision-language grounding (Desideratum 2) aided by discriminative visual representations towards enhanced ability to execute complex, localized edits. Inference-Time Capacity Scaling via Pause Tokens : Motivated by inference-time scaling techniques in large language models, such as Chain-of-Thought prompting wei2022chain and Scratchpad reasoning nye2021show , we introduce a novel mechanism for adaptive inference-time capacity scaling for in-context image generative architecture via Pause Tokens goyal2023think . These special tokens embedded within the inference process allow the model to dedicate additional compute to refine its understanding and execute complex edits (Desideratum 3). This effectively boosts model’s capacity at inference, enabling it to navigate challenging solution spaces and achieve more precise, coherent, and contextually aware visual manipulations. In Summary, our work introduces and evaluates novel training and inference techniques addressing key desiderata for visual editing with following contributions: • We introduce a dual corruption training that synergistically combines masked image modeling and noise-based denoising. This enables MAgD to learn discriminative and compositional visual representations, demonstrably enhancing fine-grained vision-language grounding critical for precise editing within computationally and data efficient setting. • Inference-Time Scaling with Pause Tokens, a mechanism that enables diffusion based image generative models to scale in-context capacity at inference , improving their ability to solve harder, grounded editing tasks (without requiring any retraining). • We empirically establish the effectiveness of MAgD and expressive prompt-based training through comprehensive experiments. Our evaluations span diverse benchmarks, including IdeaBench and Complex-Edit suite featuring expressive prompts, enabling robust assessment of zero-shot editing capabilities. Crucially, our evaluation methodology also captures inherent trade-off between faithfulness and instruction-following within visual editing. (a) (b) Figure 1 : a) Comparison of Standard Denoising (Top) and MAgD Objective (Bottom) . The top row illustrates standard denoising, The bottom row depicts the MAgD objective, where the target image is randomly masked, and noise is then added to the entire image before denoising. b) Encapsulating Pause tokens for Inference time scaling : The red box in the figure denotes the impact of additional "thinking time" on latent processing steps without additional training 2 Related work Masked reconstruction approaches such as MAE he2022masked shows strong representation learning via masked reconstruction. While MaskGIT chang2022maskgit ; chang2023muse use masked diffusion for synthesis.In contrast, we integrate masked modeling within the continuous diffusion training loop, introducing a dual forward process that better supports partial reconstructions for editing.A closest to our work zheng2023fast focusses on training time complexity and not representational properties or editing. Editing specific models, inpainting models: Visual editing, like "make two people look at each other", is a complex inverse problem requiring identifying regions, modifying them, and maintaining consistency. Standard diffusion models (UNet, DiT) often struggle with the dynamic capacity and context reuse needed for such grounded manipulations. In contrast, in-context architectures inspired by LLMs, such as OmniGen xiao2024omnigen , are better suited due to their ability to retrieve and integrate reference information during generation. Dense prompt for strong vision-language grounding: LLM-inspired visual models such a qi2025cogcom . fang2025gotunleashingreasoningcapability , xu2025llavacotletvisionlanguage are emerging as promising frameworks that have shown that integrating dense prompts through CoT style prompting is beneficial for understanding and generation. These architectures treat image generation and editing as a conditioned sequence modeling problem, offering flexible in-context control. Our methods are designed to complement such architectures enhancing their editing capabilities via richer training objectives and inference-time flexibility. Figure 2 : Baseline vs MAgD-I (w/o inference scaling) vs MAgD-I . While baseline follows the prompt, it destroys the scene composition in (top) and (bottom). In (middle) we observe that the object appearance is also modified. While MAgD-I w/o scaling alleviates this problem and retains the scene composition. We observe that with

Read the complete text