Toward the Frontiers of Reliable Diffusion Sampling via Adversarial Sinkhorn Attention Guidance

Kwanyoung Kim · 2025

Read Toward the Frontiers of Reliable Diffusion Sampling via Adversarial Sinkhorn Attention Guidance by Kwanyoung Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07499.txt.

Introduction

Kwanyoung Kim seeks firmer ground for guidance methods that sharpen diffusion model outputs. Popular guidance improves a target result by deliberately worsening another, using hand-designed distortions. That trick works in practice but lacks a principled basis and can misfire.

Adversarial Sinkhorn Attention Guidance reframes attention scores through optimal transport, the math of moving mass at least cost. The Sinkhorn algorithm then disrupts that transport cost on purpose to steer sampling. Guidance becomes a defined operation rather than folklore. The aim is reliable sampling with fewer artifacts, giving practitioners a method they can reason about and tune with confidence.

Catalog record

CREATORKwanyoung Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07499

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Toward the Frontiers of Reliable Diffusion Sampling via Adversarial Sinkhorn Attention Guidance Kwanyoung Kim Abstract Diffusion models have demonstrated strong generative performance when using guidance methods such as classifier-free guidance (CFG), which enhance output quality by modifying the sampling trajectory. These methods typically improve a target output by intentionally degrading another, often the unconditional output, using heuristic perturbation functions such as identity mixing or blurred conditions. However, these approaches lack a principled foundation and rely on manually designed distortions. In this work, we propose A dversarial S inkhorn A ttention G uidance ( ASAG ), a novel method that reinterprets attention scores in diffusion models through the lens of optimal transport and intentionally disrupt the transport cost via Sinkhorn algorithm. Instead of naively corrupting the attention mechanism, ASAG injects an adversarial cost within self-attention layers to reduce pixel-wise similarity between queries and keys. This deliberate degradation weakens misleading attention alignments and leads to improved conditional and unconditional sample quality. ASAG shows consistent improvements in text-to-image diffusion, and enhances controllability and fidelity in downstream applications such as IP-Adapter and ControlNet. The method is lightweight, plug-and-play, and improves reliability without requiring any model retraining. Figure 1: Qualitative comparison. (a) unconditional generation, (b) conditional generation with other guidance sampling methods, and (c) conditional generation using ControlNet and IP-Adapter. Our method, ASAG , significantly improves visual quality in both unconditional and conditional settings. It also remarkably enhances external frameworks like ControlNet and IP-Adapter. Crucially, ASAG requires no additional training, making it broadly compatible and readily deployable. Introduction Diffusion models have led to substantial progress in the field of image and video generation ( Rombach et al. 2022 ; Esser et al. 2024a ; Ruiz et al. 2023 ; Chen et al. 2024b ; Xie et al. 2024 ; Blattmann et al. 2023 ; Chen et al. 2024a ) . Despite their effectiveness, direct or naïve sampling often results in subpar output quality. A widely adopted solution is Classifier-Free Guidance (CFG) ( Ho and Salimans 2022 ) , which enhances class-conditional generation by computing the difference between the score functions of conditional and unconditional models and applying a weighted adjustment. Although CFG improves sample fidelity, it introduces additional training and can lead to degraded outputs when the guidance scale is too large. Inspired by CFG, a number of guidance-based sampling techniques have emerged ( Hong et al. 2023 ; Karras et al. 2024 ; Ahn et al. 2025 ; Hong 2024 ; Chung et al. 2024 ; Sadat et al. 2024 ; Li et al. 2024 ) . A common strategy involves generating ”weakened outputs” to serve as auxiliary signals that guide the primary model toward better sampling. While such approaches have shown empirical success, they come with inherent limitations. For example, AutoGuidance (AG) ( Karras et al. 2024 ) relies on a poorly trained model, which is often unstable and difficult to optimize. To avoid retraining, attention based alternatives have been proposed. Perturbed Attention Guidance ( Ahn et al. 2025 ) distorts attention maps using identity masking, and Smooth Energy Guidance (SEG) ( Hong 2024 ) applies Gaussian blur to attention weights. Although these methods aim to weaken model outputs for improved guidance, they rely on naive heuristic functions and lack clear theoretical justification as to why such perturbations consistently enhance sample quality. This theoretical gap naturally raises a fundamental question: what constitutes an optimal perturbation for weakening outputs in a theoretically grounded manner? To address this question, we first revisit the intrinsic connection between attention mechanisms and optimal transport (OT) theory. One of the key contributions of this paper is the discovery and reinterpretation of classical results from optimal transport, highlighting how attention mechanisms can be framed within the OT framework. Prior studies, such as those by ( Sander et al. 2022 ; Kim, Oh, and Ye 2024 ) , have shown that attention computations can be improved by employing OT-inspired methods, specifically the Sinkhorn-Knopp algorithm ( Cuturi 2013 ) , to produce doubly stochastic attention maps. While these previous studies have largely utilized OT theory to boost attention performance in vision and language tasks ( Chen et al. 2020 ; Liu et al. 2020 ; Chen et al. 2022 ) , our approach deviates significantly from this established perspective. Specifically, we propose a novel guidance framework, termed Adversarial Sinkhorn Attention Guidance (ASAG), which explicitly reinterprets self-attention scores—representing pixel-level similarities and interactions—through the lens of OT theory. In contrast to classical OT-based approaches that minimize transport cost to encourage semantic alignment between image embeddings, ASAG adopts an adversarial perspective by intentionally minimizing their interactions. This leads to an entropy-maximizing attention map, which corresponds to an optimally perturbed self-attention distribution. In doing so, we establish a theoretically grounded method for systematically disrupting similarity-based alignments in diffusion models. To the best of our knowledge, this is the first work to leverage OT theory to construct an adversarial attention perturbation, leading to significant improvements in the fidelity and controllability of diffusion-based image generation. Building upon these theoretical foundations and insights, we demonstrate that our proposed method, ASAG, effectively improves generation quality in both unconditional and conditional diffusion sampling settings. Furthermore, extensive experiments show that when combined with existing frameworks such as ControlNet and IP-Adapter, ASAG consistently outperforms other guidance approaches by a significant margin. Our key contributions can be summarized: • We propose ASAG, a novel and theoretically grounded diffusion guidance method that adversarially disrupts attention mechanisms by intentionally minimizing interaction between image embeddings. • We provide an in-depth theoretical analysis, leveraging OT theory to establish a clear and rigorous foundation for our guidance method. To the best of our knowledge, our study is the first to reinterpret OT theory from an adversarial perspective for perturbing attention scores to improve diffusion generative performance. • We empirically demonstrate that ASAG significantly enhances performance across general generative tasks, including unconditional and conditional image generation. Furthermore, we show the broad applicability and generalizability of ASAG by integrating it with widely-used generative frameworks such as ControlNet and IP-Adapter, achieving substantial improvements over existing guidance approaches. Preliminary Diffusion Models Diffusion models (DM) ( Ho, Jain, and Abbeel 2020 ; Song et al. 2021 ) are a class of generative models that produce samples by reversing a known forward diffusion process through estimation of the score function of a given data distribution. Specifically, given samples \mathbf{x}_{0}\sim q_{\text{data}}(\mathbf{x}) , DMs define a forward noise-adding Markov process q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}),\quad t=1,\dots,T, where the variance schedule \{\beta_{t}\}_{t=1,\dots,T} is predefined. Consequently, the distribution at any intermediate timestep t can be explicitly expressed as q(\mathbf{x}_{t})=\mathcal{N}(\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0},(1-\bar{\alpha}_{t})\mathbf{I}), with the cumulative coefficient defined as \bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{i} , where \alpha_{t}=1-\beta_{t} . As t approaches T , the distribution q(\mathbf{x}_{T}) converges to an isotropic Gaussian distribution, q(\mathbf{x}_{T})\approx\mathcal{N}(0,\mathbf{I}). To generate samples, diffusion models parameterize the reverse diffusion process as, p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\boldsymbol{\mu}_{\theta}(\mathbf{x}_{t},t),\Sigma_{\theta}(\mathbf{x}_{t},t)), where the model parameters \theta can be optimized by denoising score matching (DSM) objective ( Vincent 2011 ) : \displaystyle\underset{\theta}{\min}\;\mathbb{E}_{\mathbf{x}_{0},\boldsymbol{\epsilon}}\left[\|\boldsymbol{\epsilon}_{\theta}(\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\boldsymbol{\epsilon},t)-\boldsymbol{\epsilon}\|^{2}_{2}\right],\quad (1) where \boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I}) . Once trained, sampling from a diffusion model proceeds by sequentially reversing the diffusion steps starting from an isotropic Gaussian noise sample. For instance, the Denoising Diffusion Implicit Model (DDIM) ( Song, Meng, and Ermon 2021 ) generates samples by iteratively applying the update rule: \displaystyle\mathbf{x}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{\mathbf{x}}_{0}(t)+\sqrt{1-\bar{\alpha}_{t-1}}\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_{t},t), (2) where the denoised estimate \hat{\mathbf{x}}_{0}(t)=\mathbb{E}[\mathbf{x}_{0}|\mathbf{x}_{t}] is

Read the complete text