Attention Frequency Modulation: Training-Free Spectral Modulation of Diffusion Cross-Attention

Seunghun Oh, Unsang Park · 2026

Read Attention Frequency Modulation: Training-Free Spectral Modulation of Diffusion Cross-Attention by Seunghun Oh, Unsang Park (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28114.txt.

Introduction

Oh and Park study how text steers image generation inside diffusion models, through cross-attention. They treat attention as a signal on the latent grid and track its Fourier power across denoising steps. A steady coarse-to-fine pattern emerges, a fingerprint of how tokens compete for space. Their method, AFM, reweights low and high frequency bands of the attention logits at inference time, with no retraining. On Stable Diffusion, AFM shifted image layout strongly while keeping the prompt's meaning.

Catalog record

CREATORSeunghun Oh, Unsang Park
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37254
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28114

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Attention Frequency Modulation: Training-Free Spectral Modulation of Diffusion Cross-Attention Seunghun Oh, Unsang Park Abstract Cross-attention is the primary interface through which text conditions latent diffusion models, yet its step-wise multi-resolution dynamics remain under-characterized, limiting principled training-free control. We cast diffusion cross-attention as a spatiotemporal signal on the latent grid by summarizing token-softmax weights into token-agnostic concentration maps and tracking their radially binned Fourier power over denoising. Across prompts and seeds, encoder cross-attention exhibits a consistent coarse-to-fine spectral progression, yielding a stable time–frequency fingerprint of token competition. Building on this structure, we introduce Attention Frequency Modulation (AFM), a plug-and-play inference-time intervention that edits token-wise pre-softmax cross-attention logits in the Fourier domain: low- and high-frequency bands are reweighted with a progress-aligned schedule and can be adaptively gated by token-allocation entropy, before the token softmax. AFM provides a continuous handle to bias the spatial scale of token-competition patterns without retraining, prompt editing, or parameter updates. Experiments on Stable Diffusion show that AFM reliably redistributes attention spectra and produces substantial visual edits while largely preserving semantic alignment. Finally, we find that entropy mainly acts as an adaptive gain on the same frequency-based edit rather than an independent control axis. 1 Introduction Diffusion-based text-to-image models achieve state-of-the-art synthesis by progressively denoising text-conditioned latent representations [ 5 , 18 ] . Despite their success, the inference-time internal dynamics that govern how global layout, fine detail, and sample-to-sample variability emerge over denoising steps remain under-explored [ 6 , 10 ] . This gap is not only an interpretability issue but also a controllability bottleneck: widely used controls such as prompt engineering and guidance tuning often act as user-facing heuristics that entangle multiple factors, offering limited insight into what changes inside the model and when those changes occur [ 6 , 3 , 1 ] . A central yet under-explored component is cross-attention, which injects textual conditioning into the U-Net at multiple resolutions and at every denoising step [ 18 , 24 ] . Because it is repeatedly applied along the diffusion trajectory, cross-attention is a natural carrier of stage-dependent signals (e.g., layout formation early vs. detail refinement late) [ 18 , 24 ] . However, existing analyses typically focus on attention heatmaps at a few timesteps or token relevance scores, leaving the time-resolved organization across spatial scales unclear and limiting principled, training-free control. We ask whether cross-attention exhibits a consistent multi-scale organization over denoising, and whether that structure can be used for controllable, inference-time edits. Hypothesis and approach. We hypothesize that diffusion cross-attention exhibits a robust coarse-to-fine spectral progression over denoising, and that steering this progression in logit space provides a training-free knob that biases token-competition toward coarser or finer spatial patterns (as measured by our attention-spectrum diagnostics), rather than directly measuring or guaranteeing image-level layout/detail disentanglement. To test this hypothesis, we (i) extract a token-agnostic attention concentration map from cross-attention and summarize its spectrum over sampling progress, and (ii) intervene by frequency-selective, token-wise reweighting of pre-softmax cross-attention logits. Empirically, encoder cross-attention exhibits the most stable and monotonic spectral trajectory across prompts and seeds, so we use it as the primary locus for both analysis and intervention, while still reporting downstream effects in middle/decoder blocks. A frequency-domain view. We operationalize cross-attention as a spatial signal on the latent grid by mapping each step’s token distribution to a token-agnostic concentration statistic (top- K ) and analyzing its radially binned Fourier power. This yields a compact time–frequency fingerprint that is stable across prompts and random seeds, revealing a consistent coarse-to-fine evolution inside the denoising trajectory. Training-free control via AFM. Motivated by this fingerprint, we introduce Attention Frequency Modulation (AFM), a plug-and-play inference-time intervention that applies frequency-selective reweighting to token-wise pre-softmax cross-attention logits. AFM provides an interpretable control handle by reshaping token competition patterns in logit space over sampling progress, without retraining or architectural modification. We emphasize that AFM does not “inject” high-frequency detail directly. Instead, it perturbs token competition in logit space, which can steer the observed coarse-to-fine spectral progression measured on post-softmax attention summaries. Unless otherwise stated, we apply AFM to encoder cross-attention modules, where the spectral trajectory is most stable, and report downstream effects in middle/decoder blocks. (a) Baseline (b) SAG (c) FreeU (d) Ours Figure 1 : Qualitative comparison on Stable Diffusion v1.5 under matched sampling settings (same prompt/seed). (a) Baseline, (b) SAG, (c) FreeU, (d) Ours (AFM). Contributions. (i) Frequency characterization of diffusion cross-attention. We characterize cross-attention using a token-agnostic concentration signal (top- K mean) and track its spectral evolution over denoising, providing a stable fingerprint of coarse-to-fine token competition across prompts and random seeds. (ii) Training-Free Attention Frequency Modulation (AFM). We propose an inference-time method that directly intervenes on cross-attention logits in the frequency domain, applied token-wise before softmax. AFM-curve suppresses late-stage high-frequency fragmentation in encoder attention (as measured by the post-softmax top- K spectrum / \rho_{s} ) while preserving alignment, indicating controllability over the internal progression. (iii) Entropy as a secondary gating signal. We analyze attention entropy as a complementary statistic capturing concentration/dispersion. When paired with AFM, entropy gates the effective intervention strength and influences variability distributionally. Related work. Training-free diffusion control. Prior work modifies diffusion inference without retraining, including attention-based editing/excitation and feature-space interventions [ 3 , 14 , 1 , 23 , 7 , 21 , 2 , 12 ] . Related controllable generation and parameter-efficient adaptation methods include ControlNet, T2I-Adapter, and LoRA [ 26 , 15 , 8 ] . We include SAG and FreeU as representative training-free baselines for direct comparison in our experiments. In contrast, we first characterize a stable frequency structure in cross-attention over timesteps and then derive a frequency-aligned, logit-space intervention. We choose SAG and FreeU as representative training-free interventions that modify the sampling dynamics without additional training or external models, matching our plug-and-play setting. Other editing methods that rely on prompt-specific inversion or extra optimization steps are orthogonal and not directly comparable under our fixed-sampling paired protocol. Frequency perspectives and mechanistic analyses. Frequency-domain tools and mechanistic analyses have been used to probe representation bias and diffusion dynamics [ 17 , 22 , 13 , 25 , 9 ] . Entropy and information measures. Entropy has long been used to quantify dispersion in probabilistic assignments [ 20 ] . We show entropy mainly acts as a gating statistic that modulates the effective strength/variability of frequency-based interventions, rather than defining an orthogonal control axis. 2 Background 2.1 Diffusion models and cross-attention along the denoising trajectory Diffusion models synthesize images via an iterative denoising process [ 5 ] . Latent diffusion models (LDMs) perform this process in a learned latent space for efficient high-resolution generation [ 18 ] . At denoising step t , given latent x_{t} and text condition c , the reverse update is x_{t-1}=f_{\theta}(x_{t},t,c), (1) where f_{\theta} is typically a U-Net with multi-resolution blocks: the downsampling path (encoder), the bottleneck (middle), and the upsampling path (decoder). Text conditioning is injected through cross-attention layers. With queries Q from latent features and keys/values K,V from text embeddings, a standard cross-attention operation is \mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d}}\right)V. (2) Because cross-attention is applied at every denoising step and at multiple spatial resolutions, it naturally admits a stage-dependent view of generation dynamics. In this work, we treat cross-attention as a temporally evolving internal

Read the complete text