Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective

Hyunmin Cho, Woo Kyoung Han, Kyong Hwan Jin · 2026

Read Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective by Hyunmin Cho, Woo Kyoung Han, Kyong Hwan Jin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27476.txt.

Introduction

In their 2026 paper, Hyunmin Cho and colleagues treat attention as a form of memory. They split the pre-softmax matrix into symmetric and skew parts, linking the first to the energy landscape and the second to circulation. Measures of stability drawn from the symmetric part track how well the model balances faithful recall against varied output, and tweaking circulation gives a control knob for it.

Catalog record

CREATORHyunmin Cho, Woo Kyoung Han, Kyong Hwan Jin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62299
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27476

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective Hyunmin Cho, Woo Kyoung Han, Kyong Hwan Jin Abstract We characterize the pre-softmax attention matrix 𝐐𝐊 ⊤ \mathbf{QK^{\top}} in transformers as an associative memory matrix encoding pairwise associations between input features. By decomposing this matrix into its symmetric and skew-symmetric parts, we interpret the symmetric component as governing the structure of the energy landscape , and the skew-symmetric component as driving circulation on that landscape. Leveraging the energy formulation induced by the symmetric component, we derive Hopfield-style stability measures that quantify the stability of retrieved features. We observe meaningful correlations between Hopfield-style stability measures and the fidelity–diversity trade-offs in generation. Finally, we propose a controllable knob to modulate this trade-off by modifying the circulation of the underlying dynamics. Code is available at our GitHub . 1 Introduction Diffusion models ( Ho et al. 2020 ; Rombach et al. 2022 ; Podell et al. 2024 ; Esser et al. 2024 ; Labs et al. 2025 ) have become a leading paradigm for image generation. Their success is largely driven by the attention mechanism ( Vaswani et al. 2017 ) , which enables the integration of global context and long-range dependency modeling throughout the denoising process ( Nichol & Dhariwal 2021 ) . While this global connectivity facilitates richer compositional associations that enhance novelty and variety ( Zhang et al. 2019 ) , it is simultaneously prone to causing spurious mixing of incompatible features, such as the blending of materials between two distinct objects ( Oriyad et al. 2025 ) . Crucially, distinguishing between such beneficial context integration and harmful semantic leakage remains non-trivial, as they share the same underlying mechanism. To address this ambiguity, our goal is to (i) identify when attention settles into spurious mixtures, and (ii) control this behavior to navigate the trade-off between coherent structure and diversity. Low \mathbf{E} (Stable) High \mathbf{E} (Unstable) Figure 1 : Skew perturbation and the fidelity–diversity trade-off. Top : We decompose \mathbf{QK^{\top}} into symmetric (energy) and skew (circulation) parts. (a) The symmetric part gives stable but low-diversity retrieval. (b) Moderate skew perturbation breaks metastable mixtures while preserving stable states. (c) Excessive perturbation destabilizes even well-formed retrievals, producing artifacts. Bottom : Moderate skew perturbation improves diversity, but excessive perturbation causes hallucinations. / denote positive/negative diversity ; / denote positive/negative fidelity . Figure 2 : Associative memory framework encoding pairwise feature interactions and its decomposition. (a)–(b) We characterize the attention mechanism as an associative memory encoding pairwise feature interactions . (a) Viewing input features \textbf{X}\in\mathbb{R}^{L\times d_{\rm in}} as a set of features \boldsymbol{x}^{(i)}\in\mathbb{R}^{L} , (b) the learned interaction matrix W encodes the association strength between these feature pairs. (c) The resulting attention matrix \mathbf{QK^{\top}} can be decomposed into a symmetric component and a skew-symmetric component . The symmetric term defines a static energy landscape governing the stability of retrieved features, while the skew-symmetric term drives circulation , acting as a directional force. (d) Control via Circulation. Standard retrieval often settles into metastable mixtures (e.g., the incoherent ‘Three legs’ structure). We propose using the skew-symmetric component as a controllable knob. Amplifying this component injects circulation-driven drift to perturb the metastable state, restoring structural coherence . The perspective of associative memory provides a principled lens on these challenges ( Amari 1972 ; Nakano 1972 ; Little 1974 ; Hopfield 1982 ) . Recent dense associative memory work further suggests that the choice of energy function can substantially reshape the landscape of local minima, even giving rise to additional emergent memories beyond stored patterns ( Hoover et al. 2026 ) . Building on the insight that transformer self-attention approximates the update rule of a modern Hopfield network ( Ramsauer et al. 2021 ) , we re-frame spurious mixing as entrapment in metastable states (local energy minima where the model settles on an incoherent combination of distinct patterns). However, standard analyses typically operate at a token-wise level, treating attention merely as a retrieval mechanism. This token-centric view restricts the capture of the rich interaction dynamics encoded in the attention matrix itself. Furthermore, these interpretations often overlook the dynamical consequences of asymmetric association matrices. In recurrent associative memories, such asymmetry is known to reshape the attractor structure, allowing non-fixed-point attractors such as limit cycles ( Hwang et al. 2019 ) . This structural property is crucial, as it induces circulation that helps perturb and destabilize metastable mixtures ( Singh et al. 1995 ; Chengxiang et al. 2000 ) . In this work, we characterize the attention matrix \mathbf{QK^{\top}} as a dynamic associative memory that encodes pairwise feature associations ( Figure 2 ). Unlike prior token-level analyses, our view exposes the association structure that governs the mixing dynamics. Concretely, we decompose \mathbf{QK^{\top}} into a symmetric and a skew component: the symmetric component defines a Hopfield-style energy landscape . In contrast, the skew-symmetric component drives circulation , acting as a directional force to perturb metastable states ( Figure 1 ). This decomposition reveals that generation quality hinges on the balance between energy-based stability and circulation-driven dynamics. Leveraging this insight, we derive Hopfield-style stability measures, enabling us to identify metastable mixtures (Goal (i)). Finally, we exploit the skew-symmetric circulation as a tunable knob to control the retrieval process, facilitating the perturbation of metastable mixtures (Goal (ii)). To summarize our contributions: • We establish an associative memory framework that encodes pairwise feature associations for the attention matrix and introduce a symmetric/skew-symmetric decomposition that disentangles energy-based stability from circulation-driven drift. • Leveraging the symmetric component, we derive Hopfield-style stability measures that quantify the stability of retrieved features, demonstrating their correlation with the fidelity–diversity trade-off ( Table 1 ). • We propose the skew-symmetric component as a controllable ‘circulation knob’ for test-time intervention, which injects directional drift to perturb metastable mixtures and restore structural coherence ( Table 3 ). 2 Related Work Denoising diffusion models generate samples by learning to invert a progressive noising process, initially introduced in Sohl-Dickstein et al. 2015 and popularized as DDPMs in Ho et al. 2020 . Subsequent formulations unify diffusion with score matching and continuous-time SDE/ODE views ( Song & Ermon 2019 ; Song et al. 2021 ) , and related continuous-time objectives such as flow matching regress vector fields that transport noise to data ( Lipman et al. 2023 ; Liu et al. 2023 ) . Complementing these algorithmic formulations, recent theoretical works have reinterpreted these generative dynamics through the lens of associative memory, analyzing how diffusion trajectories disperse information and balance memorization with generalization ( Ambrogioni 2023 ; Hoover et al. 2023 ; Pham et al. 2025 ) . Associative memory networks are grounded in the classical Hopfield network, which defines an energy landscape over binary states. In these models, the system evolves to minimize energy based on the local field inputs ( Amari 1972 ; Nakano 1972 ; Little 1974 ; Hopfield 1982 ) . To overcome the storage limitations inherent to these classical pairwise-interaction models, Krotov & Hopfield 2016 introduced Dense Associative Memories (DAMs), which generalize the energy function by replacing the quadratic interaction term with a rapidly growing nonlinear function (e.g., polynomial or exponential) defined over the stored patterns. The gradient of this energy governs the update dynamics, resulting in sharper basins

Read the complete text