Self-Attention Decomposition for Training Free Diffusion Editing Tharun Anand, Mohammad Hassan Vali, Arno Solin, Green Rosh, B H Pawan Prasad Abstract Diffusion models achieve remarkable fidelity in image synthesis, yet precise control over their outputs for targeted editing remains challenging. A key step toward controllability is to identify interpretable directions in the model’s latent representations that correspond to semantic attributes. Existing approaches for finding interpretable directions typically rely on sampling large sets of images or training auxiliary networks, which limits efficiency. We propose an analytical method that derives semantic editing directions directly from the pretrained parameters of diffusion models, requiring neither additional data nor fine-tuning. Our insight is that self-attention weight matrices encode rich structural information about the data distribution learned during training. By computing the eigenvectors of these weight matrices, we obtain robust and interpretable editing directions. Experiments demonstrate that our method produces high-quality edits across multiple datasets while reducing editing time significantly by 60% over current benchmarks. † † address: 1 Samsung R&D Institute India Bangalore 2 Department of Computer Science, Aalto University, Finland Figure 1 : Illustration of editing capabilities with our method. (a) The first row demonstrates the linear editing property as the perturbation strength \alpha is varied. (b) The second row highlights the ability of our method to perform precise edits to facial attributes. 1 Introduction Diffusion models [ 1 ] have emerged as the state-of-the-art for generative modeling, producing diverse, high-fidelity images by iteratively denoising random noise. A crucial application is to control their internal representations to support inference-time tasks such as semantic image editing. However, unlike single-pass generators, the multi-step, iterative nature of the diffusion process results in a unstructured latent space, where semantics are difficult to disentangle. To address this, prior works have explored various techniques for identifying interpretable editing directions. Some methods focus on semantic information within bottleneck features (e.g., the h-space framework) [ 2 ] , while others operate directly on the noise latent space [ 3 , 4 ] . Some other employ techniques like Riemannian geometry [ 5 ] and contrastive learning [ 6 ] to find robust editing vectors. However, these approaches face two key limitations. First, they typically rely on sampling a large number of images from the latent space, causing the discovered directions to be inherently tied to the sampled data distribution. This dependency can introduce a bias and limit the generalization to unseen images. Second, discovering such directions often involves computationally expensive operations [ 3 , 5 ] or additional model training [ 6 , 2 ] . These limitations substantially increase the editing time. To address these limitations, we propose a training-free method to discover interpretable semantic directions directly from the parameters of pretrained diffusion models. Motivated by the closed-form factorization technique [ 7 ] in GANs that derive semantic directions from linear weight matrices, we extend this paradigm to the more complex architecture of diffusion models, an area that has been largely unexplored. We address this gap by examining the pretrained Query , Key , and Value projection matrices of the self-attention layers. Our core hypothesis is that the self-attention layers of the diffusion models, which are known to encode rich structural and semantic information [ 8 , 9 ] , can be utilized to extract interpretable editing directions. Hence, we perform eigen-decomposition on the pretrained query (Q), key (K), and value (V) projection matrices within these layers. The computed eigenvectors serve as robust, sample-independent editing directions that align with prominent data attributes such as age, gender, or smile (for a pretrained face model). In summary, this paper makes these key contributions: 1. To the best of authors knowledge, we are the first to derive interpretable editing directions directly from the pretrained weights of diffusion models. We achieve this through an eigen-analysis of the self-attention weights, extracting eigenvectors without any additional training. 2. We provide a theoretical derivation based on sensitivity of the self-attention operation, demonstrating that principal components of the combined query-key-value weight matrices yield semantically meaningful directions. 3. We empirically validate our method on facial attribute editing covering gender, age, and expression. Our results demonstrate that our sample-independent directions generalize effectively across different images, achieving significant reduction in latency. 2 Proposed Method 2.1 Preliminaries Problem Formulation: Diffusion models typically employ a U-Net backbone with self-attention layers to capture long-range dependencies during denoising. At timestep t , let Z_{t}\in\mathbb{R}^{H\times W\times C} denote the latent feature map entering a self-attention block. We reshape Z_{t} into N=H\,W tokens of dimension d\ =C , forming Z_{t}\in\mathbb{R}^{N\times d} . For brevity, we drop the timestep subscript and refer to this matrix simply as Z . Within a self-attention block, three pretrained projection matrices of W_{Q},\>W_{K},\>W_{V}\in\mathbb{R}^{d\times d} map the input representation Z into queries, keys, and values as Q=ZW_{Q},\qquad K=ZW_{K},\qquad V=ZW_{V}\;. (1) We further decompose the attention computation into intermediate components of L=\frac{QK^{\!\top}}{\sqrt{d}},\quad S=\mathrm{softmax}(L),\quad\mathrm{Attn}(Z)=SV\;. (2) To uncover meaningful editing directions, we analyze the effect of perturbing the latent features along a candidate direction n\in\mathbb{R}^{N\times d} . Specifically, we introduce a perturbation: Z^{\prime}=Z+\alpha\,n, (3) Let \alpha\ll 1 and n\in\mathbb{R}^{N\times d} with n_{i} its i -th row, \|n_{i}\|_{2}=1 . The change in the attention output is: \Delta\mathrm{Attn}=\mathrm{Attn}(Z^{\prime})-\mathrm{Attn}(Z). (4) Our objective is to derive directions n^{*} that maximize the impact on the attention output, thus directly correspond to axes of semantic variation [ 10 ] : n^{*}=\arg\max_{n\in\mathbb{R}^{N\times d},|n_{i}\|_{2}=1}\|\Delta\mathrm{Attn}\|_{2}^{2}. (5) Latent Whitening: Our derivation is based on the observation that the U-Net’s internal representations are whitened, i.e., their covariances are approximately identity during early denoising steps. This extends the empirically observed latent whitening effect, where intermediate feature covariances approach identity [ 11 , 12 ] and enables the key analytical simplification \mathbb{E}[X^{\top}X]\approx I for our intermediate matrices. 2.2 Weight space Eigen decomposition Given a small perturbation to the input latents: Z^{\prime}=Z+\alpha n,\qquad\|n_{i}\|_{2}=1,\qquad\alpha\text{ is small}, (6) This induces perturbations in attention projections: \Delta Q=\alpha nW_{Q},\quad\Delta K=\alpha nW_{K},\quad\Delta V=\alpha nW_{V} (7) To calculate the change in score matrix \Delta L induced by the pertubration, while neglecting the \mathcal{O}(\alpha^{2}) term: \displaystyle\Delta L \displaystyle=\frac{1}{\sqrt{d}}\bigl(Q\,\Delta K^{\!\top}+\Delta Q\,K^{\!\top}\bigr). (8) Applying the first-order Taylor expansion of the softmax function S to obtain \Delta S : \displaystyle\Delta S \displaystyle=J_{\mathrm{softmax}}(L)\,\Delta L, (9) where J_{\mathrm{softmax}}(L) is the Jacobian of softmax at L . Now, since \mathrm{Attn}(Z)=SV , \Delta\mathrm{Attn} decomposes as: \displaystyle\Delta\mathrm{Attn} \displaystyle=\Delta S\,V+S\,\Delta V (10) Objective Function. Maximize \Delta\mathrm{Attn} as defined in ( 5 ). \displaystyle\|\Delta\mathrm{Attn}\|_{F}^{2}=\|\Delta S\,V+S\,\Delta V\|_{F}^{2}. (11) Upon simplification: \displaystyle\bigl\|\Delta S\,V+S\,\Delta V\bigr\|_{F}^{2} \displaystyle=\bigl\|\Delta S\,V\bigr\|_{F}^{2}+\bigl\|S\,\Delta V\bigr\|_{F}^{2} (12) \displaystyle+2\,\mathrm{tr}\!\bigl((\Delta S\,V)^{\top}(S\,\Delta V)\bigr) Using whitening approximation (Sec. 2.1 ), which makes query–key and value perturbations nearly uncorrelated, we drop the cross term: \displaystyle\mathbb{E}\|\Delta\mathrm{Attn}\|_{F}^{2}\approx\mathbb{E}\|\Delta S\,V\|_{F}^{2}+\mathbb{E}\|S\,\Delta V\|_{F}^{2}. (13) Term 1: For the first term, we utilize the cyclic property of the trace of a matrix: \|\Delta S\,V\|_{F}^{2}=\text{tr}(V^{\top}(\Delta S)^{\top}\Delta S\,V)=\text{tr}((\Delta S)^{\top}\Delta S\,V.V^{\top}). (14) From equations ( 8 ) and ( 16 ), we have \Delta S : \displaystyle\Delta S=\frac{\alpha}{\sqrt{\mathrm{d}}}\,\bigl[QW_{K}^{\top}n+KW_{Q}^{\top}n\bigr]. (15) With approximation ( 2.1 ), the latent-dependent covariance reduces to identity ( \mathbb{E}[VV^{\top}]\approx I ), isolating weight matrices: \displaystyle\mathbb{E}[\|\Delta S\,V\|_{F}^{2}]\approx\alpha^{2}n^{\top}(W_{Q}^{\top}W_{Q}+W_{K}^{\top}W_{K})n. (16) Term 2: \displaystyle\|S\,\Delta V\|_{F}^{2}=\text{tr}((\Delta V)^{\top}S^{\top}S\,\Delta V). (17) Since \mathbb{E}[S^{\top}S]\approx I , (Sec 2.1 ). The term then reduces to : \displaystyle\mathbb{E}[\|S\,\Delta V\|_{F}^{2}]\approx\alpha^{2}n^{\top}W_{V}W_{V}^{\top}n. (18) Figure 2 : Our method supports a diverse range of edits with improved disentanglement, enabling precise single-step manipulation of the target attribute beyond existing benchmarks. Final Expression: Combining terms 1 & 2 yields, \mathbb{E}\bigl\|\Delta\mathrm{Attn}\bigr\|_{F}^{2}\approx\alpha^{2}n^{\top}\Bigl(W_{Q}^{\top}W_{Q}+W_{K}^{\top}W_{K}+W_{V}W_{V}^{\top}\Bigr)n (19) Define the combined matrix: \displaystyle C=W_{Q}^{\top}W_{Q}+W_{K}^{\top}W_{K}+W_{V}^{\top}W_{V}. (20) Now maximizing n^{\top}C\,n subject to \|n\|_{2}=1 is a standard Rayleigh quotient problem. The optimal direction n^{*} is the principal eigenvector of C . Subsequent eigenvectors provide orthogonal editing directions. As shown in prior work [ 13 , 3 ] , editing vectors are most effective when applied at early denoising timesteps, before image content forms, where the whitening assumption \mathbb{E}[ZZ^{\top}]\approx I holds. Following a similar strategy to prior works, we empirically choose the range 0.8T