FreeFlux: Understanding and Exploiting Layer-Specific Roles in RoPE-Based MMDiT for Versatile Image Editing

Tianyi Wei, Yifan Zhou, Dongdong Chen, Xingang Pan · 2025

Read FreeFlux: Understanding and Exploiting Layer-Specific Roles in RoPE-Based MMDiT for Versatile Image Editing by Tianyi Wei, Yifan Zhou, Dongdong Chen, Xingang Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16153.txt.

Introduction

Wei, Zhou, Chen, and Pan take apart a leading image generator to learn which layers care about position and which care about content, in a 2025 paper. The model family uses rotary position embeddings inside its attention layers. By selectively scrambling those embeddings during generation, the authors probe each layer's dependence on place versus appearance.

The dependency pattern does not simply track depth, a surprise. They turn the insight into a training-free editing framework that sorts edits into three kinds: position-driven, content-driven, and region-preserving. Each kind gets its own injection strategy. Tests show cleaner edits than rival methods, especially at keeping the original meaning intact.

Catalog record

CREATORTianyi Wei, Yifan Zhou, Dongdong Chen, Xingang Pan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16153

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FreeFlux: Understanding and Exploiting Layer-Specific Roles in RoPE-Based MMDiT for Versatile Image Editing Tianyi Wei, Yifan Zhou, Dongdong Chen, Xingang Pan Abstract The integration of Rotary Position Embedding (RoPE) in Multimodal Diffusion Transformer (MMDiT) has significantly enhanced text-to-image generation quality. However, the fundamental reliance of self-attention layers on positional embedding versus query-key similarity during generation remains an intriguing question. We present the first mechanistic analysis of RoPE-based MMDiT models (e.g., FLUX), introducing an automated probing strategy that disentangles positional information versus content dependencies by strategically manipulating RoPE during generation. Our analysis reveals distinct dependency patterns that do not straightforwardly correlate with depth, offering new insights into the layer-specific roles in RoPE-based MMDiT. Based on these findings, we propose a training-free, task-specific image editing framework that categorizes editing tasks into three types: position-dependent editing (e.g., object addition), content similarity-dependent editing (e.g., non-rigid editing), and region-preserved editing (e.g., background replacement). For each type, we design tailored key-value injection strategies based on the characteristics of the editing task. Extensive qualitative and quantitative evaluations demonstrate that our method outperforms state-of-the-art approaches, particularly in preserving original semantic content and achieving seamless modifications. Figure 1: Leveraging the layer-specific roles in RoPE-based MMDiT we discovered, versatile training-free image editing is tailored to different task characteristics, including non-rigid editing, object addition, background replacement, object movement, and outpainting. 1 Introduction After years of rapid development, diffusion [ 10 , 34 ] -based text-to-image generation models have become the default modeling paradigm. Recently, new models, exemplified by FLUX [ 16 ] and Stable Diffusion 3 [ 5 ] , have pushed generation quality to an unprecedented level. Compared to previous state-of-the-art text-to-image models such as SD 1 [ 30 ] , SD 2, and SDXL [ 25 ] , these models share several key improvements: a rectified flow formulation [ 20 ] , a novel Multimodal Diffusion Transformer (MMDiT) [ 24 ] replacing the conventional U-Net architecture [ 31 ] , and a unified information interaction scheme where text and image tokens are concatenated and processed through self-attention with the classic query-key-value design. A key distinction between FLUX and SD3 lies in how positional information is incorporated. SD3 injects Positional Embedding only at the input layer of the network, whereas FLUX applies Rotary Position Embedding (RoPE) [ 35 ] to both queries and keys at every self-attention layer. This more explicit encoding of absolute and relative positional information enables FLUX to achieve superior generation quality and better high-resolution extrapolation, positioning it as a standout model in the text-to-image generation field. This naturally leads us to ask an important question: During generation, when each layer performs self-attention, does FLUX rely more on the similarity between queries and keys, or on the positional embedding? To answer this question, we propose a novel automated strategy to probe the dependence of different self-attention layers on positional relationships during generation. Specifically, during sampling, we manipulate each self-attention layer in FLUX by preserving the RoPE for queries while either removing or shifting the RoPE for keys, thereby obtaining the corresponding generated results. By measuring the similarity between the original sampled results and the modified outputs, we can infer the functional role of each layer: lower similarity indicates a stronger reliance on positional relationships, whereas higher similarity suggests a greater dependence on the content similarity between queries and keys. Surprisingly, we find that layers relying on positional information and those relying on content similarity do not exhibit a simple correlation with their indices within the network. With these dependency patterns as a guideline, we suggest that training-free image editing based on FLUX should be tailored to the specific characteristics of the editing task. Versatile image editing shares a classic editing mechanism [ 4 ] : it generates both the source image and the edited image in parallel. During generation, editing is achieved by injecting keys and/or values from the source image into the edited image. Based on the nature of different editing tasks, we categorize versatile image editing into three types and design corresponding injection strategies for them: (1) Position-Dependent Editing , (2) Content Similarity-Dependent Editing , and (3) Region-Preserved Editing . For Position-Dependent Editing , such as object addition, we leverage the layers that are more reliant on positional information to propose a reasoning-before-generation strategy, effectively mitigating conflicts between object addition and consistency with the original content. For Content Similarity-Dependent Editing , such as non-rigid editing, we perform modifications in layers that rely more on content similarity. For Region-Preserved Editing , using background replacement as an example, we achieve perfect preservation of regions that should remain unchanged by injecting values across all layers. Furthermore, we demonstrate that this full-layer injection approach enables a highly harmonious blending effect, significantly outperforming latent blending [ 1 ] . With minimal modifications, it can also be extended to object movement and outpainting tasks. Diverse image editing results are presented in Figure 1 . Qualitative and quantitative comparisons with state-of-the-art editing methods on object addition, non-rigid editing, and background replacement tasks demonstrate the superiority of our approach. Comprehensive ablation studies further validate the effectiveness of our key designs. To summarize, our contributions are three-fold as below: • We are the first to reveal the existence of different dependency mechanisms in self-attention layers of RoPE-based MMDiT models during generation. • Guided by this mechanism, we tailor different editing methods for versatile image editing tasks based on their characteristics. • The strong editing performance demonstrates the great potential of our approach. 2 Related Work Text-to-Image Diffusion Models. Marking the emergence of Latent Diffusion [ 30 ] , diffusion-based models [ 8 , 25 , 32 , 27 ] have come to dominate the text-to-image generation field. Among them, the U-Net-based Stable Diffusion family (SD 1, SD 2, SDXL) gained immense popularity due to its high synthesis quality and open-source nature. Recently, more powerful MMDiT-based models, such as SD3 [ 5 ] and FLUX [ 16 ] , have drawn significant attention from the community. Both SD3 and FLUX adopt a rectified flow formulation [ 20 , 19 ] and replace the U-Net [ 31 ] with MMDiT [ 24 ] , which incorporates numerous self-attention layers that enable rich interactions between textual and visual information. Unlike SD3, which injects positional embeddings only at the input layer, FLUX explicitly applies Rotary Position Embedding (RoPE) [ 35 ] to both queries and keys in every self-attention layer. This distinction raises our curiosity about the role of RoPE in self-attention computations. In this paper, we investigate the positional dependency mechanisms in self-attention layers of RoPE-based MMDiT models and leverage these insights to develop task-specific editing strategies for different image editing tasks. Text-Guided Image Editing. Text-guided image editing is a challenging task that aims to modify images based on textual descriptions. Early text-driven image editing methods [ 23 , 41 , 39 , 40 , 17 ] were typically based on Generative Adversarial Networks (GANs) [ 7 , 12 , 13 , 14 ] . While they achieved promising results in specific domains ( e.g . , faces), they often struggled with editing images across arbitrary domains. With the rapid advancement of diffusion-based text-to-image models, diffusion-based image editing techniques have become the mainstream approach. Diffusion-based image editing

Read the complete text