FAME: Fairness-aware Attention-modulated Video Editing

Zhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Zhidong Li, Longbing Cao · 2025

Read FAME: Fairness-aware Attention-modulated Video Editing by Zhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Zhidong Li, Longbing Cao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22960.txt.

Introduction

Zhangkai Wu leads six researchers tackling gender bias in training-free video editing. When a prompt names a job, such editors often fall back on stereotypes for who appears on screen. Fixing that bias by brute force can break motion and weaken the match to the prompt.

Their system, FAME, adds soft debiasing tokens drawn from minority examples into the text encoder. It also tunes both time-based attention and prompt-to-region attention, with masks and time decay that protect motion. The goal is fairer casting in edited clips while keeping scenes coherent and true to the words given.

Catalog record

CREATORZhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Zhidong Li, Longbing Cao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22960

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FAME: Fairness-aware Attention-modulated Video Editing Zhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Zhidong Li, Longbing Cao Abstract Training-free video editing (VE) models tend to fall back on gender stereotypes when rendering profession-related prompts. We propose FAME for Fairness-aware Attention-modulated Video Editing that mitigates profession-related gender biases while preserving prompt alignment and temporal consistency for coherent VE. We derive fairness embeddings from existing minority representations by softly injecting debiasing tokens into the text encoder. Simultaneously, FAME integrates fairness modulation into both temporal self attention and prompt-to-region cross attention to mitigate the motion corruption and temporal inconsistency caused by directly introducing fairness cues. For temporal self attention, FAME introduces a region constrained attention mask combined with time decay weighting, which enhances intra-region coherence while suppressing irrelevant inter-region interactions. For cross attention, it reweights tokens to region matching scores by incorporating fairness sensitive similarity masks derived from debiasing prompt embeddings. Together, these modulations keep fairness-sensitive semantics tied to the right visual regions and prevent temporal drift across frames. Extensive experiments on new VE fairness-oriented benchmark FairVE demonstrate that FAME achieves stronger fairness alignment and semantic fidelity, surpassing existing VE baselines. 1 Introduction Video editing with text prompts has witnessed rapid development thanks to the rise of text-to-image (T2I) Diffusion Models (DM) such as Stable Diffusion (SD) ( Rombach et al. 2022 ) and DALLE-E ( Ramesh et al. 2021 ) . Building upon these pre-trained T2I models, recent training-free VE methods ( Qi et al. 2023 ; Geyer et al. 2024 ; Liu et al. 2024b ; Yang et al. 2025 ) enable flexible control over object appearance, motion, and semantics, by manipulating temporal self-attention and cross-frame attention maps. On the other hand, generating fair content from pre-trained vision-language models has become an increasingly critical problem. ( Cho, Zala, and Bansal 2023 ) first revealed that T2I models often reproduce gender and racial stereotypes, even for given neutral prompts ( Cho, Zala, and Bansal 2023 ; Shen et al. 2024 ; Gandikota et al. 2024 ) . This social bias largely stems from imbalanced image–text pairs, where data associated with perpetuate majority groups is more frequently collected, and from inherent biases in large-scale language models used in text encoders such as CLIP ( Ross, Katz, and Barbu 2021 ; Birhane, Prabhu, and Kahembwe 2021 ) . However, existing fairness-aware T2I methods ( Li et al. 2024a ; Kim et al. 2025 ) focus on static images only, which often rely on prompt-image alignment assumptions that may not be generalized to temporally coherent video generation. Figure 1: Training-free VE methods (e.g., TokenFlow ( Geyer et al. 2024 ) ) fail to correct gender stereotypes in profession related prompts, as shown in the 1st row, even with debiasing prompt, causing motion corruption, with missing actions, ghosting, and temporal discontinuity. Videograin ( Yang et al. 2025 ) partly reduces bias but still retains stereotypical features such as outfit, illustrated in the 2nd row. In contrast, FAME combines soft prompt encoding and attention modulation to remove both profession bias and motion corruption, generating coherent videos with accurate debiased profession details. Bridging these two lines of research reveals a significant gap: fairness-aware VE in a training-free setting has not been formally explored. Existing T2I models exhibit gender bias toward profession related prompts, defaulting to stereotypical gender content. When debiasing prompts are applied, these models often suffer from motion corruption, manifesting as semantic mismatch, visual inconsistency, and degraded controllability. More details are revealed in Figure 1 and our Prompt Responsiveness Test in Appendix Figure 5 .These observations highlight the need for a dedicated framework that can achieve fairness control without compromising motion consistency. To bring fairness into VE while still preserving prompt meaning and motion quality, FAME (Fairness-aware Attention-modulated VE) as a training-free approach tackles both bias and motion corruption together. To reduce gender bias in profession related prompts while preserving their original meaning, we introduce soft debiasing prompt encoding , which gently adjusts prompt embeddings toward fairness-aware semantics without altering the intended content. Yet debiasing at the prompt level alone cannot stop fairness cues from fading over time, so we add temporal self-attention modulation to maintain fairness consistency across frames and avoid temporal attention dilution. Finally, since fairness tokens can still drift from their correct visual regions during denoising, we apply cross-attention reweighting to better align fairness related prompts with the corresponding spatial regions. On open source benchmarks, FAME consistently outperforms state-of-the-art training-free VE models in fairness preservation, semantic accuracy, and visual consistency, as measured by both CLIP based metrics and human evaluations. Our main contributions are summarized as follows: • FairVE , as the first benchmark that exploits existing open source datasets and protocols to evaluate fairness in T2I based VE models. • FAME , as a training-free framework to jointly mitigate bias and motion corruption through prompt encoding, temporal attention modulation, and cross attention reweighting at the inference time. • The state-of-the-art performance in fairness preservation, semantic alignment, and temporal coherence by FAME. Figure 2: Framework of FAME. In the denoising process, the soft debiasing prompt encoding module will encode the debiasing prompt {\boldsymbol{\mathbf{P}}}_{\texttt{tar}} explicitly and fuse the self-attention map {\boldsymbol{\mathbf{Q}}}{\boldsymbol{\mathbf{K}}}^{T} with time decay {\boldsymbol{\mathbf{S}}} and element weight {\boldsymbol{\mathbf{M}}} with cross-attention map reweighting. 2 Video Editing Models and Fairness Video Editing Models Latent diffusion models are commonly used, which first encode the input image sequence \mathcal{X} into a latent representation {\boldsymbol{\mathbf{z}}}_{0} and then gradually corrupt {\boldsymbol{\mathbf{z}}}_{0} through a forward diffusion process, resulting in intermediate latent variables \{{\boldsymbol{\mathbf{z}}}_{t}\}_{t} and their associated attention maps. \displaystyle q\left({\boldsymbol{\mathbf{z}}}_{t}\mid{\boldsymbol{\mathbf{z}}}_{t-1}\right)= \displaystyle\mathcal{N}\left({\boldsymbol{\mathbf{z}}}_{t};\sqrt{1-\beta_{t}}{\boldsymbol{\mathbf{z}}}_{t-1},\beta_{t}{\boldsymbol{\mathbf{I}}}\right), where t=1,\dots,T , with T denoting the total number of steps in the forward diffusion process. The parameter \beta_{t} controls the noise strength injected at timestep t . After defining the forward process, a backward inversion process is used to denoise the noise data into original data, with each step defined as: \displaystyle p_{\boldsymbol{\mathbf{\theta}}}\left({\boldsymbol{\mathbf{z}}}_{t-1}\mid{\boldsymbol{\mathbf{z}}}_{t}\right)=\mathcal{N}\left({\boldsymbol{\mathbf{z}}}_{t-1};{\boldsymbol{\mathbf{\mu}}}_{\boldsymbol{\mathbf{\theta}}}({\boldsymbol{\mathbf{z}}}_{t},t),\sigma_{t}^{2}{\boldsymbol{\mathbf{I}}}\right), where {\boldsymbol{\mathbf{\mu}}}_{\boldsymbol{\mathbf{\theta}}}({\boldsymbol{\mathbf{z}}}_{t},t) denotes the prediction of {\boldsymbol{\mathbf{x}}}_{0} at step t , and \sigma_{t}^{2} is the variance in the inverse step. In practice, the notable U-Net architecture is usually involved in designing the neural architecture of {\boldsymbol{\mathbf{\mu}}}_{\boldsymbol{\mathbf{\theta}}}({\boldsymbol{\mathbf{z}}}_{t},t) . Further, linear transformed denoising model {\boldsymbol{\mathbf{\epsilon}}}_{{\boldsymbol{\mathbf{\theta}}}}({\boldsymbol{\mathbf{z}}}_{t},t) is usually used to predict the residual noise, replacing {\boldsymbol{\mathbf{\mu}}}_{\boldsymbol{\mathbf{\theta}}}({\boldsymbol{\mathbf{z}}}_{t},t) for computational convenience as: \displaystyle{\boldsymbol{\mathbf{z}}}_{t-1}=\sqrt{\frac{\alpha_{t-1}}{\alpha_{t}}}{\boldsymbol{\mathbf{z}}}_{t}+\left(\sqrt{\frac{1-\alpha_{t-1}}{\alpha_{t-1}}}-\sqrt{\frac{1-\alpha_{t}}{\alpha_{t}}}\right){\boldsymbol{\mathbf{\epsilon}}}_{{\boldsymbol{\mathbf{\theta}}}}({\boldsymbol{\mathbf{z}}}_{t},t), where \alpha_{t}=1-\beta_{t} defines the retention ratio of signal. Classifier-free methods ( Ho and Salimans 2022 ) have become a mainstream approach for T2I generation, with the residual noise model {\boldsymbol{\mathbf{\epsilon}}}_{{\boldsymbol{\mathbf{\theta}}}}({\boldsymbol{\mathbf{z}}}_{t},t) modified as: \tilde{\epsilon}_{{\boldsymbol{\mathbf{\theta}}}}\left({\boldsymbol{\mathbf{z}}}_{t},t,{\boldsymbol{\mathbf{\phi}}}({\boldsymbol{\mathbf{p}}}_{\texttt{ref}}),\emptyset\right)=w\cdot\epsilon_{{\boldsymbol{\mathbf{\theta}}}}\left({\boldsymbol{\mathbf{z}}}_{t},t,{\boldsymbol{\mathbf{\phi}}}({\boldsymbol{\mathbf{p}}}_{\texttt{ref}})\right)\\ \quad\quad\quad\quad\quad\quad\quad\quad+(1-w)\cdot\epsilon_{{\boldsymbol{\mathbf{\theta}}}}\left({\boldsymbol{\mathbf{z}}}_{t},t,\emptyset\right), (1) where {\boldsymbol{\mathbf{\phi}}}(\cdot) is the text encoder. The combined noise prediction \tilde{\epsilon}_{{\boldsymbol{\mathbf{\theta}}}} is obtained by linearly interpolating between the conditional prediction (conditioned on the prompt) and the unconditional prediction, with a weighting factor w\in[0,1] by null-text optimization trick ( Ho and Salimans 2022 ; Mokady et al. 2023 ) . In training-free VE models, attention maps in the residual noise model {\boldsymbol{\mathbf{\epsilon}}}_{{\boldsymbol{\mathbf{\theta}}}}({\boldsymbol{\mathbf{z}}}_{t},t,{\boldsymbol{\mathbf{\phi}}}({\boldsymbol{\mathbf{p}}}_{\texttt{ref}}),\emptyset) are extracted during the inversion stage to preserve the spatio-temporal consistency across frames. In detail, these models replace the reference prompt {\boldsymbol{\mathbf{p}}}_{\texttt{ref}} with target prompt {\boldsymbol{\mathbf{p}}}_{\texttt{tar}} to predict the noise component \epsilon_{{\boldsymbol{\mathbf{\theta}}}}({\boldsymbol{\mathbf{z}}}_{t},t,{\boldsymbol{\mathbf{\phi}}}({\boldsymbol{\mathbf{p}}}_{\texttt{tar}})) via the following reverse process: \tilde{\epsilon}_{{\boldsymbol{\mathbf{\theta}}}}\left({\boldsymbol{\mathbf{z}}}_{t},t,{\boldsymbol{\mathbf{\phi}}}({\boldsymbol{\mathbf{p}}}_{\texttt{tar}}),\emptyset\right)=w\cdot\epsilon_{{\boldsymbol{\mathbf{\theta}}}}\left({\boldsymbol{\mathbf{z}}}_{t},t,{\boldsymbol{\mathbf{\phi}}}({\boldsymbol{\mathbf{p}}}_{\texttt{tar}})\right)\\ +(1-w)\cdot\epsilon_{{\boldsymbol{\mathbf{\theta}}}}\left({\boldsymbol{\mathbf{z}}}_{t},t,\emptyset\right). (2) This guidance mechanism enables controllable generation aligned with textual editing intents while maintaining temporal coherence in video frames. Fairness in Video Editing T2I-Based VE Models Due to the high inference cost of

Read the complete text