Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding
Seongro Yoon, Donghyeon Cho, Jinsun Park, François Brémond · 2026
Read Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding by Seongro Yoon, Donghyeon Cho, Jinsun Park, François Brémond (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30611.txt.
Introduction
Seongro Yoon, Donghyeon Cho, Jinsun Park and François Brémond improve how video models read facial expressions. Vision transformers, the dominant video architecture, learn from self-supervised pretraining, meaning they train on unlabelled video first. Their attention, though, favours big global motions and coarse timing. Subtle cues such as brief wrinkles or frowns get drowned out, which limits expression recognition in uncontrolled settings.
Their plug-in, MiRA, redistributes attention using the attention maps themselves. It scores each frame's marginal importance from its own attention maps, then shifts weight toward frames and spots carrying local expression cues. MiRA adds no new trainable parameters. An exact mode reshapes attention after the softmax step, while a lighter flashLite mode copies it earlier for speed inside fast attention kernels. On hard facial expression benchmarks, MiRA improved strong transformer baselines consistently.
Catalog record
| CREATOR | Seongro Yoon, Donghyeon Cho, Jinsun Park, François Brémond |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30611
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reweighting Framewise Attention in Video Transformers for Facial Expression Understanding Seongro Yoon, Donghyeon Cho, Jinsun Park, François Brémond Abstract Understanding facial expressions in videos requires modeling subtle and localized facial dynamics under unconstrained conditions. Although recent Vision Transformer (ViT)-based video models have shown strong performance through large-scale self-supervised pretraining, their attention mechanisms often emphasize dominant global motions and coarse temporal dynamics, limiting sensitivity to fine-grained facial variations. To address this limitation, we propose MiRA (Marginal-induced Attention Redistribution), a plug-in frame-marginal attention redistribution framework for ViT backbones that enhances spatio-temporal selectivity toward subtle facial dynamics without introducing additional trainable parameters. MiRA derives frame-level confidence and intra-frame concentration statistics from self-attention maps to estimate frame-wise marginal importance and redistribute attention toward spatiotemporally localized facial cues. We first introduce a principled exact mode based on post-softmax attention redistribution. To further improve efficiency, we propose flashLite mode , a lightweight pre-softmax approximation that integrates frame-marginal redistribution into FlashAttention kernels while preserving the effectiveness of the exact formulation. Experimental results on challenging Facial Expression Recognition (FER) benchmarks demonstrate consistent improvements over strong ViT baselines. † † footnotetext: Code and pretrained models are available at https://github.com/ysrinria/MiRA . 1 Introduction Learning effective representations of facial dynamics in videos requires capturing subtle spatio-temporal expression cues, such as transient wrinkles, frowns, and fine-grained muscle movements. These subtle facial cues are often entangled with dominant global motions ( e.g . , head movements) and external variations ( e.g . , camera motion), making it challenging to accurately model facial dynamics. Under such conditions, video-based facial expression understanding requires selectively allocating attention to frames containing informative expression dynamics, while suppressing redundant or motion-dominated frames. Recent advances in general video understanding have demonstrated impressive performance on video benchmarks through large-scale self-supervised pretraining and downstream finetuning. Various paradigms have been explored for video representation learning, including contrastive approaches [ 45 , 24 , 44 , 14 ] , self-distillation frameworks [ 1 , 4 ] , and, more recently, masked video modeling based on Masked Autoencoders (MAEs) [ 58 , 57 , 20 , 51 , 55 , 4 , 3 , 28 , 30 , 42 , 23 , 22 ] . However, these models are primarily optimized to capture dominant scene-level dynamics, often focusing on coarse object motions. When applied to facial videos, subtle intra-facial variations are often treated as redundant when co-occurring with more pronounced global movements. To address this limitation, we investigate how generic ViT-based video models can be adapted to better capture facial dynamics. Despite differences in formulation, most modern video models are built upon ViTs [ 53 , 18 , 2 ] , which have become the canonical backbone for video representation learning. Motivated by this, we redesign the self-attention mechanism in ViTs to shift attention from dominant global motions toward fine-grained facial contexts, enabling more effective modeling of subtle facial dynamics while retaining the strengths of the original ViT backbone. Figure 1 : Frame-marginal attention reweighting modules. (b) Exact mode applies principled framewise scoring and reweighting on the post-softmax attention map (a) by evaluating all query–key interactions. However, this design requires repeated memory access and is incompatible with fused kernels such as FlashAttention (c). Although FLOPs are similar, the additional memory I/O significantly slows execution. (d) FlashLite mode instead computes key-wise proxies before attention computation and integrates the resulting weights ( \alpha^{\mathrm{FL}} ) directly into the FlashAttention kernel, reducing memory I/O while preserving accuracy. ( Notation: L : tokens, d : feature dimension, Z=QK^{\top}/\sqrt{d_{K}} : attention logits, \mathcal{O}(\cdot) : memory I/O complexity.) In this work, we propose MiRA, a plug-in frame-marginal attention redistribution module for ViT backbones, and begin with its principled exact mode . In unconstrained facial videos without explicit face cropping, the proposed method redistributes attention using frame-level confidence and intra-frame concentration statistics, producing sharper and more localized attention patterns than those induced by dominant global motions. Given the self-attention maps of a ViT block, the proposed method derives two complementary frame-level statistics: a confidence score , measuring the total attention mass assigned to each frame across the sequence, and an intra-frame concentration score , computed via inverse entropy to quantify the spatial concentration of attention within each frame. These statistics are combined to estimate frame-wise marginal importance, which is then used to redistribute attention toward salient and localized facial dynamics. The exact mode , while effective, incurs additional memory I/O overhead since it requires access to the full post-softmax attention map, leading to repeated memory access beyond standard attention computation (see Fig. 1 ). To improve efficiency, we introduce flashLite mode , which integrates frame-marginal redistribution into FlashAttention kernels [ 12 , 13 ] . While fused kernels achieve high efficiency by computing attention on the fly without materializing the full attention map, the exact mode relies on post-softmax attention values and therefore cannot be directly incorporated. We address this limitation through a pre-softmax approximation that injects redistribution factors directly into the fused attention computation. As a result, flashLite preserves the effectiveness of the exact mode while inheriting FlashAttention’s efficiency benefits. Our contributions are summarized as follows: \bullet We propose a plug-in frame-marginal attention redistribution module for ViT backbones that enhances sensitivity to subtle facial dynamics without introducing additional trainable parameters. \bullet We further introduce flashLite mode, a lightweight pre-softmax approximation that integrates the proposed frame-marginal redistribution into FlashAttention kernels, reducing memory I/O while preserving the effectiveness of the exact mode. \bullet We demonstrate consistent improvements over strong ViT baselines on large-scale video FER benchmarks without relying on task-specific heuristics ( e.g . , tight facial alignment), which are often impractical in scalable pretraining and real-world video settings. 2 Related Work Video Representation Learning. The success of ViTs in image recognition has inspired extensive efforts to extend them to video understanding tasks, where the goal is to model rich spatio-temporal dependencies across frames [ 2 ] . Early approaches adapt the vanilla ViT by incorporating temporal attention to jointly learn spatial and temporal correlations within video clips. To improve efficiency and scalability, subsequent works introduced architectural refinements, including factorized spatio-temporal attentions [ 6 ] , locality bias through windowed attention [ 38 ] , and multiscale or pooled attention mechanisms that capture motion at different temporal resolutions [ 2 , 34 , 61 , 35 ] . In parallel, masked prediction has emerged as a dominant paradigm for self-supervised video representation learning. Inspired by the success of masked language modeling [ 16 ] and masked autoencoders in the image domain [ 26 , 66 , 41 ] , these approaches train models to reconstruct missing spatio-temporal information from sparsely observed inputs [ 58 , 57 , 20 , 51 ] . More recent developments enhance temporal reasoning by enforcing consistency across paired frames [ 42 , 23 ] , introducing adaptive and content-aware masking strategies [ 59 , 28 , 3 , 30 ] , and scaling pretraining to longer input sequences for improved temporal coverage [ 22 ] . Facial Dynamics Modeling in Video. The main challenge of facial expression modeling in videos lies in capturing fine-grained intra-facial dynamics under diverse and unconstrained conditions. A growing body of work has proposed transformer-based models that directly model spatio-temporal facial dynamics, incorporating mechanisms such