Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders Conference: ACM Multimedia; 2026 CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning

Atahan Dokme, Sriram Vishwanath · 2026

Read Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders Conference: ACM Multimedia; 2026 CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning by Atahan Dokme, Sriram Vishwanath (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03919.txt.

Introduction

Atahan Dokme and Sriram Vishwanath studied sparse autoencoders on video, where standard methods break the flow of features over time. Hard TopK picks make feature tags jump from frame to frame, cutting autocorrelation by 36 percent. Spatio-temporal contrastive aims plus Matryoshka hierarchical grouping restored the flow and even beat the raw baseline. Contrastive features also lifted action tagging and text-video search, while knock-out tests showed the key signal packed into a few clear features.

Catalog record

CREATORAtahan Dokme, Sriram Vishwanath
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40470
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03919

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interpreting Video Representations with Spatio-Temporal Sparse Autoencoders Conference: ACM Multimedia; 2026 CCS: Computing methodologies Computer vision CCS: Computing methodologies Machine learning Atahan Dokme, Sriram Vishwanath Abstract . We present the first systematic study of Sparse Autoencoders (SAEs) on video representations. Standard SAEs decompose video into interpretable, monosemantic features but destroy temporal coherence: hard TopK selection produces unstable feature assignments across frames, reducing autocorrelation by 36%. We propose spatio-temporal contrastive objectives and Matryoshka hierarchical grouping that recover and even exceed raw temporal coherence. The contrastive loss weight controls a tunable trade-off between reconstruction and temporal coherence. A systematic ablation on two backbones and two datasets shows that different configurations excel at different goals: reconstruction fidelity, temporal coherence, action discrimination, or interpretability. Contrastive SAE features improve action classification by +3.9% over raw features and text-video retrieval by up to 2.8 × \times R@1. A cross-backbone analysis reveals that standard monosemanticity metrics contain a backbone-alignment artifact: both DINOv2 and VideoMAE produce equally monosemantic features under neutral (CLIP) similarity. Causal ablation confirms that contrastive training concentrates predictive signal into a small number of identifiable features. 1. Introduction Sparse Autoencoders (SAEs) have transformed the interpretability of language models ( 2 ; 21 ) and, more recently, static image encoders ( 15 ; 19 ) . Yet video vision models remain underexplored. From self-supervised encoders like DINOv2 ( 14 ) and VideoMAE ( 22 ) to multimodal retrieval and captioning systems, we lack tools to understand what these models learn internally. Whether SAEs can meaningfully decompose video representations, where features are spatio-temporal rather than spatial, is an open question. We observe that the challenge is structural. A video clip produces a feature tensor spanning T frames and P spatial patches, where the same semantic concept (a hand, a moving object) should ideally activate consistent features across time. Standard SAEs, encoding each patch independently, are oblivious to this structure. This raises two questions: Can SAEs reveal interpretable structure in video features? And does the temporal dimension require new methodology? We find that standard SAEs produce monosemantic video features but fail to preserve temporal coherence, making it impossible to track concepts across frames. Our central finding is a design principle: the contrastive strength controls the trade-off between reconstruction fidelity and temporal coherence. Both the loss coefficient \lambda and the temperature \tau modulate this strength through different mechanisms ( \lambda scales the loss magnitude, \tau controls the sharpness of positive/negative separation), but both move variants along the same Pareto frontier. This is not a limitation to work around. It is a controllable axis that practitioners can set based on their application. Our contributions: (1) Systematic empirical study of SAEs on video representations. We train TopK SAEs on two backbones (DINOv2 and VideoMAE) and two datasets (SSv2 and K400), demonstrating that SAEs decompose video features into monosemantic, interpretable concepts such as scene elements, objects, and action-correlated features. Video-native models (VideoMAE) score substantially higher on DINOv2-based monosemanticity metrics than image models (MS =0.972 vs. 0.359 ), but a CLIP-based control reveals both produce equally monosemantic features (MS \approx 0.68), identifying a metric alignment artifact in the standard evaluation protocol. (2) Diagnosis of temporal incoherence as a core failure mode. We show that standard SAEs destroy temporal structure: lag-1 autocorrelation drops 36%, with 95% of features below the 0.3 coherence threshold. We trace this to the hard top- k selection mechanism and show via causal ablation that the resulting features, while temporally unstable, are nonetheless predictively important for action discrimination (5 \times stronger ablation signal than random features under probe-based ablation). (3) A controllable reconstruction–coherence trade-off. Extending T-SAE ( 1 ) from text to video, we evaluate three contrastive variants (temporal, separate, raster-scan) and Matryoshka hierarchical grouping. A systematic ablation across 29 configurations reveals that the contrastive strength (via \lambda or \tau ) controls where a variant sits on the Pareto frontier. At low weight ( \lambda{=}0.01 ), all variants preserve reconstruction ( R^{2}=0.31{-}0.42 ). At high weight, Temporal+M exceeds raw temporal coherence (lag-1 =0.462 vs. 0.435). The best configurations improve action discrimination by +3.9% over raw features and boost text-video retrieval by 2.8 \times R@1 on DINOv2. Matryoshka grouping concentrates action-relevant information into 20% of the dictionary. 2. Related Work Sparse Autoencoders for Interpretability. SAEs were introduced to decompose neural network activations into sparse, monosemantic features ( 2 ; 4 ) . Subsequent work scaled SAEs to frontier language models ( 21 ) . Key architectural innovations include TopK activation ( 6 ) , JumpReLU ( 18 ) , BatchTopK with Matryoshka grouping ( 3 ) for hierarchical feature organization, and auxiliary losses for dead latent recovery. SAEs for Vision Models. Recent work has extended SAEs to vision encoders, establishing that sparse decomposition yields interpretable visual features. Pach et al. ( 15 ) train SAEs on CLIP and introduce the MonoSemanticity evaluation framework we adopt in this work, demonstrating that SAE interventions on the vision encoder can steer multimodal LLM outputs. Stevens et al. ( 19 ) enable patch-level causal edits via SAEs on frozen ViTs, providing a rigorous causal methodology that motivates our ablation experiments. Joseph et al. ( 9 ) provide a steerability analysis of CLIP’s vision transformer, and Lim et al. ( 12 ) introduce PatchSAE for concept-level analysis during adaptation. While these works demonstrate the promise of SAEs for vision, they all operate on static images. None examine how the temporal structure of video interacts with sparse decomposition, which is the focus of our study. Temporal SAEs. Bhalla et al. ( 1 ) propose Temporal SAEs (T-SAEs) for language models, adding contrastive losses on adjacent tokens to disentangle semantic from positional features. They demonstrate that temporal consistency improves feature quality for text sequences. We extend this temporal contrastive principle from 1D text sequences to spatio-temporal video representations, where the challenge is qualitatively different: video tokens have both spatial adjacency within frames and temporal adjacency across frames, and the visual content at a given spatial position can change substantially between frames due to motion. Our raster-scan variant (Section 3.2 ) addresses this by unifying spatial and temporal contrastive objectives in a single loss. Temporal Consistency in Video Representations. Object-centric approaches such as Slot Contrast ( 25 ) learn temporally consistent slots by contrasting object representations across frames. Spatio-temporal attention attribution methods ( 24 ) provide post-hoc explanations of video model predictions. Temporal contrastive self-supervised methods such as TCLR ( 5 ) promote temporal diversity, while multi-pretext frameworks like CSTP ( 26 ) learn temporal structure during pretraining. Optical flow methods like RAFT ( 20 ) provide dense correspondences that could serve as temporal pairing targets. Our approach is complementary to these methods. We decompose frozen features into sparse dictionaries rather than learning new representations or modifying pretraining. Our adjacency-based contrastive losses operate on fixed spatial positions rather than tracked objects. This is a deliberate design choice: SAE features should decompose the model’s representation as-is, without introducing external correspondence mechanisms. We discuss the limitations of fixed spatial pairing under camera motion in Section 7 . Interpretability for Action Models. Bear et al. ( 8 ) apply mechanistic interpretability to Vision-Language-Action models for robotics using logit lens projections. Khan et al. ( 11 ) use SAEs on VLA residual streams

Read the complete text