Concepts in Motion: Temporal Bottlenecks for Interpretable Video Classification
Patrick Knab Sascha Marton Philipp J. Schubert, Drago Guggiana Christian Bartelt · 2025
Read Concepts in Motion: Temporal Bottlenecks for Interpretable Video Classification by Patrick Knab Sascha Marton Philipp J. Schubert, Drago Guggiana Christian Bartelt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20899.txt.
Introduction
A video classifier that explains itself in plain concepts is easier to trust than a black box. Patrick Knab and four co-authors build such a classifier in a 2025 paper called Concepts in Motion. Their framework, MoTIF, extends concept bottleneck models from still images to video.
The model predicts from sequences of concept activations that are grounded in time. Per-concept attention tracks when each concept recurs and how its pattern over time shapes the prediction. A discovery module uses a vision-language model to extract object and action concepts from training data, conditioned on the class. The design keeps the final decision readable: users can see which concepts, appearing when, drove the classification.
Catalog record
| CREATOR | Patrick Knab Sascha Marton Philipp J. Schubert, Drago Guggiana Christian Bartelt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20899
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Concepts in Motion: Temporal Bottlenecks for Interpretable Video Classification Patrick Knab Sascha Marton Philipp J. Schubert, Drago Guggiana Christian Bartelt Abstract Conceptual models such as Concept Bottleneck Models (CBMs) have driven substantial progress in improving interpretability for image classification by leveraging human-interpretable concepts. However, extending these models from static images to sequences of images, such as video data, introduces a significant challenge due to the temporal dependencies inherent in videos, which are essential for capturing actions and events. In this work, we introduce MoTIF (Moving Temporal Interpretable Framework), an architectural design inspired by a transformer that adapts the concept bottleneck framework for video classification and handles sequences of arbitrary length. Within the video domain, concepts refer to semantic entities such as objects, attributes, or higher-level components (e.g., “bow,” “mount,” “shoot”) that reoccur across time—forming motifs collectively describing and explaining actions. Our design explicitly enables three complementary perspectives: global concept importance across the entire video, local concept relevance within specific windows, and temporal dependencies of a concept over time. Our results demonstrate that the concept-based modeling paradigm can be effectively transferred to video data, enabling a better understanding of concept contributions in temporal contexts while maintaining a competitive performance. Code available at github.com/patrick-knab/MoTIF . Figure 1: MoTIF. The framework takes videos as input and produces local concept explanations for local windows, global explanations for entire videos, and temporal dependency maps from the attention heads of the transformer module. Model represents MoTIF (ViT-L14) and sample frames are from HMDB51 ( Kuehne et al. 2011 ) , licensed under CC BY 4.0. 1 Introduction Modern deep learning models already achieve outstanding results in video understanding tasks such as video classification, action recognition, and event detection ( Liu et al. 2021 ; Bertasius et al. 2021 ) . Despite their success, these models are commonly perceived as black boxes since their internal workings are not interpretable in a way that reveals their decision-making process ( Molnar et al. 2020 ; Knab et al. 2025b ) . Concept Bottleneck Models (CBMs) ( Koh et al. 2020 ) address this issue by enforcing an intermediate bottleneck layer of human-understandable concepts, which are then used by a linear classifier to generate the final prediction. While CBMs have been extensively studied in the image domain ( Prasse et al. 2025 ; Yang et al. 2023 ; Sun et al. 2025 ; Schrodi et al. 2025 ) , their extension to video remains largely unexplored ( Jeyakumar et al. 2022 ) . Videos differ from images in that they contain a temporal component : concepts evolve over time and many actions cannot be deduced from a single frame ( Lee et al. 2025 ; Chen et al. 2025 ) . In addition, the length of a video is not fixed, making the adaptation of existing CBM architectures non trivial, since they always require the same embedding dimensionality. Transformer-based models ( Vaswani et al. 2017 ) are powerful for modeling such long-range dependencies ( Bertasius et al. 2021 ) , but their dense feature mixing hinders clear attribution at the concept level, making their interpretation infeasible ( Hao et al. 2021 ; Molnar et al. 2020 ) . In this work, we introduce MoTIF ( Mo ving T emporal I nterpretable F ramework), a concept bottleneck model tailored for video classification. MoTIF builds on transformer-inspired blocks and introduces a per-channel temporal self-attention (diagonal attention) mechanism that isolates temporal reasoning for each concept via depthwise 1{\times}1 convolutions. To illustrate how MoTIF extends beyond static images, Fig. 1 shows our framework: it processes a video, tracks its concepts through time, and explains which concepts drive the final prediction. In this example, the model identifies reoccurring semantic motifs such as a bow being mounted and the arrow being shot by Legolas (Lord of the Rings), which together form the basis for recognizing the higher-level action “shooting a bow.” Temporal dependency maps further reveal that specific concepts primarily attend to characteristic frames, i.e., the moments when the bow is mounted and the arrow released. As this example shows, the design preserves concept separation across time and enables analysis of both global concept contributions over an entire video and local relevance within specific frames. Our key contributions are: • A CBM framework, MoTIF , for video sequences that supports arbitrary-length inputs and integrates seamlessly with vision–language backbones. • Per-channel temporal self-attention that preserves concept independence within transformer blocks and models temporal dynamics on a per-concept basis. • MoTIF is the first method to enable three complementary explanation modes : (i) global concept relevance via log-sum-exp (LSE) pooling, (ii) localized temporal explanations using windowed concept attributions, and (iii) attention-based temporal maps that visualize how a concept channel distributes its focus across time. Extensive experiments and ablations (window size, backbone selection, temperature \tau , full vs. diagonal attention, etc.) demonstrate that MoTIF achieves robust performance while providing fine-grained and faithful explanations. 2 Method In this section, we present the MoTIF framework for interpretable video classification. The goal is to preserve concept-level transparency while leveraging temporal modeling capabilities. MoTIF is a CBM with a transformer architecture for video classification that operates directly on sequences of per-window concept activations (Figure 2 ). A video is represented by T temporal windows, each described by C scalar concept activations, with k=|C| ; the input is X\in\mathbb{R}^{T\times C} . The framework enforces strict per-concept interpretability : temporal dependencies are modeled independently for each concept channel, so that recurrent temporal patterns— motifs —remain attributable to individual concepts. After temporal processing, per-concept activations are refined via a nonnegative affine transformation, and a classifier produces per-time-step logits. Video-level predictions are obtained via log-sum-exp pooling, which also yields a time-importance profile for additional explanation. In contrast, a conventional transformer mixes channels during attention ( Vaswani et al. 2017 ) . For comparison, we also evaluate a variant with full multi-head attention , but emphasize that this removes explicit concept attribution. A detailed analysis of this and other design choices is provided in Section 3.2.2 and Appendix C . For completeness, an algorithmic overview of the framework and its subparts is included in Appendix E . Figure 2: MoTIF pipeline. Videos are embedded with a vision–language backbone and mapped to concept activations via cosine similarity. Per-channel temporal self-attention models dynamics independently for each concept, followed by a nonnegative affine transformation and classification. MoTIF enables explanations across three views: global concepts, local concepts, and temporal dependencies. Sample frames from SSv2 ( Materzynska et al. 2020 ) with MoTIF (ViT-L14). Channels as concepts. We use the term channel to denote one dimension of the concept space. Each video n yields y^{(n)}\in\mathbb{R} classes, X^{(n)}\in\mathbb{R}^{T_{n}\times C} concept activations, where the second axis ( C ) indexes concepts; each slice X^{(n)}_{:,c}\in\mathbb{R}^{T_{n}} corresponds to the temporal activation sequence of concept c\in C . Unlike CNN channels, these dimensions are semantically interpretable by design. Notation. Capital and lowercase letters denote matrices and vectors, respectively. During training, a batch of videos is represented as X\in\mathbb{R}^{B\times T\times C}, (1) where B is the batch size, T is the (padded) sequence length, and C is the number of concepts. Thus X_{:,:,c}\in\mathbb{R}^{B\times T} contains the activations of concept c across the batch. 2.1 Transformer Bottleneck Model Video and