PCBEAR: Pose Concept Bottleneck for Explainable Action Recognition

Jongseo Lee, Wooil Lee, Gyeong-Moon Park, Seong Tae Kim, Jinwoo Choi · 2025

Read PCBEAR: Pose Concept Bottleneck for Explainable Action Recognition by Jongseo Lee, Wooil Lee, Gyeong-Moon Park, Seong Tae Kim, Jinwoo Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13140.txt.

Introduction

Jongseo Lee and four co-authors explain action recognition with body poses in a 2025 paper. Deep models name actions well but hide their reasons, a flaw where clear accounts matter. Past explainers lean on pixel credit or fixed text tags, both weak at catching motion over time. Poses, as bare skeleton tracks, hold the motion and little else.

Their frame, PCBEAR, routes each clip through pose concepts before the call. Still pose concepts catch one frame stances; moving pose concepts catch patterns across frames. Clustering finds the concepts on its own, with no hand tags. On three known benches the method scored high while giving motion based reasons a person can check. Users can also step in at test time to fix a call.

Catalog record

CREATORJongseo Lee, Wooil Lee, Gyeong-Moon Park, Seong Tae Kim, Jinwoo Choi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13140

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PCBEAR: Pose Concept Bottleneck for Explainable Action Recognition Jongseo Lee, Wooil Lee, Gyeong-Moon Park, Seong Tae Kim, Jinwoo Choi Abstract Human action recognition (HAR) has achieved impressive results with deep learning models, but their decision-making process remains opaque due to their black-box nature. Ensuring interpretability is crucial, especially for real-world applications requiring transparency and accountability. Existing video XAI methods primarily rely on feature attribution or static textual concepts, both of which struggle to capture motion dynamics and temporal dependencies essential for action understanding. To address these challenges, we propose P ose C oncept B ottleneck for E xplainable A ction R ecognition (PCBEAR), a novel concept bottleneck framework that introduces human pose sequences as motion-aware, structured concepts for video action recognition. Unlike methods based on pixel-level features or static textual descriptions, PCBEAR leverages human skeleton poses, which focus solely on body movements, providing robust and interpretable explanations of motion dynamics. We define two types of pose-based concepts: static pose concepts for spatial configurations at individual frames, and dynamic pose concepts for motion patterns across multiple frames. To construct these concepts, PCBEAR applies clustering to video pose sequences, allowing for automatic discovery of meaningful concepts without manual annotation. We validate PCBEAR on KTH, Penn-Action, and HAA500, showing that it achieves high classification performance while offering interpretable, motion-driven explanations. Our method provides both strong predictive performance and human-understandable insights into the model’s reasoning process, enabling test-time interventions for debugging and improving model behavior. 2 2 footnotetext: Corresponding authors. 1 Introduction Figure 1 : Comparison with existing XAI approaches for video action recognition. (a) Supervoxel-based concepts [ 31 ] cluster similar regions but lack motion dynamics. (b) Grad-CAM [ 45 ] highlights spatial regions but fails to capture temporal dependencies. (c) Textual concepts [ 37 ] provide abstract labels that miss fine-grained motion cues. (d) PCBEAR uses pose-based concepts to model spatial and temporal dynamics, offering interpretable, motion-aware explanations. Advances in deep neural networks, including CNNs [ 25 , 49 , 10 , 52 , 18 , 19 ] and transformers [ 14 , 5 , 8 , 39 , 32 , 51 ] , have significantly advanced human action recognition (HAR) performance. However, these models remain opaque, making it difficult to understand their decision-making process. Ensuring interpretability is crucial for reliability, accountability, and safety, particularly in real-world applications [ 13 , 26 ] , leading to increased research in explainable AI (XAI) for HAR. Existing video XAI methods primarily follow two paradigms: feature attribution and concept-based approaches. Feature attribution methods [ 7 , 33 , 53 , 23 , 59 ] localize spatiotemporal regions that influence model predictions. However, they suffer from inherent limitations, such as sensitivity to perturbations and lack of robustness, often leading to unreliable explanations [ 50 , 3 , 24 , 48 ] . To overcome these issues, recent work explores Concept-based XAI (C-XAI), which explains models using high-level semantic concepts rather than pixel-level importance [ 31 , 27 , 42 ] . These methods define spatiotemporal concepts by clustering [ 1 ] visually similar regions from videos. However, as shown in Figure 1 (a), supervoxel-based concepts [ 31 ] highlight localized regions e.g . lower body, but fail to capture underlying motion dynamics e.g . stretching, making them ineffective for action recognition. Additionally, these methods require per-sample optimization, making them computationally expensive and impractical for real-world applications [ 31 , 27 , 42 ] . An alternative is to adapt image-based XAI methods for HAR models. One prominent category is feature attribution [ 41 , 45 , 20 , 36 , 35 , 21 , 59 , 61 ] , which highlights spatially important regions that influence predictions. However, as shown in Figure 1 (b), Grad-CAM [ 45 ] predominantly highlights static human presence in each frame, failing to capture the temporal dependencies essential for action recognition. To address this, Explainable-by-Design (EBD) methods [ 30 , 11 , 6 , 17 , 37 , 40 , 58 , 46 ] enforce models to predict predefined semantic concepts e.g . symbols, text, or prototypes before classification. Unlike feature attribution, which assigns importance to low-level spatial features, EBD ensures structured interpretability by enforcing concept-based reasoning. However, these methods primarily capture static appearance or object attributes rather than dynamic motion patterns, they struggle to capture fine-grained motion cues essential for action recognition. For instance, applying text-based concepts to HAR often results in abstract explanations, such as Arm swing as shown in Figure 1 (c), which fail to fully explain how the model predicts the action Boxing . These limitations highlight the necessity of explicitly motion-aware concept representations for explaining HAR models. To address these challenges, we propose P ose C oncept B ottleneck for E xplainable A ction R ecognition ( PCBEAR ), a novel concept bottleneck framework that enables transparent and interpretable human action recognition. Unlike existing methods relying on feature attribution [ 45 , 27 , 42 , 61 , 22 ] or static textual concepts [ 37 , 46 , 40 , 58 ] , PCBEAR provides structured explanations by explicitly modeling both spatial configurations and temporal motion dynamics. To achieve this, PCBEAR introduces two types of motion-aware pose-based concepts: (i) static pose-based concepts, which capture spatial configurations at individual frames, and (ii) dynamic pose-based concepts, which encode motion patterns across multiple frames. Since human skeleton poses focus purely on body movements, they are inherently robust to static biases, such as background changes and lighting variations [ 57 , 16 , 47 , 9 ] . Thanks to these properties, employing human pose as concepts enables PCBEAR to effectively captures temporal dynamics, making it well-suited for providing interpretable, motion-aware explanations for action recognition. Furthermore, to eliminate the need for manual annotations, we construct pose-based concepts using unsupervised clustering [ 43 ] . By grouping similar pose sequences automatically, PCBEAR discovers meaningful concepts without explicit human labeling, ensuring robust concept discovery while avoiding the need for manual annotation. This enables PCBEAR to learn pose-based concepts effectively, making HAR transparent and interpretable without requiring skeleton inputs. To validate the effectiveness of PCBEAR , we conduct extensive experiments on three datasets where temporal context is critical for action recognition: KTH [ 44 ] , Penn-Action [ 60 ] , and HAA500 [ 12 ] . We quantitatively compare PCBEAR to multiple baselines regarding concept efficiency and classification accuracy, demonstrating the strength of our method. For interpretability evaluation, we qualitatively analyze pose-based concepts and their contributions at both the sample and group levels, clearly demonstrating how PCBEAR reasons about actions. Additionally, we validate our pose-based explanations through controlled experiments and demonstrate test-time interventions that modify model predictions in an interpretable manner. The results demonstrate that PCBEAR achieves strong action recognition performance while providing explicit, motion-aware explanations. To summarize, our major contributions are as follows: • We tackle the under-explored problem of explaining human action recognition. Instead of relying on RGB or textual concepts, we introduce human pose-based concepts as structured and motion-aware representations. • We propose PCBEAR , a novel concept bottleneck framework that learns pose-based concepts directly from RGB videos without requiring explicit skeleton inputs, providing an interpretable representation of motion dynamics essential for action understanding. • We conduct extensive

Read the complete text