Neurosymbolic Framework for Concept-Driven Logical Reasoning in Skeleton-Based Human Action Recognition
Talha Ilyas, Deval Mehta, Zongyuan Ge · 2026
Read Neurosymbolic Framework for Concept-Driven Logical Reasoning in Skeleton-Based Human Action Recognition by Talha Ilyas, Deval Mehta, Zongyuan Ge (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07140.txt.
Introduction
Talha Ilyas, Deval Mehta, and Zongyuan Ge make skeleton-based action recognition explain itself. Systems that read body-joint motion can label actions well, but most remain black boxes. Their neurosymbolic framework reframes recognition as logical reasoning over motion concepts, using rules written in first-order logic.
A skeleton encoder extracts motion features, which a decoder turns into readable pose and dynamics concepts. Differentiable logic layers combine those concepts into human-readable rules for each action. Descriptions generated by language models help align the concepts. On major action datasets, the system stays competitive while offering explicit explanations grounded in its logical structure.
Catalog record
| CREATOR | Talha Ilyas, Deval Mehta, Zongyuan Ge |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.07140
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neurosymbolic Framework for Concept-Driven Logical Reasoning in Skeleton-Based Human Action Recognition Talha Ilyas, Deval Mehta, Zongyuan Ge Abstract Skeleton-based human activity recognition has achieved strong empirical performance, yet most existing models remain black boxes and difficult to interpret. In this work, we introduce a neurosymbolic formulation of skeleton-based HAR that reframes action recognition as concept-driven first-order logical reasoning over motion primitives. Our framework bridges representation learning and symbolic inference by grounding first-order logic predicates in learnable spatial and temporal motion concepts. Specifically, we employ a standard spatio-temporal skeleton encoder to extract latent motion representations, which are then mapped to interpretable concept predicates via a spatio-temporal concept decoder that explicitly separates pose-centric and dynamics-centric abstractions. These concept predicates are composed through differentiable first-order logic layers, enabling the model to learn human-readable logical rules that govern action semantics. To impose semantic structure on the learned concepts, we align skeleton representations with LLM-derived descriptions of atomic motion primitives, establishing a shared conceptual space for perception and reasoning. Extensive experiments on NTU RGB+D 60/120 and NW-UCLA demonstrate that our approach achieves competitive recognition performance while providing explicit, interpretable explanations grounded in logical structure. Our results highlight neurosymbolic reasoning as an effective paradigm for interpretable spatio-temporal action understanding. Code † . 2 2 footnotetext: Extended version of paper accepted at IJCAI 2026 1 1 footnotetext: Corresponding author 1 Introduction Skeleton-based human activity recognition (HAR) provides an efficient, privacy-preserving alternative to video-based methods by focusing on joint coordinates that isolate human structure from environmental noise 7 ; 20 ; 1 . While deep learning has advanced classification accuracy, these models remain black boxes, problematic for high-stakes domains such as healthcare and rehabilitation where transparent decision-making is essential 12 ; 23 . Neurosymbolic AI has gained traction for combining neural perception with formal logic interpretability 8 ; 4 . However, its application to skeleton-based HAR remains largely unexplored due to fundamental integration challenges. This work addresses these challenges to achieve interpretable skeleton-based HAR. Symbol Grounding. A key challenge is bridging continuous joint trajectories and discrete symbolic predicates. Traditional joint heuristics (e.g., knee angle <90^{o} for bent knee ) 24 are brittle, as small pose errors or variations in viewpoint and execution can prevent predicate activation for reliable HAR. Inspired by Concept Bottleneck Models (CBMs) 15 , we instead learn abstract motion concepts and construct an LLM-generated concept bank of spatial primitives (e.g., arm_raise , knee_bend ) that serve as symbolic predicates. We also enforce semantic grounding by aligning GCN-extracted skeleton features with pretrained text encoder based concept embeddings. Temporal Dynamics. Actions are inherently temporal, varying in speed, duration, and execution order. WearingGlasses and TakeOffGlasses share similar static poses ( hands near the face , arm engagement ) but differ in temporal dynamics: the former involves grasping and upward motion, while the latter requires release and downward motion. Purely spatial logic cannot capture such distinctions, and incorporating temporal logical rules into neurosymbolic frameworks is challenging. To address this, we introduce explicit temporal concepts capturing motion direction (e.g., motion_upward , motion_downward ), hand state transitions (e.g., hand_grasp , hand_release ), and execution order (e.g., seq_hands_first ). Our Spatio-Temporal Concept Decoder (STC-Decoder) then independently decodes spatial and temporal concepts, enabling distingushing semantically similar actions such as (\texttt{arm\_raise}\land\texttt{hand\_grasp}\land\texttt{motion\_upward})\Rightarrow\textsc{WearGlasses} versus (\texttt{arm\_lower}\land\texttt{hand\_release}\land\texttt{motion\_downward})\Rightarrow\textsc{TakeOffGlasses} (Fig (b)). End-to-End Learning. Discrete Boolean operators block gradient propagation, preventing joint optimization of perception and reasoning components 29 . While differentiable fuzzy logic 2 shows promise, scalability to large-scale, fine-grained skeleton-based HAR remains open. We employ differentiable logic layers with soft AND/OR operations 25 and gradient grafting 34 , enabling discrete inference with end-to-end differentiability. Final predictions aggregate triggered rules via learned weights, defining actions as logical compositions of motion concepts. Together, these components form an end-to-end neurosymbolic framework (Fig (b)) in which a GCN encoder extracts spatio-temporal features with sequence–concept alignment, the STC-Decoder predicts interpretable concepts, and differentiable logic layers compose them via first-order operations ( \land , \lor , \neg ) to derive action predictions for action recognition. Grounding predictions in logical rules over concepts provides both local (concepts) and global interpretability (rules). Experiments on NTU RGB+D 60/120 30 ; 19 and NW-UCLA 33 demonstrate we achieve competitive performance with transparent explanations. To summarize, our major contributions are: • REASON ( R ule-based E xplainable A ction via S ymbolic c ON cepts), a neurosymbolic framework for skeleton-based HAR that formulates action recognition as logical reasoning over learnable motion concepts and integrates concept-grounded representations with differentiable first-order logic for end-to-end training. • A Spatio-Temporal Concept Decoder (STC-Decoder) that extracts spatial body-configuration and temporal motion-dynamics concepts, yielding interpretable abstractions robust to execution variability. • A concept bank of spatial and dynamic concepts for actions available in NTU RGB+D and NW-UCLA. 2 Related Work Skeleton-based HAR. ST-GCN 37 established the GCN paradigm by modeling joints as graph nodes with spatio-temporal convolutions. Subsequent work (Fig (a)-(i),(ii)) addresses topology rigidity: 2s-AGCN 31 learns adaptive joint relationships, CTR-GCN 5 refines topology channel-wise, and InfoGCN 6 incorporates information-theoretic objectives. Recent architectures like HD-GCN 17 and BlockGCN 40 capture hierarchical interactions. Self-supervised approaches (Fig (a)-(iv)) including SkeletonMAE 35 and MotionBERT 41 improve generalization via masked reconstruction. Despite strong performance, these models lack interpretable action semantics. Vision-Language for HAR. Recent works have used LLMs for skeleton-based HAR (Fig (a)-(iii),(iv)). SUGAR 38 and LLM-AR 26 leverage LLM priors for classification or incorporate action prompts 36 . However, these methods yield generic, non-instance-specific explanations and cannot distinguish execution variants (e.g., arms-first vs. legs-first), and are impractical for edge deployment. In contrast, our framework learns efficient, compositional logical rules over motion predicates that capture multiple execution pathways. Figure 1: (a) Overall neurosymbolic, concept-grounded framework for skeleton-based human activity recognition. (b) STC-Decoder for decoupling encoder features into spatial and temporal streams to learn spatio-temporal motion concepts. (c) First-order logic layers, for learning logical predicates and compositional rules over predicted concepts. (d) Example visualization for a Jump action, showing top activated concepts and learned rules. Explainable HAR and Neurosymbolic AI. Prior interpretable HAR work has focused on RGB video via post-hoc concept attribution 16 or neurosymbolic pipelines with expert-defined rules 24 , inheriting privacy concerns and sensitivity to environmental noise. While neurosymbolic systems enable traceable reasoning 10 , skeleton data exhibit substantial subject and viewpoint variability, making hand-crafted rules brittle. We address this with learnable concept predicates combined with differentiable logic layers to discover action-defining rules from data. Concept Bottleneck Models. CBMs constrain representations to interpretable concept layers, enabling intervention and debugging 15 , with extensions like Logic CBMs 32 incorporating differentiable logic for concept compositions. Recent work further automates concept bank construction using LLM priors 22 ; 13 , reducing manual effort. Despite success in static image classification tasks 14 , CBMs have not addressed skeleton-based HAR, which requires joint modeling of spatial and temporal dynamics. We bridge this gap by formulating skeleton-based HAR as logical reasoning over learnable spatio-temporal motion concepts. 3 Method Our work centres on the question: Can concepts serve as an intermediate representation for neurosymbolic skeleton-based HAR? We address this by first constructing a concept bank of spatial and temporal motion primitives (Sec. 3.1 ), and then developing a framework to predict concepts from skeleton sequences (Sec. 3.2 & 3.3 ) and composing them via differentiable