NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions
Tue M. Cao 1 Nhat X. Hoang 2 Hieu H. Pham 3 Phi Le Nguyen 1* My T. Thai 2 · 2025
Read NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions by Tue M. Cao 1 Nhat X. Hoang 2 Hieu H. Pham 3 Phi Le Nguyen 1* My T. Thai 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16105.txt.
Introduction
Tue Cao and co-authors present a framework for interpreting neural networks by neuron groups rather than single neurons in a 2025 paper. Single neurons often encode several unrelated features, which makes neuron-by-neuron reading messy and costly. Groups of neurons that work together may give a cleaner picture of what a network computes.
Their automated system first finds core neurons, then clusters them by shared function. From those clusters it builds a layered circuit of interactions across the network. Tests check that the circuits faithfully reflect the model's behavior. The team also show uses in debugging image models and in labeling concepts automatically. The approach aims to make large networks easier to audit.
Catalog record
| CREATOR | Tue M. Cao 1 Nhat X. Hoang 2 Hieu H. Pham 3 Phi Le Nguyen 1* My T. Thai 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-16105
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions Tue M. Cao 1 Nhat X. Hoang 2 Hieu H. Pham 3 Phi Le Nguyen 1* My T. Thai 2 Abstract Understanding the inner workings of neural networks is essential for enhancing model performance and interpretability. Current research predominantly focuses on examining the connection between individual neurons and the model’s final predictions, which suffers from challenges in interpreting the internal workings of the model, particularly when neurons encode multiple unrelated features. In this paper, we propose a novel framework that transitions the focus from analyzing individual neurons to investigating groups of neurons, shifting the emphasis from neuron-output relationships to the functional interactions between neurons. Our automated framework, NeurFlow, first identifies core neurons and clusters them into groups based on shared functional relationships, enabling a more coherent and interpretable view of the network’s internal processes. This approach facilitates the construction of a hierarchical circuit representing neuron interactions across layers, thus improving interpretability while reducing computational costs. Our extensive empirical studies validate the fidelity of our proposed NeurFlow. Additionally, we showcase its utility in practical applications such as image debugging and automatic concept labeling, thereby highlighting its potential to advance the field of neural network explainability. 4 4 4 Source code: https://github.com/tue147/neurflow 1 1 footnotetext: Institute for AI Innovation and Societal Impact (AI4LIFE), Hanoi University of Science and Technology, Hanoi, Vietnam ([email protected], [email protected]) . 2 2 footnotetext: University of Florida, Gainesville, Florida, USA {hoangx, mythai}@ufl.edu . 3 3 footnotetext: VinUni-Illinois Smart Health Center, VinUniversity, Hanoi, Vietnam ( [email protected] ). 1 Introduction The explainable AI (XAI) field has seen significant advancement in understanding the mechanisms of deep neural networks (DNNs). This field emerges from the growing need in decoding the internal representations, in hope of reverse engineering deep models into human interpretable program. Prior works have initiated on breaking down convolutional neural networks (CNNs) into interpretable neurons, understanding the models in the most fundamental units ( Nguyen et al. 2016 ; Zeiler & Fergus 2014 ; O’Mahony et al. 2023 ; Bykov et al. 2024 ) . Extending further, one can examine the relation between neurons to gain insights on how the model works, within one layer ( Vu et al. 2022 ) , and between multiple layers ( Cammarata et al. 2020 ) . Ultimately, recent works try to generate circuits ( Cammarata et al. 2020 ; Bykov et al. 2024 ; Wang et al. 2022c ; Conmy et al. 2023 ) that create exhaustive explanations of how features are processed and evolve throughout the model. The majority of existing methods focuses on individual neurons Oikarinen & Weng 2024b ; La Rosa et al. 2023a and their relationship to the model’s final predictions Ghorbani & Zou 2020b ; Wang et al. 2022b ; Ghorbani & Zou 2020a , while giving less attention to exploring and quantifying the relationships and interactions between neurons across different layers. These approaches are not only constrained by scalability challenges arising from the extensive number of neurons, but they also hinder a comprehensive understanding of the underlying mechanisms of DNNs. A notable example is the polysemantic phenomenon Mu & Andreas 2020 ; O’Mahony et al. 2023 ; Olah et al. 2020 , where a single neuron is activated by several unrelated concepts. This phenomenon complicates the task of associating each neuron with a distinct feature and hampers the interpretation of how a model processes concepts based on the relationships among neurons. Drawing inspiration from human inference, which synthesizes information from a variety of sources, we contend that, in addition to individual neuron encoding multiple concepts (as demonstrated in prior studies O’Mahony et al. 2023 ; Olah et al. 2020 ), groups of neurons within each layer also collectively encode the same concept. Furthermore, the decision-making process in neural networks is shaped not solely by the interactions between individual neurons, but rather by interactions among neuron groups. This study seeks to explore the roles and interactions of neuron groups in shaping and developing concepts, enabling the execution of specific tasks. Due to the complex connections between large number of neurons, identifying those functions and there interactions is a daunting task. To overcome this, we demonstrate that for a particular task, only a subset of neurons—referred to as core concept neurons —play a crucial role as influential and concept-defining elements in neural networks. These neurons, when deactivated, significantly alter the associated concepts. Focusing on core concept neurons allows us to view the intricate network in a simplified way, revealing the most important interactions between the groups of neurons. Therefore, we propose NeurFlow framework that (1) identifies core concept neurons, (2) clusters these neurons into groups, and (3) investigates the functions and interactions of these groups. To enhance interpretability, we represent each neuron group by the set of visual features it encodes (i.e., named as neuron group’s concept). Focusing on classification models, we construct, for each class of interest, a hierarchical tree in which nodes represent neuron groups (defined by the concepts they encode), and edge weights quantify the interactions between these groups. Our key contributions are summarized as follows: i) We introduce an innovative framework that systematically builds a circuit to elucidate the mechanisms by which core concept neuron groups operate and interact to achieve specific tasks. This entire process is automated, necessitating no human intervention or predefined concept labels. To our knowledge, we are the first to employ neuron groups as the fundamental units for explaining the internal workings of deep neural networks. ii) We perform empirical studies to validate the proposed framework, demonstrating the optimality and fidelity of core concept neurons, and the reliability of interaction weights between core concept neuron groups. iii) We provide experimental evidence showing that our framework can be applied to various tasks, including image debugging and automatic neuron concept labeling. Specifically, we confirm the biases found by Kim et al. 2024 on ImageNet ( Russakovsky et al. 2015 ) , which have not been proven, by masking the core concept neurons related to the biased features. 2 Related work In an effort to understand the inner mechanism of DNNs, several branches of research have emerged: Concept based. Kim et al. 2018 show that a model can be rigorously understood by assigning meaning to the activations, referred to as concept activation vectors. Subsequent works ( Ghorbani et al. 2019 ; Zhang et al. 2021 ) have explored more complex methods for extracting these meanings, however, the relationships between concepts remain understudied. Fel et al. 2023 ; Kowal et al. 2024 address this limitation by constructing a graph of concepts with edges that quantify the relations. Their main intention is to see the evolution of concepts throughout the network layers. Nevertheless, they are unable to explain which parts of the model are responsible for these concepts. Neuron based. Nguyen et al. 2016 ; Mu & Andreas 2020 ; O’Mahony et al. 2023 ; Bykov et al. 2024 invest effort in studying the meaning of neurons, in parallel, Vu et al. 2022 ; Ghorbani & Zou 2020c ; Khakzar et al. 2021b propose different approaches in identifying important neurons to the model output. These researches shed light on the function of individual neurons and their impact