Minimalist Explanation Generation and Circuit Discovery

P. Suhail, A. Anand, A Sethi · 2025

Read Minimalist Explanation Generation and Circuit Discovery by P. Suhail, A. Anand, A Sethi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25686.txt.

Introduction

Suhail, Anand, and Sethi make short whys for image model calls that also map its paths in this twenty twenty-five paper. A trained model holds scores of rules that could back one call it makes. In vast image space those rules are hard to spot and read by eye.

Inside, a light code net learns masks that keep just the zones a call turns on. Drill goals span layer match, call keep, mask thinness, and faith to the true call. Those masks then feed a path hunt that scores links by weight times spark. Path graphs at the wire scale show which wires drive the call in the end. The pair of tools bridges short whys and deep how-it-works reads.

Catalog record

CREATORP. Suhail, A. Anand, A Sethi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25686

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Minimalist Explanation Generation and Circuit Discovery P. Suhail, A. Anand, A Sethi Abstract Machine learning models, by virtue of training, learn a large repertoire of decision rules for any given input, and any one of these may suffice to justify a prediction. However, in high-dimensional input spaces, such rules are difficult to identify and interpret. In this paper, we introduce an activation-matching–based approach to generate minimal and faithful explanations for the decisions of pre-trained image classifiers. We aim to identify minimal explanations that not only preserve the model’s decision but are also concise and human-readable. To achieve this, we train a lightweight autoencoder to produce binary masks that learns to highlight the decision-wise critical regions of an image while discarding irrelevant background. The training objective integrates activation alignment across multiple layers, consistency at the output label, priors that encourage sparsity, and compactness, along with a robustness constraint that enforces faithfulness. The minimal explanations so generated also lead us to mechanistically interpreting the model internals. In this regard we also introduce a circuit readout procedure wherein using the explanation’s forward pass and gradients, we identify active channels and construct a channel-level graph, scoring inter-layer edges by ingress weight magnitude times source activation and feature-to-class links by classifier weight magnitude times feature activation. Together, these contributions provide a practical bridge between minimal input-level explanations and a mechanistic understanding of the internal computations driving model decisions. 1 Introduction The ability to generate explanations is key to making the decisions of modern machine learning models transparent and trustworthy. While deep neural networks achieve impressive predictive accuracy, their outputs arise from complex, high-dimensional computations that are not directly interpretable. Such models learn a wide range of decision rules during training, and any one of these may suffice for a given input. Without explanations, however, it is difficult to determine which rule was used or which aspects of the input were responsible for the prediction. The opacity of such processes means that the precise basis of a decision often remains hidden, limiting transparency and accountability. A natural way to make explanations more interpretable is to focus on minimality. By isolating the smallest subset of input features sufficient to support a prediction, one obtains an explanation that is both faithful to the model’s internal computation and human-readable. Minimal explanations highlight a compact subset of pixels in the case of images, or features in general, that directly support the output. Such explanations serve not only as cognitive aids for human understanding but also as a practical diagnostic tool: they can explain counterfactual behaviors, highlight shortcut learning, and reveal when the model relies on inappropriate evidence. This is critical in safety-sensitive applications such as medical diagnostics, autonomous driving, and security, where knowing the precise basis for a decision can determine whether the system is trustworthy. In this work, we propose an activation-matching based approach that, given an input image and a frozen pretrained classifier, trains a lightweight autoencoder to generate a binary mask selecting the minimal set of pixels needed to preserve the model’s prediction and internal activations. We further leverage these explanations to uncover concise, channel-level views of the model’s computation, revealing sparse, data-dependent subcircuits sufficient for the decision. Beyond the forward activation pathways, we also couple this analysis with gradient information, tracing back the most prominent gradients from the output class logit toward earlier layers and the input. This dual perspective—forward activations and backward gradients—enables us to connect minimal input-level explanations with both mechanistic insight into signal flow and attribution of decision-critical pathways, yielding a more comprehensive understanding of how deep networks arrive at their predictions. 2 Prior Work 2.1 Model Explainability via Inversion Inversion techniques aim to reconstruct input patterns that trigger specific outputs or internal activations in a neural network. Unlike explanations, which are inherently tied to a particular input and its corresponding decision, inversion seeks to synthesize representative stimuli that reveal what a model has learned in aggregate. Early work on multilayer perceptrons applied gradient-based inversion to visualize decision boundaries, though the resulting reconstructions were often noisy or adversarial-like Kindermann & Linden 1990 ; Jensen et al. 1999 ; Saad & Wunsch 2007 . To address these limitations, researchers explored evolutionary search and constrained optimization Wong 2017 . Subsequent advances incorporated prior-based regularization, such as smoothness constraints or pretrained generative models, which enhanced both realism and interpretability of reconstructions Mahendran & Vedaldi 2014 ; Yosinski et al. 2015 ; Mordvintsev et al. 2015 ; Nguyen et al. 2016 ; Nguyen et al. 2017 . More recently, methods have emerged that stabilize inversion by learning surrogate loss landscapes Liu et al. 2022 , while generative approaches conditionally reconstruct inputs likely to produce a given output Suhail & Sethi 2024 . Alternative formulations as in Suhail 2024 encode classifiers into CNF constraints, framing inversion as a deterministic satisfiability problem. 2.2 Input-Level Explainability While inversion focuses on global characterizations of model behavior, input-level explanation methods aim to provide faithful rationales for specific predictions. Explainable AI has thus developed into a central research field Ali et al. 2023 ; Hsieh et al. 2024 ; Gilpin et al. 2018 , driven by the demand for trust, transparency, and accountability in high-stakes applications. Among post-hoc attribution methods, LIME builds local surrogate models to approximate decision boundaries Hamilton et al. 2022 , whereas Grad-CAM highlights salient image regions through gradient-weighted activations Selvaraju et al. 2019 . More recent advances emphasize concept-based explanations that map predictions to semantically interpretable parts Lee et al. 2025 . Evaluating such methods remains an open challenge: surveys have highlighted the importance of rigorous metrics combining fidelity, stability, and human-centered evaluation Zhou et al. 2021 . Explanations are also being embedded into interactive systems, allowing users to guide, debug, or refine models through explanation-driven feedback Teso et al. 2022 . Beyond heuristics, Ignatiev et al. 2018 also explore abductive reasoning approaches that provide subset- or cardinality-minimal explanations with formal guarantees. 2.3 Mechanistic Interpretability of Circuits Mechanistic interpretability investigates the circuits within a model—sparse subnetworks of neurons and connections that implement particular algorithms. Minimal explanations can reveal the smallest sufficient evidence for a prediction, offering insights into how internal components drive decisions. Early studies of circuits relied on manual inspection, limiting scalability. Recent approaches automate this process: ACDC Conmy et al. 2023 introduced a systematic framework that rediscovered known transformer circuits through activation patching. Building on this, Rajaram et al. 2024 extended circuit discovery to vision models, extracting subnetworks responsible for concept recognition and demonstrating that targeted edits could alter predictions and enhance robustness. Further work Nainani et al. 2024 explored how circuits generalize across diverse inputs, revealing that networks often reuse a shared set of components while flexibly adapting their connectivity—a manifestation of representational superposition. Figure 1: Schematic Approach to generating mask ’m’ and explanation ’e’ for an image ’x’ by matching activations across different layers of a frozen classifier ’f’. 3 Methodology We aim to generate minimal, faithful explanations for a frozen classifier f on any input image x , and to use these explanations to expose compact internal circuits that drive its decisions. Our framework has two stages: (i) explanation generation via activation alignment and sparsity-inducing priors as shown in Figure 1 , and (ii)

Read the complete text