Dual Path Attribution: Efficient Attribution for SwiGLU-Transformers through Layer-Wise Target Propagation
Lasse Jantsch Dong-Jae Koh Seonghyeon Lee Young-Kyoon Suh School of Computer Science and Engineering, Kyungpook National University, South Korea {lassejantsch, djkoh, sh0416, yksuh}@knu.ac.kr · 2026
Read Dual Path Attribution: Efficient Attribution for SwiGLU-Transformers through Layer-Wise Target Propagation by Lasse Jantsch Dong-Jae Koh Seonghyeon Lee Young-Kyoon Suh School of Computer Science and Engineering, Kyungpook National University, South Korea {lassejantsch, djkoh, sh0416, yksuh}@knu.ac.kr (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19742.txt.
Introduction
Jantsch and co-authors introduce Layer-wise Target Propagation, a fast way to credit transformer parts. Dense attribution for large language models is usually far too costly. Their method works on the frozen model in one forward and one backward pass. It needs no counterfactual examples. A target vector is pushed through linearized paths in the network. That yields an effective representation at each residual position. The cost stays flat as the number of components grows. Benchmark tests showed top faithfulness with unmatched speed.
Catalog record
| CREATOR | Lasse Jantsch Dong-Jae Koh Seonghyeon Lee Young-Kyoon Suh School of Computer Science and Engineering, Kyungpook National University, South Korea {lassejantsch, djkoh, sh0416, yksuh}@knu.ac.kr |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72885 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19742
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dual Path Attribution: Efficient Attribution for SwiGLU-Transformers through Layer-Wise Target Propagation Lasse Jantsch Dong-Jae Koh Seonghyeon Lee Young-Kyoon Suh School of Computer Science and Engineering, Kyungpook National University, South Korea {lassejantsch, djkoh, sh0416, yksuh}@knu.ac.kr Abstract Understanding the internal mechanisms of transformer-based large language models (LLMs) is crucial for their reliable deployment and effective operation. While recent efforts have yielded a plethora of attribution methods attempting to balance faithfulness and computational efficiency, dense component attribution remains prohibitively expensive. In this work, we introduce Dual Path Attribution (DPA), a novel framework that faithfully traces information flow on the frozen transformer in one forward and one backward pass without requiring counterfactual examples. DPA analytically decomposes and linearizes the computational structure of the SwiGLU Transformers into distinct pathways along which it propagates a targeted unembedding vector to receive the effective representation at each residual position. This target-centric propagation achieves O ( 1 ) O(1) time complexity with respect to the number of model components, scaling to long input sequences and dense component attribution. Extensive experiments on standard interpretability benchmarks demonstrate that DPA achieves state-of-the-art faithfulness and unprecedented efficiency compared to existing baselines 1 1 1 github.com/lab-paper-code/dual_path_attribution . † † footnotetext: * Equal Contributions † † footnotetext: † Corresponding author 1 Introduction Transformer-based large language models (LLMs) are the dominant model paradigm for general-purpose AI; however, the internal mechanisms driving this performance remain poorly understood. A central challenge is attribution, identifying which inputs or internal components are responsible for a model’s predictions. While a plethora of attribution approaches were applied to LLMs Ferrando et al. 2024 , they are often computationally expensive Sundararajan et al. 2017 ; Conmy et al. 2023 , ignore important model components Hong et al. 2025 , or suffer from approximation errors or noise Shrikumar et al. 2019 ; Balduzzi et al. 2018 . One promising direction is decomposition based approaches attempting to fully linearize the computational graph of the transformers. We propose Dual Path Attribution (DPA), a decomposition-based framework for efficient input and component attribution. By propagating the unembedding vector of the target token through the frozen transformer, DPA approximates an effective target for each residual position in one forward and one backward pass. While DPA can be applied to all transformer architectures, this paper focuses on the SwiGLU transformer, as it allows for the decomposition of its bilinear interactions into distinct pathways: a content pathway that propagates representations through the value and up projections, and a control pathway that governs their routing and transformation through attention and gate scores. Experiments on standard interpretability benchmarks demonstrate the faithfulness of DPA for both input and component attribution. Figure 1 shows a high-level overview of our approach. Figure 1: Overview of the Dual Path Attribution (DPA) framework for efficient input and component attribution. DPA operates in two stages: (1) One-pass forward execution , in which the model processes the input once while caching activations required for the back propagation; and (2) top-down target propagation , where the unembedding vector of the targeted token is recursively propagated backward through Transformer modules to identify an effective target at each residual position. Our contributions are summarized as follows: • We introduce Dual Path Attribution (DPA), a novel decomposition based framework, efficiently attributing logit scores in one forward and one backward pass without the need for counterfactual examples. • By leveraging the bilinear structure of the SwiGLU Transformers, we show that the computational graph is decomposable in distinct control and content pathways. • Through extensive experiments on standard interpretability benchmarks, we demonstrate that DPA achieves superior faithfulness and efficiency compared to state-of-the-art baselines, effectively identifying causal mechanisms. 2 Related Work Attribution research for Transformer-based language models broadly aims to localize the sources of model behavior, either by attributing predictions to input tokens or by identifying responsible internal components ( Ferrando et al. 2024 ) . Input attribution. Gradient-based input attribution approaches use the first order Taylor-Expansion as importance signal approximating prediction sensitivity ( Simonyan et al. 2013 ; Denil et al. 2014 ; Li et al. 2015 ) . Input x Gradient Denil et al. 2015 calculates the dot product between the embedding vector and the gradient, including input magnitude. While gradient based approaches suffer from gradient saturation and shattering Shrikumar et al. 2019 ; Balduzzi et al. 2018 , approaches like Integrated Gradients ( Sundararajan et al. 2017 ) or SmoothGrad Smilkov et al. 2017 were shown effective to denoising the gradients. Layer-wise Relevance Propagation (LRP) ( Bach et al. 2015 ) conserves the importance signal by applying local propagation rules ( Voita et al. 2021 ; Ali et al. 2022 ; Achtibat et al. 2024 ) . Another line of previous work analyzes the flow of contextual information. Early approaches interpreted raw attention weights, which were later shown to be unreliable indicators of importance ( Jain and Wallace 2019 ) . Subsequent methods incorporated value-vector norms ( Kobayashi et al. 2020 ; Kobayashi et al. 2021 ) or propagated attention across layers using rollout techniques ( Abnar and Zuidema 2020a ; Ferrando et al. 2022 ; Modarressi et al. 2022 ) . More recent work favors decomposition-based approaches that explicitly expand the model’s linear structure, yielding more faithful attribution of attention-mediated effects ( Modarressi et al. 2023 ; Kobayashi et al. 2024 ; Hong et al. 2025 ) . Component-level attribution. Beyond input tokens, recent studies have focused on attributing internal components to the model output. Direct Logit Attribution (DLA) ( Elhage et al. 2021 ; nostalgebraist 2020 ) exploits the additive structure through the residual connection to measure a component’s direct contribution to output logits, and has been applied to both attention heads ( Ferrando et al. 2023 ) and individual neurons ( Geva et al. 2022 ) . DLA, however, does not capture indirect effects of downstream layers, leading to decreased interpretability in early layers. Causal intervention methods (i.e., activation patching) resolve this limitation by measuring direct and indirect effects via counterfactual forward passes ( Vig et al. 2020 ; Meng et al. 2022 ; Wang et al. 2022 ; Zhang and Nanda 2023 ) . While highly faithful, activation patching is computationally expensive for dense attribution as it requires separate forward passes for each component or interaction. Recently, automated circuit discovery – the detection of component subsets responsible for model behavior – has seen causal intervention Conmy et al. 2023 , gradient Nanda 2023 ; Syed et al. 2023 ; Syed et al. 2024 , LRP Jafari et al. 2025 , and contextual mixing Ferrando and Voita 2024 based approaches. 3 Transformer Architecture This section reviews the Transformer components relevant to our method and establishes the notation used throughout the paper. We focus on the Swish-Gated Linear Unit (SwiGLU) ( Shazeer 2020 ) architecture, as it is the current state-of-the art for dense Transformers. A general overview of decoder-only Transformers is provided in Appendix A . We formalize our exposition based on the Llama2-style architecture ( Touvron et al. 2023 ; Grattafiori et al. 2024 ) , though our approach directly generalizes to other SwiGLU-based Transformers with minimal adjustment. 3.1 Multi-Head Self-Attention The multi-head self-attention (MHSA) module differs from the original Transformer ( Radford et al. 2019 ) by employing