Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning
Md Muntaqim Meherab, Noor Islam S. Mohammad, Faiza Feroz · 2026
Read Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning by Md Muntaqim Meherab, Noor Islam S. Mohammad, Faiza Feroz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10377.txt.
Introduction
Md Muntaqim Meherab, Noor Islam S Mohammad, and Faiza Feroz propose Causal Concept Graphs for language models. The aim is to show not just where concepts sit, but how they drive each other during stepwise reasoning. Sparse autoencoders supply interpretable latent features, and differentiable structure learning recovers causal edges between them. A new Causal Fidelity Score checks whether graph-guided interventions move outputs more than random ones. On three reasoning benchmarks with GPT-2 Medium, their graphs beat tracing and ranking baselines by a wide margin, staying sparse and stable.
Catalog record
| CREATOR | Md Muntaqim Meherab, Noor Islam S. Mohammad, Faiza Feroz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25975 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10377
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning Md Muntaqim Meherab, Noor Islam S. Mohammad, Faiza Feroz Abstract Sparse autoencoders can localise where concepts live in language models, but not how they interact during multi-step reasoning. We propose Causal Concept Graphs ( CCG ): a directed acyclic graph over sparse, interpretable latent features, where edges capture learned causal dependencies between concepts. We combine task-conditioned sparse autoencoders for concept discovery with DAGMA-style differentiable structure learning for graph recovery, and introduce the Causal Fidelity Score (CFS) to evaluate whether graph-guided interventions induce larger downstream effects than random ones. On ARC-Challenge, StrategyQA, and LogiQA with GPT-2 Medium, across five seeds ( n = 15 n{=}15 paired runs), CCG achieves CFS = 5.654 ± 0.625 \mathrm{CFS}=5.654\pm 0.625 , outperforming ROME-style tracing ( 3.382 ± 0.233 3.382\pm 0.233 ), SAE-only ranking ( 2.479 ± 0.196 2.479\pm 0.196 ), and a random baseline ( 1.032 ± 0.034 1.032\pm 0.034 ), with p < 0.0001 p<0.0001 after Bonferroni correction. Learned graphs are sparse (5–6% edge density), domain-specific, and stable across seeds. 1 Introduction Mechanistic interpretability has made rapid progress: we can localise semantic features and circuits in transformers and extract sparse, monosemantic dictionaries from residual streams [ 6 , 21 , 2 , 5 , 26 ] . What remains hard is the dynamic question: for multi-step reasoning, which internal features interact, and in what order, as computation unfolds. This gap matters for reliability and safety: without tracing internal reasoning, we cannot robustly diagnose failures or distinguish genuine reasoning from shortcut strategies [ 24 ] . Existing tools only partially address this. Model editing methods (e.g., ROME/MEMIT) precisely localise single factual associations [ 17 , 18 , 9 ] but are not designed for distributed, compositional reasoning. Concept Bottleneck Models provide interpretability via an explicit concept layer [ 13 , 31 ] but require a human-specified vocabulary and supervision. We combine sparse feature discovery with causal structure learning: we first extract task-conditioned concept features from activations, then learn a DAG over those features, yielding Causal Concept Graphs (CCG) with no manual concept annotation. Contributions. We contribute (i) a task-conditioned sparse autoencoder with TopK gating and neuron resampling that achieves a stable 5.1% L0 activation rate on reasoning inputs (Section 3.1 ); (ii) a DAGMA-based causal graph learner over concept activation matrices that recovers sparse DAGs with 5–6% edge density (Section 3.2 ); (iii) the Causal Fidelity Score (CFS), a numerically stable intervention-based metric for evaluating whether the learned graph identifies concepts with large causal reach (Section 3.3 ); and (iv) multi-seed experiments on three reasoning benchmarks showing consistent, statistically significant improvements over strong baselines (Section 4 ). Figure 1 : CCG pipeline. Stage 1: task-conditioned SAE on GPT-2 Medium residual activations (Layer 12) with TopK gating ( K{=}256 , k{=}13 ; 5.1% L0). Stage 2: DAGMA learns a sparse DAG over the top-64 concepts per domain. Stage 3: CFS evaluates intervention faithfulness ( \mathrm{CFS}{=}5.654 ; p{<}0.0001 vs. baselines). 2 Related Work Mechanistic interpretability. Transformers exhibit identifiable circuits and algorithms [ 20 , 21 , 27 , 19 ] . Superposition explains widespread polysemanticity [ 6 ] , motivating sparse autoencoders for feature dictionaries [ 2 , 5 ] that remain interpretable at scale [ 26 ] . We focus on feature-to-feature interaction structure during computation. Causal tracing and model editing ROME and successors localise and edit factual associations in mid-layer computations [ 17 , 18 , 9 ] . These methods target single associations; our goal is multi-feature, multi-step causal structure. Our intervention-based evaluation is inspired by this line but operates on learned feature graphs. Causal structure learning DAG learning spans classic constraint/score-based methods [ 25 , 3 , 22 ] and continuous relaxations [ 32 ] , with DAGMA improving numerical behaviour near optima [ 1 ] . We adapt DAGMA to task-structured concept activations rather than i.i.d. tabular variables. Concept-based explanations. TCAV links user-defined concepts to predictions [ 11 ] ; CBMs and variants enforce/approximate a concept layer but rely on a predefined vocabulary or supervision [ 13 , 30 , 31 ] . CCG instead discovers concepts from activations and learns their dependencies. LLM reasoning Chain-of-thought prompting elicits explicit intermediate steps [ 28 , 14 ] , and mechanistic studies analyse specific reasoning circuits [ 27 , 15 ] . Closest in spirit are causal intervention analyses of internal computation [ 7 , 29 ] , though typically at component level; we target sparse concept features and their learned DAG structure. 3 Methodology 3.1 Stage 1: Task-Conditioned Sparse Autoencoder Let \mathbf{h}\in\mathbb{R}^{d} denote the mean-pooled residual-stream activation at layer \ell ( d{=}1024 for GPT-2 Medium). We train a sparse autoencoder [ 2 ] with TopK gating: \displaystyle\hat{\mathbf{c}} \displaystyle=\mathrm{TopK}\!\bigl(\mathbf{W}_{\mathrm{enc}}(\mathbf{h}-\mathbf{b}_{\mathrm{pre}})+\mathbf{b}_{\mathrm{enc}}\bigr), (1) \displaystyle\hat{\mathbf{h}} \displaystyle=\mathbf{W}_{\mathrm{dec}}\,\hat{\mathbf{c}}+\mathbf{b}_{\mathrm{pre}}, (2) where \mathbf{W}_{\mathrm{enc}}\in\mathbb{R}^{K\times d} , \mathbf{W}_{\mathrm{dec}}\in\mathbb{R}^{d\times K} , K{=}256 , and \mathrm{TopK} retains exactly k{=}13 nonzeros per example (5.1% L0; see Section 4.1 ). We minimise \mathcal{L}_{\mathrm{SAE}}=\|\hat{\mathbf{h}}-\mathbf{h}\|_{2}^{2}+\lambda\|\hat{\mathbf{c}}\|_{1}+\beta\bigl\|\mathrm{OffDiag}(\hat{\Sigma}_{\mathbf{c}})\bigr\|_{F}^{2}, (3) where \hat{\Sigma}_{\mathbf{c}} is the mini-batch covariance of \hat{\mathbf{c}} and \mathrm{OffDiag}(\cdot) zeros the diagonal. We use \lambda{=}5\times 10^{-2} and \beta{=}0.1 . Neuron resampling. To prevent dead features, every 10 epochs we reset any neuron with cumulative fire rate <0.5\% by reinitialising its decoder column to a unit-normalised direction sampled from high-reconstruction-loss examples in the current batch. Task conditioning. Unlike general-text SAEs [ 26 ] , we train only on reasoning prompts, which yields strongly domain-informative concept activations (Section 7 ). 3.2 Stage 2: Causal Concept Graph Learning Each example i yields a sparse concept vector \mathbf{c}_{i}\in\mathbb{R}^{K}_{\geq 0} ; stacking gives \mathbf{C}\in\mathbb{R}^{N\times K} . We select the M{=}64 most frequently active concepts and learn a weighted adjacency \mathbf{W}\in\mathbb{R}^{M\times M} via a linear SEM \mathbf{C}\approx\mathbf{C}\mathbf{W} : \min_{\mathbf{W}}\;\|\mathbf{C}-\mathbf{C}\mathbf{W}\|_{F}^{2}+\lambda_{1}\|\mathbf{W}\|_{1}+\lambda_{2}h(\mathbf{W}), (4) where h(\mathbf{W})=\mathrm{tr}(e^{\mathbf{W}\circ\mathbf{W}})-M is the DAGMA acyclicity penalty [ 1 ] ( h(\mathbf{W}){=}0 iff \mathbf{W} is a DAG), and \circ is the Hadamard product. We mask \mathrm{diag}(\mathbf{W}) to zero. Optimisation uses Adam [ 12 ] with cosine annealing for 300 epochs, \lambda_{1}{=}0.02 , \lambda_{2}{=}0.05 . We learn separate graphs per dataset. 3.3 Stage 3: Causal Fidelity Score To test whether the learned graph identifies causally influential nodes (beyond fitting correlations), we perform intervention-style evaluations inspired by do-calculus [ 22 ] . For node i , define downstream neighbours \mathcal{D}_{i}=\{j:W_{ij}>0.01\} and ablation effect \Delta_{i}=\frac{1}{|\mathcal{D}_{i}|}\sum_{j\in\mathcal{D}_{i}}\Bigl|\,[\mathbf{C}\mathbf{W}]_{\cdot j}\big|_{c_{i}=0}-[\mathbf{C}\mathbf{W}]_{\cdot j}\big|_{\text{orig}}\Bigr|_{1}. (5) The Causal Fidelity Score compares S{=}20 high-centrality targets (by out-degree) to S{=}20 random targets: \mathrm{CFS}=\frac{1}{S}\sum_{s=1}^{S}\min\!\left(\frac{\Delta_{i_{c}^{(s)}}}{\max(\Delta_{i_{r}^{(s)}},\,\delta)},\;\tau\right), (6) with \delta{=}10^{-3} (division floor) and \tau{=}10 (ratio cap). \mathrm{CFS}{=}1 corresponds to chance; \mathrm{CFS}{>}1 indicates the graph selects higher-impact nodes. 4 Experimental Setup We use GPT-2 Medium [ 23 ] (24 layers, d{=}1024 , 354.8M) with frozen weights and record residual-stream activations; all runs fit on a Tesla T4 (15.6 GB). We evaluate on ARC-Challenge [ 4 ] , StrategyQA [ 8 ] , and LogiQA [ 16 ] , using 300 examples per dataset for SAE/CCG training and the same split for evaluation. Baselines are a concept-level ROME-style tracer [ 17 ] (variance-ranked features), SAE-only (magnitude-ranked), and Random ( M{=}20 ). All methods share the same activation matrices and CFS protocol (20 intervention pairs). Results report mean \pm std over five seeds (42–46) across three datasets ( n{=}15 paired runs), with one-sided paired t -tests and Bonferroni correction. Prompt lengths differ substantially (Fig. 2 ), so we train