Geometry-Adaptive Explainer for Faithful Dictionary-Based Interpretability under Distribution Shift
Sungjun Lim, Heedong Kim, Andrew Lee, Kyungwoo Song, [1em] Yonsei University · 2026
Read Geometry-Adaptive Explainer for Faithful Dictionary-Based Interpretability under Distribution Shift by Sungjun Lim, Heedong Kim, Andrew Lee, Kyungwoo Song, [1em] Yonsei University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21849.txt.
Introduction
Sungjun Lim and coauthors show distribution shift rotates the subspace a model uses, skewing dictionary explainers trained on in-distribution activations. They define this skew as the faithfulness gap, a geometric distance between that dictionary and the shifted subspace, which controls faithfulness degradation. Their Geometry-Adaptive Explainer realigns the dictionary with the shifted subspace while keeping feature structure, needing raw shifted activations and no gradient updates. Theory bounds excess loss by the second-moment shift squared, and tests show it matches or beats training baselines on causal faithfulness.
Catalog record
| CREATOR | Sungjun Lim, Heedong Kim, Andrew Lee, Kyungwoo Song, [1em] Yonsei University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89440 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.21849
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Geometry-Adaptive Explainer for Faithful Dictionary-Based Interpretability under Distribution Shift Sungjun Lim, Heedong Kim, Andrew Lee, Kyungwoo Song, [1em] Yonsei University Abstract Mechanistic interpretability aims to explain a model’s behavior by identifying causally responsible internal structures. Dictionary-based explainers such as sparse autoencoders and transcoders are a primary tool, but their faithfulness under out-of-distribution (OOD) shift has received little systematic attention. We show that distribution shift rotates the subspace that the model actively uses, misaligning the explainer’s dictionary trained on in-distribution (ID) activations. We formalize this misalignment as the faithfulness gap , a geometric distance between the ID dictionary and the OOD-active subspace, and show that it controls OOD faithfulness degradation. To reduce this gap, we propose the Geometry-Adaptive Explainer (GAE) , which realigns the explainer’s dictionary with the OOD-active subspace while preserving the original feature structure. This requires only unlabeled OOD activations and no gradient updates. We prove that GAE improves over the unadapted ID explainer, with excess loss bounded quadratically by the second-moment shift. Empirically, GAE even matches or surpasses all training-based baselines in causal faithfulness across multiple models and OOD settings. 1 1 1 Code is available at: https://github.com/MLAI-Yonsei/GAE/ * * footnotetext: Corresponding authors: [email protected] , [email protected] . 1 Introduction Mechanistic interpretability aims to explain a model’s behavior by identifying internal structures that are causally responsible for its outputs [ 18 , 38 , 45 ] . A primary approach is to train an explainer, a post-hoc module such as a sparse autoencoder (SAE) [ 7 ] or transcoder [ 13 ] , that decomposes hidden activations into sparse combinations of learned feature directions (a dictionary). These dictionary-based explainers have recently been scaled to large language models [ 46 , 17 ] and used to uncover interpretable feature circuits [ 33 ] . A central requirement for such explanations is faithfulness: they should accurately reflect the computations the model actually uses [ 22 , 18 ] . When a model encounters out-of-distribution (OOD) inputs, the dictionary learned in-distribution (ID) can no longer capture the directions the model actively uses [ 21 ] . Prior work has addressed this vulnerability from two angles, each limited in scope. Attribution-level robustness studies [ 1 , 19 , 31 , 4 ] focus on input perturbations rather than hidden-state geometry. Dictionary-explainer remedies such as retraining on the model’s own generations [ 9 ] , upweighting rare concepts [ 29 , 37 ] , and adding residual modules [ 24 ] remain heuristic, without diagnosing the underlying misalignment. Consequently, the mechanism driving this failure and a principled correction remain unaddressed. In this work, we identify a geometric mechanism for OOD faithfulness degradation in dictionary-based explainers. These explainers learn their feature directions from ID activations, so their dictionaries reflect the geometric structure of ID hidden representations [ 5 ] . This structure is captured by the second moment of hidden activations, which distribution shift typically alters [ 27 ] , leaving the ID-trained dictionary misaligned with the OOD-active subspace. As illustrated in Figure 1 (left), we call this misalignment the faithfulness gap , a geometric distance between the ID dictionary and the OOD-active subspace. We prove that this gap controls OOD faithfulness degradation and that it is itself upper-bounded by the magnitude of the second-moment shift. Reducing this gap is therefore necessary for restoring OOD faithfulness, motivating methods that directly realign the dictionary. Figure 1 : Faithfulness gap and GAE. Left : distribution shift (illustrated as a language change) rotates the OOD-active subspace \Pi_{\mathrm{OOD}} away from the ID-trained explainer subspace \Pi_{\mathrm{dec}}\approx\Pi_{\mathrm{ID}} , opening a faithfulness gap \Delta(\Pi_{\mathrm{dec}}) . Right : GAE closes this gap in two steps. Step 1 rotates \Pi_{\mathrm{dec}} onto \Pi_{\mathrm{OOD}} via orthogonal Procrustes. Step 2 refits individual feature directions within the aligned subspace to match OOD activations while preserving the original feature structure. We instantiate this idea with the Geometry-Adaptive Explainer (GAE) , a closed-form, post-hoc method that closes the faithfulness gap using only unlabeled OOD activations. GAE first rotates the ID-trained dictionary so that its subspace aligns with the OOD-active subspace, choosing the rotation closest to the original dictionary to preserve feature structure (Step 1 in Figure 1 ). A constrained decoder refit then adjusts individual feature directions to match OOD activations while maintaining this alignment (Step 2 in Figure 1 ). The entire pipeline requires no gradient computation, yet we prove it is guaranteed to improve over the unadapted explainer (Theorem 1 ). Empirically, GAE matches or surpasses all training-based baselines in causal faithfulness across multiple language models and OOD settings, including methods that retrain from scratch on OOD data. Our main contributions are summarized as follows: • We identify the faithfulness gap as a geometric mechanism for OOD faithfulness degradation and prove that excess loss grows at most quadratically with second-moment shift. • We propose GAE , a closed-form dictionary realignment method that targets the faithfulness gap with a theoretical guarantee while preserving feature structure. • Without any gradient computation, GAE matches or surpasses all training-based baselines, including full OOD retraining, in causal faithfulness across multiple models and diverse OOD settings. 2 Related Work 2.1 Faithfulness in Mechanistic Interpretability Dictionary-based explainers such as SAEs [ 7 , 10 ] and transcoders [ 13 , 39 ] decompose hidden activations into sparse feature directions and have become a primary tool for mechanistic interpretability [ 18 , 38 , 46 , 17 , 33 ] . Faithfulness is typically evaluated via causal interventions that ablate features and measure the effect on model outputs [ 22 , 15 , 12 ] , and is a prerequisite for reliable circuit discovery and model editing [ 33 , 8 , 36 ] . These explainers’ dictionaries reflect the geometric structure of ID hidden representations [ 5 ] , so distribution shift can degrade faithfulness by misaligning the learned directions with those the model actively uses. Recent empirical reports support this concern: SAE-based features underperform dense baselines on OOD downstream tasks [ 21 ] , yet this vulnerability has received little systematic attention, with no formal diagnosis of its cause. 2.2 Interpretability under Distribution Shift The faithfulness assumption above becomes problematic when the model encounters OOD inputs. Early work showed that saliency maps are fragile under input perturbations [ 1 , 19 ] . Subsequent studies examined explanation consistency under shift more broadly [ 4 , 31 ] . These analyses primarily concern attribution methods and attribute failures to predictive degradation or input-level perturbations, rather than to structural changes in hidden representations. Yet evidence from OOD detection shows that OOD inputs produce statistically distinguishable activation patterns in hidden layers [ 27 ] . Representation comparison methods [ 25 , 42 , 47 ] enable quantifying such geometric changes across conditions, but the connection to explainer faithfulness has not been drawn. For dictionary-based explainers, proposed remedies include retraining the explainer on OOD data, training on the model’s own generations to avoid external dataset dependence [ 9 ] , upweighting tail samples during training [ 29 , 37 ] , or adding residual capacity via boosting [ 24 ] . However, these approaches either require full retraining or do not directly address the geometric misalignment between the explainer’s learned dictionary and the OOD-active subspace. Post-hoc methods that