Pando: Do Interpretability Methods Work When Models Won’t Explain Themselves?
Ziqian Zhong, Aashiq Muhamed, Mona T. Diab, Virginia Smith, Aditi Raghunathan · 2026
Read Pando: Do Interpretability Methods Work When Models Won’t Explain Themselves? by Ziqian Zhong, Aashiq Muhamed, Mona T. Diab, Virginia Smith, Aditi Raghunathan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11061.txt.
Introduction
Zhong and four colleagues introduce Pando, a benchmark for interpretability when models will not explain themselves. Many evaluations ignore whether black-box prompting recovers target behaviour. White-box gains may then reflect elicitation, not internal signal. Pando uses an axis: models give faithful, no, or unfaithful explanations of a distractor rule. Across 720 models with hidden rules, agents predict held-out decisions from pairs plus one output. Faithful explanations let black-box elicitation match white-box methods. With absent or misleading explanations, gradient attribution improves scores by several points.
Catalog record
| CREATOR | Ziqian Zhong, Aashiq Muhamed, Mona T. Diab, Virginia Smith, Aditi Raghunathan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 96878 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11061
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pando: Do Interpretability Methods Work When Models Won’t Explain Themselves? Ziqian Zhong, Aashiq Muhamed, Mona T. Diab, Virginia Smith, Aditi Raghunathan Abstract Mechanistic interpretability is often motivated for alignment auditing, where a model’s verbal explanations can be absent, incomplete, or misleading. Yet many evaluations do not control whether black-box prompting alone can recover the target behavior, so apparent gains from white-box tools may reflect elicitation rather than internal signal; we call this the elicitation confounder . We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis : models are trained to produce either faithful explanations of the true rule, no explanation, or confident but unfaithful explanations of a disjoint distractor rule. Across 720 finetuned models implementing hidden decision-tree rules, agents predict held-out model decisions from 10 10 labeled query–response pairs, optionally augmented with one interpretability tool output. When explanations are faithful, black-box elicitation matches or exceeds all white-box methods; when explanations are absent or misleading, gradient-based attribution improves accuracy by 3–5 percentage points, and relevance patching, RelP, gives the largest gains, while logit lens, sparse autoencoders, and circuit tracing provide no reliable benefit. Variance decomposition suggests gradients track decision computation , which fields causally drive the output, whereas other readouts are dominated by task representation , biases toward field identity and value. We release all models, code, and evaluation infrastructure. 0 0 footnotetext: This paper annotated with replication instructions is available at https://ar-forum.github.io/Pando/livepaper.html . 1 Introduction Mechanistic interpretability aims to reveal how models compute their outputs by inspecting internal representations and circuits ( 22 ; 4 ) . This is especially appealing for alignment auditing , where auditors must assess whether a seemingly helpful model pursues hidden objectives not apparent from surface behavior ( 17 ) . In such settings, a model’s own verbal explanations are unreliable: models can produce fluent rationales that do not reflect the computation actually driving their outputs (chain-of-thought unfaithfulness; 25 ). If we cannot trust what models say about themselves, we need tools that can look inside. Do current white-box interpretability methods improve our ability to predict model behavior beyond what is available from black-box interaction alone? This question is easy to answer incorrectly because of the black-box elicitation confounder : when the target behavior is recoverable from black-box interaction alone, apparent gains from white-box tools may simply reflect better elicitation rather than additional internal signal. To isolate interpretability-specific value, benchmarks should explicitly control the explanation channel (faithful / none / misleading) and compare each white-box method to a budget-matched black-box baseline given the same query–response transcript. Model Organism Planted rule Decision trees Finetuned LLMs 720+ LoRAs Evaluation I/O pairs White-box / black-box info Rule extraction p GPT-5.1 Held-out prediction GPT-4.1 Explanation Axis Controls what the verbal channel reveals Faithful True rule verbalized None No explanation Unfaithful Wrong rule verbalized varies controls black-box info Trained with faithful explanation Held-out prediction accuracy (%) blackbox baseline prefill (blackbox) +8.8 relp +6.4 gradient +2.8 logit lens –0.2 sae –1.0 circuit tracer –2.3 Trained with unfaithful explanation Held-out prediction accuracy (%) blackbox baseline relp +3.7 gradient +2.6 logit lens +0.5 circuit tracer +0.5 sae –0.1 prefill (blackbox) –1.9 Figure 1: Overview of Pando. A decision tree is sampled and finetuned into a language model. The explanation axis controls whether the model explains itself faithfully, not at all, or with a misleading distractor rule. An agent receives k{=}10 query–response pairs plus optional white-box tool outputs and must predict the model’s decisions on held-out inputs. Right panels show prediction accuracy averaged over scenarios and models. Blackbox baseline (sample_only) receives only query–response pairs. We introduce Pando (Figure 1 ), a model-organism benchmark ( 13 ) designed to avoid the elicitation confounder. The key design choice is an explanation axis that directly controls the quality of black-box information: models are trained to provide a faithful explanation of the true rule, no explanation, or a confident but unfaithful explanation of a disjoint distractor rule. Together with a fixed query budget k , this controls both what each interaction reveals and how many interactions are available, letting us measure exactly when white-box tools add value beyond what prompting alone provides. We sample depth 1–4 decision-tree rules over p labeled input fields and finetune language models so that the rule is internalized in model parameters. Given only k{=}10 labeled query–response pairs, optionally augmented with the output of a single interpretability method computed on the same queries (e.g., gradient attributions), an agent must predict the model’s decisions on held-out inputs. Because the planted rule is known, Pando provides exact ground truth about which fields and thresholds determine the output and supports both end-to-end metrics (held-out accuracy, field F1) and direct mechanistic analyses of tool signal. When explanations are faithful, black-box prompting matches or exceeds all white-box methods. When explanations are absent or misleading, gradient-based attribution, and especially relevance patching ( 14 ) , provides consistent gains (+3–5 pp in held-out accuracy), while representation-based ( 20 ) , SAE-based ( 6 ) , and circuit-tracing ( 1 ) methods provide limited benefit despite internal access. A variance decomposition suggests that gradient scores align with decision computation (field-level decision relevance), whereas many other readouts are dominated by task representation effects (field identity/value) unrelated to the finetuned rule. White-box access helps only when the readout isolates decision-relevant features. An automated research loop (78 experiments, {\sim} 25.5 hours) finds only modest gains beyond gradient attribution. Taken together, these results suggest that auditing can remain difficult even in this controlled, favorable setting. Our contributions include: 1. Pando , a controllable model-organism benchmark that isolates interpretability-specific signal from black-box elicitation via an explanation axis (faithful/none/unfaithful). We finetune 720 models to implement randomly sampled depth 1–4 decision trees over 10 labeled input fields, and release the models, code, and evaluation infrastructure. 2. An empirical evaluation of black-box baselines and white-box interpretability methods on Pando across this explanation axis. When explanations are faithful, black-box prompting matches or exceeds all white-box methods; when explanations are absent or unfaithful, gradient-based attribution (especially RelP) yields consistent gains (+3–5 pp in held-out accuracy), while representation-based, SAE-based, and circuit-tracing methods provide limited benefit. An automated research loop (78 experiments, {\sim} 25.5 hours) finds only incremental improvements beyond gradient attribution. 3. A mechanistic analysis explaining why most white-box methods fail despite internal access. A variance decomposition separates task representation (field identity and value) from decision computation (decision relevance): gradient methods track the latter, while other readouts are dominated by the former. 2 Benchmark Design Pando evaluates behavior recovery : can an agent predict a model’s decisions on unseen inputs from limited interaction, and when do white-box tools help beyond the black-box transcript? Each scenario defines a tabular input space \mathcal{X} with p{=}10 named fields (field names are shown explicitly in the input); an input is x=(x^{(1)},\ldots,x^{(p)}) , and the model outputs a binary label y\in\{0,1\} . For each model instance, we sample a depth- d decision-tree rule T:\mathcal{X}\to\{0,1\} and finetune a language model f_{\theta} to implement T . At evaluation time, each agent receives the same k{=}10 random query–response pairs D_{Q}=\{(x_{i},y_{i}=f_{\theta}(x_{i})):i\in Q\} with |Q|=k . Agents equipped with an interpretability method \mathcal{I} additionally receive tool outputs \{\mathcal{I}(f_{\theta},x_{i})\}_{i\in Q} computed on the same queries. Performance is measured by