Learning Self-Interpretation from Interpretability Artifacts: Training Lightweight Adapters on Vector-Label Pairs
Keenan Pepper, Alex McKenzie, Florin Pop, Stijn Servaes, Martin Leitgab, Mike Vaiana, Judd Rosenblatt, Michael S. A. Graziano, Diogo de Lucena · 2026
Read Learning Self-Interpretation from Interpretability Artifacts: Training Lightweight Adapters on Vector-Label Pairs by Keenan Pepper, Alex McKenzie, Florin Pop, Stijn Servaes, Martin Leitgab, Mike Vaiana, Judd Rosenblatt, Michael S. A. Graziano, Diogo de Lucena (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10352.txt.
Description
Self-interpretation asks language models to describe their own inner states, but hyperparameter sensitivity makes them unreliable. A 2026 paper led by Keenan Pepper, with eight co-authors, shows that training lightweight adapters on interpretability artifacts, with the language model fully frozen, yields dependable self-interpretation across tasks and model families. One scalar affine adapter with barely more parameters than model dimension is enough.
Trained adapters generate sparse autoencoder feature labels beating the training labels themselves, seventy percent versus fifty percent scoring at seventy-billion scale. They identify topics with ninety-four percent recall at one versus one percent for untrained baselines. They also decode bridge entities in multi-hop reasoning appearing in neither prompt nor response. The learned bias vector alone accounts for eighty-five percent of improvement. Simpler adapters generalize better than expressive alternatives, and self-interpretation gains outpace capability gains from seven billion to seventy-two billion parameters.
Catalog record
| CREATOR | Keenan Pepper, Alex McKenzie, Florin Pop, Stijn Servaes, Martin Leitgab, Mike Vaiana, Judd Rosenblatt, Michael S. A. Graziano, Diogo de Lucena |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-10352