Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers
Thomas Heap, Tim Lawson, Lucy Farnik, Laurence Aitchison · 2025
Read Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers by Thomas Heap, Tim Lawson, Lucy Farnik, Laurence Aitchison (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17727.txt.
Description
Thomas Heap, Tim Lawson, Lucy Farnik, and Laurence Aitchison sound a warning about tools used to read the minds of AI models. Their twenty twenty-five paper studies sparse autoencoders, which pull sparse, named traits from a transformer's inner signals. Scores for these tools are often taken as proof the traits are real.
The team ran the same tools on random transformers, whose weights were never trained. Across many model sizes and random schemes, the scores often matched those from trained models. That means high scores alone do not prove real traits were found. The team urges routine random baselines and sharper tests of how abstract a trait is.
Catalog record
| CREATOR | Thomas Heap, Tim Lawson, Lucy Farnik, Laurence Aitchison |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17727