Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers

Thomas Heap, Tim Lawson, Lucy Farnik, Laurence Aitchison · 2025

Read Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers by Thomas Heap, Tim Lawson, Lucy Farnik, Laurence Aitchison (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17727.txt.

Description

Thomas Heap, Tim Lawson, Lucy Farnik, and Laurence Aitchison sound a warning about tools used to read the minds of AI models. Their twenty twenty-five paper studies sparse autoencoders, which pull sparse, named traits from a transformer's inner signals. Scores for these tools are often taken as proof the traits are real.

The team ran the same tools on random transformers, whose weights were never trained. Across many model sizes and random schemes, the scores often matched those from trained models. That means high scores alone do not prove real traits were found. The team urges routine random baselines and sharper tests of how abstract a trait is.

Catalog record

CREATORThomas Heap, Tim Lawson, Lucy Farnik, Laurence Aitchison
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-17727

Browse the full library · About the Tower of Babel