The Dead Salmons of AI Interpretability
Maxime Méloux, Giada Dirupo, François Portet, Maxime Peyrard, [6pt] Université Grenoble Alpes, CNRS, Grenoble INP, LIG · 2025
Read The Dead Salmons of AI Interpretability by Maxime Méloux, Giada Dirupo, François Portet, Maxime Peyrard, [6pt] Université Grenoble Alpes, CNRS, Grenoble INP, LIG (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18792.txt.
Description
Maxime Méloux and three co-authors warn that AI interpretability has a dead salmon problem. Neuroscientists once found 'brain activity' in a dead fish because their statistics were sloppy; the authors argue explanation tools can similarly find meaningful patterns in random neural networks. Their paper, posted to arXiv in December 2025, examines feature attribution, probing, sparse autoencoders, and causal analyses that all risk this trap.
Their prescription is to treat explanations as statistical estimates. An interpretability method infers parameters of a model from computational traces, so its claims should face explicit alternative hypotheses with quantified uncertainty. They stress identifiability: some popular questions may not have unique answers at all. Placing interpretability inside standard statistical inference, they argue, is how a field full of plausible stories becomes a rigorous science.
Catalog record
| CREATOR | Maxime Méloux, Giada Dirupo, François Portet, Maxime Peyrard, [6pt] Université Grenoble Alpes, CNRS, Grenoble INP, LIG |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-18792