The Dead Salmons of AI Interpretability

Maxime Méloux, Giada Dirupo, François Portet, Maxime Peyrard, [6pt] Université Grenoble Alpes, CNRS, Grenoble INP, LIG · 2025

Read The Dead Salmons of AI Interpretability by Maxime Méloux, Giada Dirupo, François Portet, Maxime Peyrard, [6pt] Université Grenoble Alpes, CNRS, Grenoble INP, LIG (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18792.txt.

Description

Maxime Méloux and three co-authors warn that AI interpretability has a dead salmon problem. Neuroscientists once found 'brain activity' in a dead fish because their statistics were sloppy; the authors argue explanation tools can similarly find meaningful patterns in random neural networks. Their paper, posted to arXiv in December 2025, examines feature attribution, probing, sparse autoencoders, and causal analyses that all risk this trap.

Their prescription is to treat explanations as statistical estimates. An interpretability method infers parameters of a model from computational traces, so its claims should face explicit alternative hypotheses with quantified uncertainty. They stress identifiability: some popular questions may not have unique answers at all. Placing interpretability inside standard statistical inference, they argue, is how a field full of plausible stories becomes a rigorous science.

Catalog record

CREATORMaxime Méloux, Giada Dirupo, François Portet, Maxime Peyrard, [6pt] Université Grenoble Alpes, CNRS, Grenoble INP, LIG
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18792

Browse the full library · About the Tower of Babel