Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe

Gaofei Shen Martijn Bentum Tom Lentz Afra Alishahi Grzegorz Chrupała · 2026

Read Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe by Gaofei Shen Martijn Bentum Tom Lentz Afra Alishahi Grzegorz Chrupała (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00607.txt.

Description

Gaofei Shen and 4 co-authors reverse the usual direction of probing, the standard technique for asking what a language model knows inside. A decoding probe checks which features can be read out of a model's representations. That approach has two flaws: what different features add cannot be compared head-on, and links between features can distort the results. Their Encoding Probe instead rebuilds those states from readable features.

By rebuilding model states from feature sets covering sound, speech sounds, syntax, word choice, and speaker identity, the method measures how much each family contributes. They test it on text and speech transformer models. Effects tied to the speaker varied strongly with training goal and dataset, while syntax and word features each added their own share. The authors present the probe as a second lens on the model's inside, one that goes beyond asking what can merely be decoded.

Catalog record

CREATORGaofei Shen Martijn Bentum Tom Lentz Afra Alishahi Grzegorz Chrupała
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00607

Browse the full library · About the Tower of Babel