Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe
Gaofei Shen Martijn Bentum Tom Lentz Afra Alishahi Grzegorz Chrupała · 2026
Read Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe by Gaofei Shen Martijn Bentum Tom Lentz Afra Alishahi Grzegorz Chrupała (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00607.txt.
Description
Gaofei Shen and 4 co-authors reverse the usual direction of probing, the standard technique for asking what a language model knows inside. A decoding probe checks which features can be read out of a model's representations. That approach has two flaws: what different features add cannot be compared head-on, and links between features can distort the results. Their Encoding Probe instead rebuilds those states from readable features.
By rebuilding model states from feature sets covering sound, speech sounds, syntax, word choice, and speaker identity, the method measures how much each family contributes. They test it on text and speech transformer models. Effects tied to the speaker varied strongly with training goal and dataset, while syntax and word features each added their own share. The authors present the probe as a second lens on the model's inside, one that goes beyond asking what can merely be decoded.
Catalog record
| CREATOR | Gaofei Shen Martijn Bentum Tom Lentz Afra Alishahi Grzegorz Chrupała |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00607