Entropy-Lens: The Information Signature of Transformer Computations
Riccardo AliDepartment of Computer ScienceUniversity of CambridgeCambridge, UK [email protected], Francesco CasoDIAGSapienza University of RomeRome, Italy [email protected], Christopher IrwinDISITUniversity of Eastern PiedmontAlessandria, Italy [email protected] Pietro LiòDepartment · 2025
Read Entropy-Lens: The Information Signature of Transformer Computations by Riccardo AliDepartment of Computer ScienceUniversity of CambridgeCambridge, UK [email protected], Francesco CasoDIAGSapienza University of RomeRome, Italy [email protected], Christopher IrwinDISITUniversity of Eastern PiedmontAlessandria, Italy [email protected] Pietro LiòDepartment (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16570.txt.
Description
Riccardo Ali and co-authors read a language model's mind by watching a single number per layer: entropy. Interpretability work usually peers at the model's inner vectors, not at what the model is about to say. Projected back into token space at each layer, though, the predictions have an entropy, a plain measure of how undecided the model still is. Their 2025 paper builds the Entropy-Lens on that signal.
Across model families and sizes, the entropy profile over depth proved revealing. Models visibly expand their options and then prune them, in rhythms that are specific to each family yet stable when depth is rescaled, and characteristic of the task and the answer's format. The expansion phase mattered more to final performance than the pruning. The authors offer the profile as an information signature: a cheap, comparable trace of how a transformer thinks, layer by layer.
Catalog record
| CREATOR | Riccardo AliDepartment of Computer ScienceUniversity of CambridgeCambridge, UK [email protected], Francesco CasoDIAGSapienza University of RomeRome, Italy [email protected], Christopher IrwinDISITUniversity of Eastern PiedmontAlessandria, Italy [email protected] Pietro LiòDepartment |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-16570