Entropy-Lens: The Information Signature of Transformer Computations

Riccardo AliDepartment of Computer ScienceUniversity of CambridgeCambridge, UK [email protected], Francesco CasoDIAGSapienza University of RomeRome, Italy [email protected], Christopher IrwinDISITUniversity of Eastern PiedmontAlessandria, Italy [email protected] Pietro LiòDepartment · 2025

Read Entropy-Lens: The Information Signature of Transformer Computations by Riccardo AliDepartment of Computer ScienceUniversity of CambridgeCambridge, UK [email protected], Francesco CasoDIAGSapienza University of RomeRome, Italy [email protected], Christopher IrwinDISITUniversity of Eastern PiedmontAlessandria, Italy [email protected] Pietro LiòDepartment (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16570.txt.

Description

Riccardo Ali and co-authors read a language model's mind by watching a single number per layer: entropy. Interpretability work usually peers at the model's inner vectors, not at what the model is about to say. Projected back into token space at each layer, though, the predictions have an entropy, a plain measure of how undecided the model still is. Their 2025 paper builds the Entropy-Lens on that signal.

Across model families and sizes, the entropy profile over depth proved revealing. Models visibly expand their options and then prune them, in rhythms that are specific to each family yet stable when depth is rescaled, and characteristic of the task and the answer's format. The expansion phase mattered more to final performance than the pruning. The authors offer the profile as an information signature: a cheap, comparable trace of how a transformer thinks, layer by layer.

Catalog record

CREATORRiccardo AliDepartment of Computer ScienceUniversity of CambridgeCambridge, UK [email protected], Francesco CasoDIAGSapienza University of RomeRome, Italy [email protected], Christopher IrwinDISITUniversity of Eastern PiedmontAlessandria, Italy [email protected] Pietro LiòDepartment
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-16570

Browse the full library · About the Tower of Babel