Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Models

Ahmad Dawar Hakimi, Ali Modarressi, Philipp Wicke, Hinrich Schütze · 2025

Read Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Models by Ahmad Dawar Hakimi, Ali Modarressi, Philipp Wicke, Hinrich Schütze (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03434.txt.

Description

Ahmad Dawar Hakimi and co-authors track how parts of a big word model grow into jobs as training runs in a 2025 study. The team watched one open model at many points in its long training run. Each inner part was sorted by role: broad help, name spotting, link reading, or fact recall.

Early on, the model leaned on broad parts, then the roles sharpened once right answers came more often. Some parts got reshaped for new roles after that. The link parts shifted jobs the most, while the deep stack layers held steadier. Place-based links clicked into place sooner than name-based ones.

Catalog record

CREATORAhmad Dawar Hakimi, Ali Modarressi, Philipp Wicke, Hinrich Schütze
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03434

Browse the full library · About the Tower of Babel