Embryology of a Language Model

George Wang, Garrett Baker, Andrew Gordon, Daniel Murfet · 2025

Read Embryology of a Language Model by George Wang, Garrett Baker, Andrew Gordon, Daniel Murfet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00331.txt.

Description

George Wang, Garrett Baker, Andrew Gordon, and Daniel Murfet watched a language model grow in a 2025 paper. They treat training like the growth of an embryo, and ask how the model's inner parts form over time. Their tool comes from physics and tracks how a small change in one part affects the rest.

By mapping those links, they saw a clear body plan take shape as training went on. Known parts, such as a circuit that copies names, appeared where they were expected. A new part also showed up, one that seems to count space marks in text. The authors argue this view can find hidden parts, not just check ones already known.

Catalog record

CREATORGeorge Wang, Garrett Baker, Andrew Gordon, Daniel Murfet
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE192 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00331

Browse the full library · About the Tower of Babel