Embryology of a Language Model
George Wang, Garrett Baker, Andrew Gordon, Daniel Murfet · 2025
Read Embryology of a Language Model by George Wang, Garrett Baker, Andrew Gordon, Daniel Murfet (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00331.txt.
Description
George Wang, Garrett Baker, Andrew Gordon, and Daniel Murfet watched a language model grow in a 2025 paper. They treat training like the growth of an embryo, and ask how the model's inner parts form over time. Their tool comes from physics and tracks how a small change in one part affects the rest.
By mapping those links, they saw a clear body plan take shape as training went on. Known parts, such as a circuit that copies names, appeared where they were expected. A new part also showed up, one that seems to count space marks in text. The authors argue this view can find hidden parts, not just check ones already known.
Catalog record
| CREATOR | George Wang, Garrett Baker, Andrew Gordon, Daniel Murfet |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 192 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00331