Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate
A. Bochkov · 2025
Read Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate by A. Bochkov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07129.txt.
Description
Bochkov studies how to grow a transformer in stages without reopening finished parts in this 2025 paper. The token interface stays fixed, old blocks stay frozen, and only the newest blocks train, keeping the active parameter count roughly constant as depth grows.
In a nine layer test on a frozen text substrate, the grown model used 105 million trainable parameters against 180 million for a matched single piece baseline. An extreme version froze even the token codes, each token a sixteen step binary tag. Growth still worked: a sixteen layer model trained over that interface on about sixty-nine billion tokens reached a notable score on a knowledge benchmark after a light adapter stage. The author frames the result as proof of viability, with a clear cost in final perplexity.
Catalog record
| CREATOR | A. Bochkov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07129