Efficient Construction of Model Family through Progressive Training Using Model Expansion
Kazuki Yano, Sho Takase, Sosuke Kobayashi, Shun Kiyono, Jun Suzuki · 2025
Read Efficient Construction of Model Family through Progressive Training Using Model Expansion by Kazuki Yano, Sho Takase, Sosuke Kobayashi, Shun Kiyono, Jun Suzuki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00623.txt.
Description
Kazuki Yano and co-authors published a 2025 paper on building a family of language models for a quarter less compute. Vendors offer models in many sizes, from small to huge, to fit different budgets. Training each size from scratch repeats most of the work. The authors instead grow models: train a small one, expand it, and keep training, step by step up the size ladder.
Across a family from one to eight billion parameters, the grown models matched the quality of separately trained ones. Total computing cost fell by about twenty-five percent. The schedule of when to expand matters, and tuning the largest size's training rate added further gains. The method reframes model families as one long training run with growth spurts. For labs that ship several sizes, the savings are real money and energy. The paper adds evidence that big models need not start big.
Catalog record
| CREATOR | Kazuki Yano, Sho Takase, Sosuke Kobayashi, Shun Kiyono, Jun Suzuki |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00623