Efficient Construction of Model Family through Progressive Training Using Model Expansion

Kazuki Yano, Sho Takase, Sosuke Kobayashi, Shun Kiyono, Jun Suzuki · 2025

Read Efficient Construction of Model Family through Progressive Training Using Model Expansion by Kazuki Yano, Sho Takase, Sosuke Kobayashi, Shun Kiyono, Jun Suzuki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00623.txt.

Description

Kazuki Yano and co-authors published a 2025 paper on building a family of language models for a quarter less compute. Vendors offer models in many sizes, from small to huge, to fit different budgets. Training each size from scratch repeats most of the work. The authors instead grow models: train a small one, expand it, and keep training, step by step up the size ladder.

Across a family from one to eight billion parameters, the grown models matched the quality of separately trained ones. Total computing cost fell by about twenty-five percent. The schedule of when to expand matters, and tuning the largest size's training rate added further gains. The method reframes model families as one long training run with growth spurts. For labs that ship several sizes, the savings are real money and energy. The paper adds evidence that big models need not start big.

Catalog record

CREATORKazuki Yano, Sho Takase, Sosuke Kobayashi, Shun Kiyono, Jun Suzuki
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00623

Browse the full library · About the Tower of Babel