TimelyFreeze : Adaptive Parameter Freezing Mechanism for Pipeline Parallelism

Seonghye Cho, Jaemin Han, Hyunjin Kim, Euisoo Jung, Jae-Gil Lee · 2026

Read TimelyFreeze : Adaptive Parameter Freezing Mechanism for Pipeline Parallelism by Seonghye Cho, Jaemin Han, Hyunjin Kim, Euisoo Jung, Jae-Gil Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05754.txt.

Description

Training a model too large for one device splits it across a pipeline, and idle gaps called bubbles waste much of the hardware time. Seonghye Cho, Jaemin Han, and 3 co-authors propose TimelyFreeze, which freezes some parameters adaptively to skip backward computation and speed training. Earlier freezing methods often froze too much and cost accuracy.

TimelyFreeze models the pipeline schedule as a directed acyclic graph and solves a linear program for freeze ratios. The ratios minimize batch execution time while respecting accuracy constraints. On a large language model test, training throughput rose substantially, up to forty percent in the best case, with comparable accuracy. The authors report that convergence held up and that the method generalizes across pipeline parallel settings. The work targets teams that train frontier scale models on clusters. It shows scheduling mathematics, not just hardware, can reclaim idle pipeline time.

Catalog record

CREATORSeonghye Cho, Jaemin Han, Hyunjin Kim, Euisoo Jung, Jae-Gil Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05754

Browse the full library · About the Tower of Babel