Revisiting Parameter Server in LLM Post-Training

Xinyi Wan, Penghui Qi, Guangxing Huang, Chaoyi Ruan, Min Lin & Jialin Li · 2026

Read Revisiting Parameter Server in LLM Post-Training by Xinyi Wan, Penghui Qi, Guangxing Huang, Chaoyi Ruan, Min Lin & Jialin Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19362.txt.

Description

Xinyi Wan and five co-authors bring back the parameter server for a new era of model training. Modern data parallel training favors collective communication, where all workers sync at once. That works when workloads are balanced, but language model post training has wild variance in sequence lengths.

Under imbalance, synced steps make fast devices wait on slow ones, wasting the fleet. The team revisits the older parameter server design, where workers report to a server on their own clock. For post training's lumpy loads, the old design may beat the new default.

Catalog record

CREATORXinyi Wan, Penghui Qi, Guangxing Huang, Chaoyi Ruan, Min Lin & Jialin Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19362

Browse the full library · About the Tower of Babel