Performance of Small Language Model Pretraining on FABRIC: An Empirical Study
Praveen Rao · 2026
Read Performance of Small Language Model Pretraining on FABRIC: An Empirical Study by Praveen Rao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02632.txt.
Description
Praveen Rao measures how best to pretrain small language models on a free academic testbed with ordinary graphics processors, in a 2026 arXiv paper. Pretraining large models demands enormous compute, but smaller models suit limited data sets and follow known scaling laws. The testbed, FABRIC, offers commodity GPU clusters to academic users at no charge.
Rao compares data parallelism, parallelism inside each operator, workflow parallelism between operators, and their combinations. Clusters varied from uniform hardware to mixed hardware, and some spread GPUs across geographic distance. GPT-2 medium and large models were pretrained with two open-source systems. Execution plans that jointly optimized intra-operator and workflow parallelism performed best when GPUs were geographically distributed, especially with latencies in the tens of milliseconds. The paper closes with a systematic approach for choosing a pretraining technique that cuts both time and GPU count.
Catalog record
| CREATOR | Praveen Rao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02632