Mapping the Schedule × Bit-Width Boundary in Sub-100M Quantisation-Aware Training
Christian Thomassen ([email protected]) — Dwarf A/S, Copenhagen, Denmark · 2026
Read Mapping the Schedule × Bit-Width Boundary in Sub-100M Quantisation-Aware Training by Christian Thomassen ([email protected]) — Dwarf A/S, Copenhagen, Denmark (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25966.txt.
Introduction
Thomassen asks whether the best learning-rate schedule for quantisation-aware training changes with bit-width. That training anticipates low-precision arithmetic in small decoder models. A 720-run grid finds one answer everywhere: a 33 percent final slowdown wins at every bit-width and size. Follow-up runs totalling 625 more trials leave that null result intact. The result holds across optimiser, schedule shape, training length and size range. The six-bit penalty follows a log-linear law that predicts five held-out sizes well.
Catalog record
| CREATOR | Christian Thomassen ([email protected]) — Dwarf A/S, Copenhagen, Denmark |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50747 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25966
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mapping the Schedule × Bit-Width Boundary in Sub-100M Quantisation-Aware Training Christian Thomassen ([email protected]) — Dwarf A/S, Copenhagen, Denmark Abstract Abstract We test whether the optimal learning-rate schedule depends on bit-width during from-initialisation quantisation-aware training (QAT) for sub-100M decoder language models. A 720-run factorial grid (Phase 2) over bit-width × warmdown fraction × LR magnitude × model size × seed (FP16/INT8/INT6, 15M–100M, 5 seeds) finds the optimal warmdown is 33% at every (bit-width, size) cell. The primary hypothesis — that INT6 QAT requires a different schedule than higher-precision training — is falsified at FP16/INT8/INT6. A 625-run follow-up (Phase 5) probes the null along five axes: optimiser (AdamW), schedule shape (cosine), training length (up to 9× more iterations), an extended size sweep (5M–350M), and an INT4 sweep from 3M to 100M. The null is robust under all three setup changes. The INT6 penalty follows a log-linear scaling law whose fit on Phase 2 predicts the five held-out Phase 5 sizes (5M, 8M, 175M, 250M, 350M) within their 95% prediction intervals (5/5). For INT4 the picture is sharper than the higher precisions: at 50M and 100M, wd33 is decisively optimal (paired z ≈ 12–15, 10/10 seeds); below 50M, across the six tested sizes from 3M to 30M, no individual size shows a statistically significant schedule preference and the per-size mean penalty oscillates within seed-level noise. The boundary is therefore a transition between a noise-dominated regime below 50M and a decisive wd33 regime at and above 50M, not a clean wd10 region. A weight-to-grid-distance probe falsifies the simplest mechanism for the FP16/INT8/INT6 null result (rapid grid-snapping): pre-warmdown, INT6-QAT weights sit at essentially the same distance from the INT6 grid as FP16 weights (ratio ≈ 1.04). Practical recommendation: at sub-100M scale, tune the LR schedule once at FP16 and apply unchanged to INT8/INT6 QAT; for INT4 at 50M+ use wd33; for INT4 below 50M the schedule choice is in the noise. 1. Introduction Small language models deployed at the edge benefit substantially from low-bit weight quantisation. Post-training quantisation (PTQ) methods such as GPTQ (Frantar et al. 2023) and AWQ (Lin et al. 2024) dominate the billion-parameter regime, but quantisation-aware training (QAT) remains the method of choice when target precisions drop below 8 bits and when the model is small enough that PTQ leaves accuracy on the table (Nielsen and Schneider-Kamp 2024; Chen et al. 2025). Recent compute-optimal QAT analyses (Dremov et al. 2025) have begun to map the interactions between bit-width, model size, training compute, and optimiser hyperparameters at the 86M–2.2B scale. A natural question follows: do those interactions persist below 100M, where edge-deployment economics are most acute and where sub-bit work (Nielsen and Schneider-Kamp 2024; Nielsen et al. 2025) has already shown that QAT recipes can differ substantially from large-model norms? This paper isolates one such interaction — between learning-rate schedule and bit-width — and tests it under tightly controlled conditions across multiple axes. The intuition under test is widely repeated in practitioner communities: aggressive weight quantisation destabilises late-stage training and therefore benefits from a longer warmdown phase than full-precision training. If true, QAT pipelines should treat the LR schedule as a per-precision hyperparameter. If false, a single tuned schedule transfers across bit-widths. We test the hypothesis with a 720-cell factorial grid (Phase 2; FP16/INT8/INT6 × 4 warmdowns × 3 LR magnitudes × 4 sizes × 5 seeds, 9000 iterations per cell). At every size and bit-width tested, the optimal warmdown fraction is 33%. The bit-width curves are statistically indistinguishable in shape. The primary hypothesis is falsified. A naive reading of this result would stop here: “no interaction, use the same schedule.” We instead pursue several follow-up questions that probe whether the null result is robust to setup choices, and where (if anywhere) it eventually breaks . Phase 5 (625 cells, ~1320 GPU-hours) consists of seven ablations: • D1 — Optimiser. Replicate the 240-cell Phase 2 grid at lr1x using AdamW instead of Muon. • D2 — Training length. Re-run the 30M × {4 warmdowns} × {3 bits} cells at 27k and 81k iterations (vs 9k in Phase 2). • D3 — Schedule shape. Re-run the wd33 cells at every size and bit-width using a cosine warmdown instead of linear. • D4 — Size extension. Run the lr1x/wd33 cells at five new sizes outside the Phase 2 range (5M, 8M, 175M, 250M, 350M), and use them as a held-out validation of the log-linear scaling law fit on Phase 2. • D5 — Aggressive precision. Run an INT4 grid across the 4 warmdowns × 4 Phase 2 sizes × 5 seeds. • D6 — INT4 below 15M. Extend the INT4 grid to four smaller sizes (3M, 5M, 8M, 10M) × 4 warmdowns × 5 seeds, to characterise where (if anywhere) the schedule × INT4 interaction sharpens as model size shrinks. • M2 — Mechanism probe. Re-run 30M wd33 lr1x cells with high-frequency snapshots of weight-to-INT6-grid distance, to test whether the bit-width-agnosticism observed in Phase 2 is mediated by rapid grid-snapping under aggressive QAT. The combined picture from Phase 2 + Phase 5 changes the headline: 1. The null result is robust across optimiser (Muon, AdamW), schedule shape (linear, cosine), training length (9k–81k iterations), and model size (15M–100M) for FP16, INT8, and INT6. 2. The INT6 penalty scales log-linearly in model size , and the fit on Phase 2 predicts held-out sizes (5M to 350M) within 95% CI — a 70× extrapolation that holds (Figure 2). 3. The INT4 picture has two regimes, not a clean wd10 region. At 50M and above, wd33 is decisively optimal for INT4 (paired z ≈ 12–15, 10/10 seeds). Below 50M, across six tested sizes from 3M to 30M, no individual size has a statistically significant schedule preference; the per-size mean penalty oscillates within seed-level noise (Figure 3). The transition between these regimes is sharp and sits between 30M and 50M. The INT4 quality penalty itself is an order of magnitude larger than INT6 (~15–40 mBPB vs ~3 mBPB) and drops most steeply across the same 30M → 50M step. 4. The INT6 penalty grows with training length at fixed size and schedule: +3.0 mBPB at 9k iterations grows to +6.5 mBPB at 81k iterations at 30M wd33 (Figure 5). The null result on schedule × bit-width persists in absolute terms — wd33 is still optimal at 81k iters for every bit-width — but the absolute cost of switching to INT6 is not iteration-invariant. 5. The mechanism is not grid-snapping (Figure 6). Weight-to-INT6-grid distance pre-warmdown is essentially equal for FP16, INT8, and INT6 trained models (within 4%). The bit-width-agnosticism of the schedule cannot be explained by INT6-QAT weights “having already converged” to the grid before warmdown begins. The actual mechanism remains an open question; one obvious candidate is now ruled out. Contributions. • A 720-cell factorial study (Phase 2) showing optimal warmdown is bit-width-agnostic across FP16/INT8/INT6 at every size from 15M to 100M. • A 625-cell follow-up study (Phase 5) demonstrating the result is robust to optimiser, schedule shape, and 9× training length; and validating a log-linear scaling law for the INT6 penalty on five held-out sizes. • A precision-boundary characterisation: INT4 has a sharp transition between a noise-dominated schedule-preference regime below 50M and a decisive wd33 regime at 50M and