CAGE: C urvature- A ware G radient E stimation For Accurate Quantization-Aware Training
Rush Tabesh, Mher Safaryan, Dan Alistarh · 2025
Read CAGE: C urvature- A ware G radient E stimation For Accurate Quantization-Aware Training by Rush Tabesh, Mher Safaryan, Dan Alistarh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18784.txt.
Description
Soroush Tabesh and his four co-authors close an accuracy gap in low-precision training. Neural networks are trained to run with rounded weights. The rounding still distorts learning. The standard trick passes gradients through as if rounding were invisible. That biases the updates. Their method is called CAGE.
CAGE corrects the gradient with local curvature data. It estimates how the rounding-induced loss increase bends the landscape. The method grows from a two-goal view of training. Accuracy is balanced against quantization limits. The authors prove convergence guarantees for smooth problems. They supply an efficient implementation using common optimizer statistics. In fine-tuning, accuracy loss halved versus the past best method. In pretraining, three-bit precision matched what rivals needed four bits to reach.
Catalog record
| CREATOR | Rush Tabesh, Mher Safaryan, Dan Alistarh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18784