CAGE: C urvature- A ware G radient E stimation For Accurate Quantization-Aware Training

Rush Tabesh, Mher Safaryan, Dan Alistarh · 2025

Read CAGE: C urvature- A ware G radient E stimation For Accurate Quantization-Aware Training by Rush Tabesh, Mher Safaryan, Dan Alistarh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18784.txt.

Description

Soroush Tabesh and his four co-authors close an accuracy gap in low-precision training. Neural networks are trained to run with rounded weights. The rounding still distorts learning. The standard trick passes gradients through as if rounding were invisible. That biases the updates. Their method is called CAGE.

CAGE corrects the gradient with local curvature data. It estimates how the rounding-induced loss increase bends the landscape. The method grows from a two-goal view of training. Accuracy is balanced against quantization limits. The authors prove convergence guarantees for smooth problems. They supply an efficient implementation using common optimizer statistics. In fine-tuning, accuracy loss halved versus the past best method. In pretraining, three-bit precision matched what rivals needed four bits to reach.

Catalog record

CREATORRush Tabesh, Mher Safaryan, Dan Alistarh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-18784

Browse the full library · About the Tower of Babel