Low-Rank Correction for Quantized LLMs

Meyer Scetbon, James Hensman · 2024

Read Low-Rank Correction for Quantized LLMs by Meyer Scetbon, James Hensman (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07902.txt.

Description

Meyer Scetbon and James Hensman repair shrunken language models in a 2024 paper. Quantization squeezes a trained model to fewer bits per number. Errors creep into both weights and activations. The pair add small full precision matrices to cancel those errors.

Their low rank add ons act on the unquantized activations. A joint optimization tunes the quantized weights and the corrections together. The test case is aggressive four bit compression of both weights and activations. Corrections sized at a tenth of the original cut the accuracy gap by over half. At thirty percent, the gap closed completely. Four recent models, including Llama and Mixtral versions, showed the effect. The method needs only a small calibration data set.

Catalog record

CREATORMeyer Scetbon, James Hensman
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-07902

Browse the full library · About the Tower of Babel