Low-Rank Correction for Quantized LLMs
Meyer Scetbon, James Hensman · 2024
Read Low-Rank Correction for Quantized LLMs by Meyer Scetbon, James Hensman (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07902.txt.
Description
Meyer Scetbon and James Hensman repair shrunken language models in a 2024 paper. Quantization squeezes a trained model to fewer bits per number. Errors creep into both weights and activations. The pair add small full precision matrices to cancel those errors.
Their low rank add ons act on the unquantized activations. A joint optimization tunes the quantized weights and the corrections together. The test case is aggressive four bit compression of both weights and activations. Corrections sized at a tenth of the original cut the accuracy gap by over half. At thirty percent, the gap closed completely. Four recent models, including Llama and Mixtral versions, showed the effect. The method needs only a small calibration data set.
Catalog record
| CREATOR | Meyer Scetbon, James Hensman |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07902