LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs
Jung Hyun Lee, June Yong Yang, Jungwook Choi, Eunho Yang · 2026
Read LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs by Jung Hyun Lee, June Yong Yang, Jungwook Choi, Eunho Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29756.txt.
Introduction
Jung Hyun Lee and three colleagues fix a spot in low-bit quantization for long-form generation. Block-wise methods match precision on basic tasks yet fade on long answers and thought chains. The team traces the fade to two gaps: the output layer escapes tuning, and mean-squared-error training misaligns token odds. Their fix, LFQ, quantizes the final block by matching token odds through cross-entropy against the full-precision model. Across model families it lifts complex generation over block-wise rivals while holding parity on basics.
Catalog record
| CREATOR | Jung Hyun Lee, June Yong Yang, Jungwook Choi, Eunho Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50343 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.29756
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs Jung Hyun Lee, June Yong Yang, Jungwook Choi, Eunho Yang Abstract As large language models continue to scale, low-bit weight-only post-training quantization (PTQ) offers a practical solution to their memory-efficient deployment. Although block-wise PTQ is capable of matching the full-precision (FP) baseline on basic language modeling and understanding, its quality is degraded for generative tasks—especially at longer responses and extended chains of thought, which is critical in boosting task accuracy. We attribute this shortfall to two factors: (i) the omission of the unembedding layer (the LM head) in block-wise optimization and (ii) the reliance on the mean squared error (MSE) objective. Both factors cause the token probability distribution of the quantized model to misalign with that of the FP model, yielding notable accuracy drops on text generation benchmarks. To rectify the discrepancy, we introduce Logit-aware Final-block Quantization (LFQ) , a simple yet effective enhancement to block-wise PTQ that quantizes the final Transformer block by minimizing the cross-entropy between the logits of the FP model and those of its quantized counterpart. By aligning token probabilities at the logit level in the final block, LFQ consistently improves the accuracy of complex generation tasks over state-of-the-art block-wise PTQ across diverse model families, while maintaining parity with FP baselines on language modeling and understanding. 1 Introduction The evident success of large language models (LLMs) ( 10 ; 25 ; 28 ) is largely attributed to their ever-increasing number of parameters ( 12 ) . However, the proportionally increasing memory footprint of the model significantly impedes the cost-effective deployment of LLMs. Not only is a large model difficult to fit in commercial devices, but the serving cost of the model also increases sharply with the model size. To this end, quantization has been widely adopted to increase the inference efficiency of LLMs by employing lower precision data types. Recently, weight-only quantization ( 9 ; 19 ) has emerged as a particularly attractive methodology due to its high compression ratio and effective preservation of model quality. By quantizing the LLM weights into low-precision but retaining difficult-to-quantize activations in full precision (FP), memory pressure is effectively relieved while reducing the accuracy degradation. Low-bit weight-only quantization is obtained either via quantization-aware training (QAT) or post-training quantization (PTQ). Although 21 shows that QAT is capable of restoring the degraded accuracy even under sub-4-bit settings, the computational resource required to conduct QAT makes it prohibitively memory-intensive and time-consuming. On the other hand, layer-wise PTQ can be conducted with a relatively small amount of resources, but suffers from model quality degradation. Block-wise PTQ ( 14 ; 27 ; 5 ; 15 ; 4 ) strikes an effective balance between the two ends, achieving effective and efficient degradation recovery. By minimizing the mean squared error (MSE) between the outputs of an FP Transformer block and those of its quantized counterpart, cross-layer dependencies within each transformer block are accounted for, recovering the performance comparable to FP baselines on tasks such as language modeling (e.g. WikiText2 ( 22 ) ) and general natural language understanding (e.g. MMLU ( 11 ) ). However, relatively little attention has been shed on the degradation of generation quality caused by the low-bit quantization of LLMs. It is particularly alarming considering the increasing trend towards generating longer responses for increased task performance. Emerging large reasoning models ( 7 ; 1 ) scale inference-time compute to produce extended chains of thought, thereby achieving higher accuracy on complex multi-step reasoning tasks. As this trend toward generating more tokens continues in pursuit of increased accuracy, serving costs rise sharply, necessitating a strong demand for an efficient quantization method that can maintain the generation quality of FP baselines. In this work, we uncover that the standard block-wise PTQ approach—while effective at language modeling and understanding—suffers from the degradation of generation quality. The limitation is attributed to the fact that block-wise PTQ only preserves the quality of output activations of a transformer block, rather than preserving the fidelity of the next-token sampling distribution . Specifically, (i) existing block-wise PTQ methods completely ignore the unembedding layer (also known as the LM head), and (ii) rely on the MSE as optimization objective. Even when the MSE between the outputs of a quantized block and its FP counterpart is minimized, the actual probabilities assigned to plausible tokens can be perturbed, producing substantial shifts in distribution. Such misalignment is less observable on language understanding tasks, which does not involve autoregressive generation, but becomes pronounced in long-form generation as compounding probability distortions steer the generation trajectory away from the FP baseline. Motivated by the observations, we propose Logit-aware Final-block Quantization (LFQ) , which enables low-bit quantized LLMs to achieve performance close to FP baselines on text generation tasks such as IFEval ( 30 ) , GSM8K ( 6 ) , MATH500 ( 18 ) , and AIME ( 2 ) . Unlike standard block-wise PTQ, LFQ quantizes the final Transformer block by minimizing the cross-entropy loss between the logits of the FP model and those of its quantized counterpart. Specifically, all Transformer blocks from the first to the penultimate are quantized by minimizing the MSE between the outputs of the FP and quantized blocks, while the final block is optimized using cross-entropy at the logit level, aligning token probabilities with the FP model and thereby reproducing the token prediction probabilities of the FP model. Thanks to its simple design, LFQ can be seamlessly applied to existing block-wise approaches. Moreover, LFQ consistently improves the generation quality of block-wise PTQ methods, while maintaining performance comparable to FP baselines on language modeling and understanding tasks. Our contribution is threefold: • To the best of our knowledge, we are the first to show that the conventional block-wise PTQ objective—minimizing MSE at intermediate outputs—does not align with reproducing the token predictions of FP models, thereby inducing non-negligible accuracy gaps between FP baselines and their low-bit quantized counterparts on text generation tasks. • We propose Logit-aware Final-block Quantization (LFQ), which quantizes the final Transformer block by minimizing the cross-entropy between the logits of the FP model and its quantized counterpart, consistently improving the generation quality of low-bit quantized LLMs across existing block-wise PTQ methods. • We validate LFQ across diverse models—including instruction-tuned, reasoning, and Mixture of Experts (MoE) models—on text generation benchmarks such as IFEval, GSM8K, MATH500, and AIME. We further evaluate LFQ on WikiText2 and MMLU to ensure that it performs comparably to, and in some cases better than, existing block-wise PTQ techniques on language modeling and understanding tasks as well. 2 Problem Statement Block-wise PTQ progressively quantizes each Transformer block by minimizing the mean squared error (MSE), from the first to the final block. In this section, we focus our attention on the final block, which is distinctive from the other blocks as it is directly attached to the LM head that produces the token sampling distribution. Below, we provide a brief overview of notations and assumptions used for illustrative purposes throughout this paper. Let \bm{W}_{\text{FP}},\bm{W}_{q}\in\mathbb{R}^{c_{in}\times c_{out}} denote the full-precision (FP) final Transformer block and its quantized counterpart, and let \bm{X}\in\mathbb{R}^{L\times c_{in}} represent the input to the final block, where L is the sequence length. Let \mathcal{V} denote the vocabulary,