Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall
Qianli Wang Mingyang Wang Nils Feldhus Simon Ostermann Yuan Cao Hinrich Schütze Sebastian Möller Vera Schmitt · 2025
Read Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall by Qianli Wang Mingyang Wang Nils Feldhus Simon Ostermann Yuan Cao Hinrich Schütze Sebastian Möller Vera Schmitt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13963.txt.
Introduction
Qianli Wang and co-authors ask what shrinking a language model does to its memory of facts in a paper from 2025. Quantization stores model weights in fewer bits, which speeds the model and cuts its size. Its effects on many skills are known, but fact recall has drawn less study.
The team tested three common quantization methods at several bit widths. They used tasks on stored knowledge and on multi-step reasoning in the model's depths. Recall mostly fell after compression, and small models lost the most. Some low-bit models kept more than expected, which the authors probe. The work warns that small size can cost a model its facts.
Catalog record
| CREATOR | Qianli Wang Mingyang Wang Nils Feldhus Simon Ostermann Yuan Cao Hinrich Schütze Sebastian Möller Vera Schmitt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13963
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall Qianli Wang Mingyang Wang Nils Feldhus Simon Ostermann Yuan Cao Hinrich Schütze Sebastian Möller Vera Schmitt Abstract Quantization methods are widely used to accelerate inference and streamline the deployment of large language models (LLMs). Although quantization’s effects on various LLM capabilities have been extensively studied, one critical area remains underexplored: factual knowledge recall (FKR), the process by which LLMs access stored knowledge. To this end, we conduct comprehensive experiments using three common quantization techniques at distinct bit widths, in conjunction with interpretability-driven analyses on two tasks, knowledge memorization and latent multi-hop reasoning . We show that quantization typically results in information loss within LLMs, consequently diminishing their capacity for FKR. This effect is particularly amplified in smaller models within the same architectural families. However, models quantized at reduced bit precision do not consistently exhibit inferior performance and occasionally quantization may even enhance model FKR. We find that BitSandBytes demonstrates highest preservation of the original full-precision model’s FKR. Despite variability across models and methods, quantization causes modest performance degradation and remains an effective compression strategy. 1 Introduction The shift towards LLMs has created strong demand for efficient inference and accessible deployment. In response, numerous quantization techniques have been developed Dettmers et al. (2022) ; Frantar et al. (2023) ; Xiao et al. (2023) ; Lin et al. (2024) . By reducing the precision of a model’s parameters, quantization allows us to decrease model size while mostly preserving its performance Gray and Neuhoff (1998) . Although the effects of quantization in LLMs have been evaluated across various aspects, e.g., multilinguality, bias, fairness, and trustworthiness Marchisio et al. (2024) ; Gonçalves and Strubell (2023) ; Ramesh et al. (2023) ; Hong et al. (2024) ; Liu et al. (2024) ; Wang et al. (2026) , the degree , variability , and practical implications of its impact on LLMs’ ability to recall factual knowledge from pretrained memory remains underexplored and have yet to be comprehensively characterized. Figure 1: The effect of quantization on factual knowledge recall through knowledge memorization analysis and latent multi-hop reasoning analysis . To this end, we present a comprehensive study of quantization’s impact on factual knowledge recall in LLMs. Instead of solely observing the performance degradation, we perform interpretability-driven analyses on knowledge memorization and latent multi-hop reasoning tasks (§ 3.1 ), to scrutinize behavior across neuron , layer , and model levels and evaluate three LLMs across three widely adopted model quantization techniques at different bit widths (Figure 1 ). We reveal that quantization typically impairs FKR, which is particularly acute for smaller models within a given model family. LLMs quantized to a lower bit precision do not consistently underperform those with higher precision. In some cases, quantization can even paradoxically enhance factual recall. Our findings indicate that BitSandBytes is most effective at preserving the model’s FKR. We observed no significant FKR degradation from quantization that would compromise model compression effectiveness, though effects vary by model and technique. 2 Background and Related Work Quantization. Quantization techniques compress LLMs by converting model weights, activations, or the KV cache into lower-precision data types Zhu et al. (2024) . These techniques can be broadly categorized into two types: quantization-aware training (QAT) and post-training quantization (PTQ). QAT requires retraining to mitigate errors introduced by quantization, whereas PTQ facilitates the direct application of a quantized model during inference. In this paper, we primarily evaluate the impact of weight-only quantization (§ 3.3 ) on FKR (§ 4 ), which eliminates the need for retraining to address errors resulting from quantization. Impact of Quantization. Recent work has extensively examined the impact of quantization on various capabilities of LLMs. Marchisio et al. (2024) conduct a thorough analysis of quantized multilingual LLMs, focusing on performance degradation across languages. Gonçalves and Strubell (2023) ; Kirsten et al. (2025) explore the emergence of bias in the outputs generated by quantized models. Liu et al. (2024) find that in-context learning ability gradually declines in heavily quantized LLMs. Jin et al. (2024) observe that models with 4-bit quantization can still retain the alignment ability. Singh and Sajjad (2025) investigate the impact of quantization on model calibration. Wang et al. (2026) explore the impact of quantization on self-explanation quality and faithfulness. In our work, we explicitly explore how quantization affects FKR. 3 Experimental Setup We examine two representative methods (§ 3.1 ) to evaluate the impact of quantization on FKR. Specifically, we compare full-precision LLMs (§ 3.4 ) with LLMs quantized using different techniques and bit configurations (§ 3.3 ) across two datasets (§ 3.2 ). 3.1 Methods Knowledge Memorization Analysis. We investigate the memorization of a model’s factual knowledge, identifying the reasons behind potential factual forgetting Namburi et al. (2023) . By leveraging the theory of knowledge neurons, which suggests that specific neurons in LLMs are responsible for storing specific pieces of knowledge, we explore how quantization alters the storage and retrieval processes within these neurons. Latent Multi-hop Reasoning Analysis. We adapt the methodology from Yang et al. (2024) for inspecting latent multi-hop reasoning errors. Specifically, the analysis tests whether full-precision and quantized LLMs employ similar latent reasoning pathways and internal knowledge recall mechanisms to answer complex factual queries. 3.2 Datasets Our study employs two widely recognized datasets 1 1 1 Dataset examples are detailed in Appendix A . for evaluating FKR with selected interpretability methods (§ 3.1 ). LRE Hernandez et al. (2024) is a knowledge probing dataset consisting of knowledge triplets (s,r,o) , structured in a one-hop setting, where s , r , and o represents the subject, relation, and object, respectively. TwoHop-Fact Yang et al. (2024) is a dataset consisting of pairs of prompts: two-hop prompts ( \tau_{2H} ) for compositional queries, representing fact composition queries in the form ((e_{1},r_{1},e_{2}),(e_{2},r_{2},e_{3})) , where r_{1} and r_{2} are relations and e_{i} denotes an entity; and one-hop prompts ( \tau_{1H} ) for subqueries (Figure 4 ). e_{1} and e_{2} (in the second triplet) are subjects, while e_{2} (in the first triplet) and e_{3} objects. e_{2} serves as a bridge entity , linking the two triplet to form a coherent two-hop reasoning chain. 3.3 Quantization Techniques Building on the prior discussion (§ 2 ), we identify three commonly used PTQ techniques applied to the selected LLMs (§ 3.4 ) in our experiments: • GPTQ Frantar et al. (2023) uses a second-order, Hessian-based optimization to quantize weights post-training with minimal accuracy loss; • AWQ Lin et al. (2024) enhances weight quantization by handling activation outliers to preserve model accuracy at low bit-widths; • Integer quantization Dettmers et al. (2022) implemented by BitsAndBytes ( bib4 and bib8 ) enables fast and memory-efficient inference by using optimized low-bit kernels. 3.4 Models We evaluate Llama3-8B AI@Meta (2024) , Qwen2.5-7B , and Qwen2.5-14B Qwen et al. (2024) using two interpretability methods (§ 3.1 ). These models are selected due to the availability of quantized versions for each 2 2 2 Detailed information for each model, including links to their respective quantized versions, is provided in Table 1 . , ensuring the reproducibility of our results. Figure 2: Top: Distribution of high-contributing neurons across layers, showing the average number of top-300 neurons per layer for Qwen2.5-7B (