When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

Michael S. Zhang, Rishi A. Ruia, Arnav Kewalram, Saathvik Dharmapuram, Utkarsh Sharma, Kevin Zhu · 2025

Read When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models by Michael S. Zhang, Rishi A. Ruia, Arnav Kewalram, Saathvik Dharmapuram, Utkarsh Sharma, Kevin Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18934.txt.

Introduction

Michael S. Zhang and five co-authors find that shrinking a language model's numbers can make it a better student. Continual learning, teaching a model task after task, suffers catastrophic forgetting, and models are usually quantized for deployment, not for learning. Their study, posted to arXiv in December 2025, crosses the two: full FP16 precision against 8-bit and 4-bit integers, with and without small replay buffers.

FP16 wins the first task, scoring 74.44 percent on language understanding, then falls behind: quantized models beat it by 8 to 15 percent on later tasks, and INT4 nearly doubles its code score, 40 against 20 percent. A replay buffer of just 0.1 percent lifts retention from 45 to 65 percent at every precision, with INT8 the sweet spot overall. The authors suspect quantization noise regularizes learning, a practical reason to prefer 8-bit for models that must keep learning.

Catalog record

CREATORMichael S. Zhang, Rishi A. Ruia, Arnav Kewalram, Saathvik Dharmapuram, Utkarsh Sharma, Kevin Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18934

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models Michael S. Zhang, Rishi A. Ruia, Arnav Kewalram, Saathvik Dharmapuram, Utkarsh Sharma, Kevin Zhu Abstract Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency. We systematically investigate the interplay between quantization precision (FP16, INT8, INT4) and replay buffer strategies in large language models, revealing unexpected dynamics. While FP16 achieves superior initial task performance (74.44% on NLU), we observe a striking inversion on subsequent tasks: quantized models outperform FP16 by 8-15% on final task forward accuracy, with INT4 achieving nearly double FP16’s performance on Code generation (40% vs 20%). Critically, even minimal replay buffers (0.1%) dramatically improve retention—increasing NLU retention after Math training from 45% to 65% across all precision levels—with INT8 consistently achieving the optimal balance between learning plasticity and knowledge retention. We hypothesize that quantization-induced noise acts as implicit regularization, preventing the overfitting to new task gradients that plagues high-precision models. These findings challenge the conventional wisdom that higher precision is always preferable, suggesting instead that INT8 quantization offers both computational efficiency and superior continual learning dynamics. Our results provide practical guidelines for deploying compressed models in continual learning scenarios: small replay buffers (1-2%) suffice for NLU tasks, while Math and Code benefit from moderate buffers (5-10%), with quantized models requiring less replay than FP16 to achieve comparable retention. Code is available at https://github.com/Festyve/LessIsMore. 1 Introduction Although large language models (LLMs) have achieved state-of-the-art performance across a range of natural language and reasoning tasks, their ability to retain knowledge over time remains a key limitation, particularly when models must be continually updated with new data. Scaling alone does not address this challenge, as repeated fine-tuning often causes older capabilities to deteriorate 9 ; 16 , a phenomenon known as catastrophic forgetting 18 ; 3 . In real-world deployments, where models are expected to adapt continuously, this liability presents a major obstacle. To solve this, researchers use replay-based methods. The practical deployment of these models has necessitated considerable research focus on quantization, a compression methodology that transforms model parameters into low-precision representations 13 ; 10 . Quantization offers substantial gains in efficiency, enabling large models to train and run on commodity hardware, yet it also introduces new risks. Replay-based methods provide one of the most effective tools to mitigate forgetting. By selectively reintroducing samples from prior tasks during fine-tuning, replay buffers help stabilize model performance 4 . However, replay itself introduces another constraint: larger buffers improve retention but add computational and storage cost. This trade-off has not been studied. This motivates our key research question: 1. How does the trade-off between quantization precision and replay buffer size shape catastrophic forgetting in foundational models? To answer these questions, we fine-tune LLaMA-3.1-8B across three quantization levels (FP16, 8-bit, and 4-bit) using Low-Ranking Adaptation 12 for efficient adaptation, and evaluate on sequential tasks from the LoRI benchmark spanning natural language understanding, mathematical reasoning, and code generation 26 . We vary replay buffer sizes (0%, 0.1%, 0.5%, 1%, 2%, 5%, 10%, 20%) of prior data to construct a quantization replay trade-off map, identifying where performance collapses and where minimal replay suffices to preserve accuracy. Our results show that the cost of forgetting under quantization is not uniform: at higher precision, minimal replay is sufficient, while under 4-bit quantization, buffer size becomes a decisive factor in preserving prior knowledge; while under 8-bit, quantization noise acts as a natural regularizer, enabling replay to be more effective even at modest buffer sizes; while under 16-bit, the absence of such noise makes the model more prone to overwriting prior knowledge despite replay, leading to steeper forgetting across tasks. From these findings, we provide empirical guidelines for balancing memory and accuracy in compressed continual learning and introduce a benchmark framework for future studies. 2 Methods and Experimental Setup Model and quantization conditions. We study a LLaMA-3.1-8B model in three precision levels: FP16, 8-bit, and 4-bit. All fine-tuning uses LoRA with rank r=8 , alpha a=16 and 0.0 dropout across precisions; weights are frozen and adapters are learned. For 8-bit we use standard inference-time weight quantization with LoRA adapters in higher precision; for 4-bit we use QLoRA-style NF4 quantization. The entire process is repeated independently for each precision. Following the LoRI benchmark protocol 26 , we train for exactly 1 epoch per dataset to ensure comparability with prior work. All experiments use a learning rate of 2e-4, training batch size of 8, and evaluation batch size of 32. Runs were executed on a single NVIDIA B200 (180 GB VRAM) with 28 vCPUs. Tasks and datasets. We partition continual tasks into (i) Natural Language Understanding (NLU): The model is trained on an aggregation of eight NLU datasets 11 , including BoolQ 7 , PIQA 2 , SocialIQA 23 , ARC-Challenge 7 , ARC-Easy 7 , OpenBookQA 20 , HellaSwag 24 , and Winogrande 22 . We evaluate accuracy on the individual test split for each dataset. (ii) Mathematical Reasoning (Math): The model is trained on the GSM8K 8 training split and evaluated on the GSM8K test split. (iii) Code Generation (Code): The model is trained on CodeAlpaca 4 and evaluated using pass@1 on HumanEval 5 . Continual learning with replay. Training proceeds in three stages, each followed by evaluation on all datasets (NLU, Math, Code): • Stage A (initial): Train on 100% NLU. • Stage B (second): Train on 100% GSM8K while interleaving replay of size B\in\{20,10,5,2,1,0.5,0.1,0\}\% for all NLU datasets respectively (uniform random sampling per dataset). • Stage C (third): Train on 100% Code while interleaving replay of size B\in\{20,10,5,2,1,0.5,0.1,0\}\% for all NLU and GSM8K datasets respectively (uniform random sampling per dataset). 3 Analysis Figure 1 : Forward Accuracy. Forward Accuracy. Current task performance across different quantization levels and replay buffer sizes (log scale). NLU performance is unaffected by replay buffers. Figure 2 : Backward Accuracy. Retention of previous task performance after training on subsequent tasks. Dashed lines indicate retention metrics. X-axis uses log scale to emphasize critical low-replay region. We analyze the effect of quantization and replay on continual learning performance, reporting both average forward accuracy (performance immediately after training on each task, Figure 1) and backward accuracy (retention of previous tasks, Figure 2). Baseline degradation. In isolation, quantization had only a minor effect on performance: 8-bit models were within 1–2% of FP16, and 4-bit models degraded by 3–5%. This confirms prior findings that quantization-aware training can preserve model accuracy at low bit-widths as demonstrated by 6 . Amplification under continual learning. When tasks were learned sequentially without replay, these modest differences became amplified. FP16 and 8-bit models retained much of their earlier-task accuracy, while 4-bit models exhibited sharper degradation. For example, under 0% replay, 4-bit models lost over 30% absolute accuracy on NLU (from 72.31% to 42.50% average), compared to 35% for FP16 (from 77.26% to 42.50%). Replay as a stabilizer. Increasing the replay buffer size mitigated this amplification. At 20% replay, the gap between 4-bit and FP16 shrank back to single digits (4-bit: 78.95% vs FP16: 77.91% average NLU). Thus, the critical finding is not that 4-bit quantization is intrinsically harmful, but that its interaction with replay size determines long-term retention. Interpretation. We posit that the

Read the complete text