The Uneven Impact of Post-Training Quantization in Machine Translation

Benjamin Marie, Atsushi Fujita · 2025

Read The Uneven Impact of Post-Training Quantization in Machine Translation by Benjamin Marie, Atsushi Fujita (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20893.txt.

Introduction

Benjamin Marie and Atsushi Fujita test how shrinking large language models affects translation in a 2025 study. Shrinking, called quantization, lets big models run on small hardware. Its effects across many languages had not been mapped at scale.

The pair tested five models across fifty-five languages and four shrinking methods. Four-bit shrinking kept quality for rich-resource languages and large models. Two-bit shrinking hurt poor-resource and varied languages most. One file format gave the most steady results even at two bits. Matching the tuning language to the target language helped mainly in low-bit cases. The study offers clear guidance for fair multilingual deployment.

Catalog record

CREATORBenjamin Marie, Atsushi Fujita
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20893

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Uneven Impact of Post-Training Quantization in Machine Translation Benjamin Marie, Atsushi Fujita Abstract Quantization is essential for deploying large language models (LLMs) on resource-constrained hardware, but its implications for multilingual tasks remain underexplored. We conduct the first large-scale evaluation of post-training quantization (PTQ) on machine translation across 55 languages using five LLMs ranging from 1.7B to 70B parameters. Our analysis reveals that while 4-bit quantization often preserves translation quality for high-resource languages and large models, significant degradation occurs for low-resource and typologically diverse languages, particularly in 2-bit settings. We compare four quantization techniques (AWQ, BitsAndBytes, GGUF, and AutoRound), showing that algorithm choice and model size jointly determine robustness. GGUF variants provide the most consistent performance, even at 2-bit precision. Additionally, we quantify the interactions between quantization, decoding hyperparameters, and calibration languages, finding that language-matched calibration offers benefits primarily in low-bit scenarios. Our findings offer actionable insights for deploying multilingual LLMs for machine translation under quantization constraints, especially in low-resource settings. 1 Introduction † † This work was partly conducted by the first author during a research visit to NICT. Quantization is now a key technique for scaling large language models (LLMs) to real-world deployments, enabling substantial reductions in memory usage and computational overhead, particularly on consumer-grade hardware. However, most existing research on quantization focuses exclusively on English language models and tasks, overlooking the linguistic diversity that multilingual LLMs are more and more often trained to support Gemma Team et al. 2025 ; Yang et al. 2025 . In practice, multilingual applications, especially machine translation (MT), are often deployed in low-resource settings, where both language data and computing capacity are limited. Ensuring that LLMs remain robust and accurate under quantization is thus essential for inclusive and globally accessible AI. Despite its importance, the impact of quantization on MT remains underexplored. Prior work Ogueji et al. 2022 ; Marchisio et al. 2024 suggests that compression techniques such as quantization and sparsity can disproportionately affect long-tail linguistic features, often corresponding to underrepresented languages. However, a comprehensive evaluation spanning languages, model scales, and quantization methods has yet to be conducted. In this study, we address this gap by systematically evaluating a suite of LLMs on MT tasks under various quantization settings. We consider five models ranging from 1.5B to 70B parameters, apply multiple quantization algorithms, including low-bit formats such as 2-bit and 4-bit, and assess translation performance across 55 language pairs, covering both low- and high-resource languages. By focusing on the intersection of MT, multilinguality, and quantization, our work provides the first detailed analysis of how quantized LLMs perform across diverse languages and quantization strategies. Our key findings are as follows: • Quantization retains strong performance in MT, even at low bitwidths, for the larger models, and most language pairs. Smaller models (<10B parameters) show clear signs of degradation in MT, especially for low-resource languages. • Qwen3 and Llama 3.1 models demonstrate some ability to translate languages outside their official training data. However, quantization substantially degrades performance for these lower-resource languages, especially Indic languages, and in some cases, eliminates their ability to translate them into English. • Quantization method matters: A calibration-free method like BitsAndBytes (bnb) performs well for smaller models in MT but shows limitations at larger scales. Surprisingly, simple formats like GGUF Q2_K yield strong performance, even for sub-10B parameter models. • We dissect decoding temperature, nucleus sampling, and calibration-set language, showing that (i) sampling temperature dominates top- p ; (ii) calibration language helps only at extreme (2-bit) quantization; and (iii) these factors interact additively with weight precision, yielding simple heuristics for real-world MT deployment on commodity hardware. 2 Background: Post-Training Quantization Quantization is a technique for reducing the computational and memory footprint of neural networks by representing weights and optionally activations in lower-bit formats. This can be done during training, known as quantization-aware training (QAT), or applied post hoc to a pre-trained model, known as post-training quantization (PTQ). For LLMs, training directly at reduced precision, especially below 16-bit, presents significant optimization and engineering challenges. Consequently, PTQ is often preferred in practice, as it allows models to be trained at high precision and then quantized in a single pass without further gradient updates. In this work, we focus on weight-only PTQ due to its simplicity and scalability. Only the weight matrices are quantized, while activations and the KV cache remain in higher precision. A standard symmetric uniform quantization scheme maps a real-valued weight tensor w to a quantized integer tensor 1 1 1 INT data types are the most used for quantization as they are optimized for computation with modern GPUs. Other data types used for quantization can be FP8 and FP4 variants. w_{q} via w_{q}=\mathrm{round}\left(\frac{w}{s}\right),\quad\hat{w}=s\cdot w_{q} where s is the scale factor that determines the resolution of quantization. A common choice for s in symmetric quantization is s=\frac{\max(|w|)}{2^{N-1}-1} where N is the bitwidth of quantization (e.g., 8 or 4), and the denominator corresponds to the maximum representable integer value in the signed range. The rounding operation ensures that values are mapped to the nearest quantized level, and the multiplication during dequantization approximately reconstructs the original value. In practice, w_{q} is clamped to the integer range [-2^{N-1}+1,2^{N-1}-1] so the negative and positive halves are perfectly symmetric, although many quantization implementations handle this implicitly. Using a single global scale factor for an entire weight tensor (per-tensor quantization) is computationally efficient. Still, it may introduce significant errors when the distribution of weights varies widely across channels. 2 2 2 A channel is a chosen tensor axis (e.g., each row/column of a weight matrix or an attention head) along which distinct quantization parameters (scale/zero-point) are applied in per-channel schemes. As a common refinement to reduce quantization errors in low-bit settings (e.g., 4-bit or lower), group-wise quantization is often employed, in which input channels are divided into fixed-size groups (e.g., 128), and each group is assigned its own scale. These more granular approaches improve approximation quality while maintaining the benefits of reduced precision. Although uniform quantization is effective at moderate bitwidths, it often leads to significant degradation at low precision. Previous works, such as GPTQ Frantar et al. 2023 and AWQ Lin et al. 2024 , incorporate quantization-aware error minimization during the post-training process, leading to substantial improvements in accuracy for quantized LLMs. These methods extend basic weight-only PTQ by optimizing the quantization step to preserve downstream performance more effectively. 3 Related Work A growing body of research on compression techniques, and particularly quantization, explores multilingual settings. However, most studies exhibit one of two key limitations: they lack core multilingual task evaluations or fall back on English-centric benchmarks. Paglieri et al. 2024 , for instance, introduce multilingual calibration datasets but limit their analysis to English, leaving multilingual generalization untested. Kharazmi et al. 2023 demonstrate that compression-induced degradation in LSTMs is significantly more severe in multilingual contexts compared to monolingual scenarios, suggesting that language diversity intensifies performance losses. MT is a critical metric of multilingual capability, yet it is often omitted from compression research. When translation is evaluated, results are mixed: Diddee et al. 2022 show that effectiveness of knowledge distillation varies by language, influenced heavily by teacher model confidence and the use of synthetic data. Multilingual corpora exemplify the long-tail problem in NLP, where low-resource

Read the complete text