Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
Xinyu Lian, Walid Krichene, Beichen Huang, Masahiro Tanaka, Olatunji Ruwase, Li Zhang, Minjia Zhang · 2026
Read Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models by Xinyu Lian, Walid Krichene, Beichen Huang, Masahiro Tanaka, Olatunji Ruwase, Li Zhang, Minjia Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25519.txt.
Introduction
Shrinking a reasoning model to fewer bits per weight is supposed to make it cheaper to run, but the savings can leak away through longer answers. Xinyu Lian and colleagues document that hidden cost in 2026 across maths, code, science questions, and tool use tasks. Low bit quantised models often kept their accuracy while quietly writing much longer chains of thought. The team defines a token inflation ratio comparing reasoning length against the full precision original to make the effect measurable.
Closer reading of the traces found more intermediate steps and more repetition of the same ideas. Those extra tokens translate into real serving slowdowns despite faster per token decoding. Prompting tricks and sampling changes gave inconsistent relief. Quantisation aware training looked more promising, trimming both accuracy loss and inflation, and the authors urge reporting reasoning length beside accuracy in future evaluations.
Catalog record
| CREATOR | Xinyu Lian, Walid Krichene, Beichen Huang, Masahiro Tanaka, Olatunji Ruwase, Li Zhang, Minjia Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.25519
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models Xinyu Lian, Walid Krichene, Beichen Huang, Masahiro Tanaka, Olatunji Ruwase, Li Zhang, Minjia Zhang Abstract Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency. We show that low-bit post-training quantization can introduce a hidden test-time compute cost: quantized reasoning models often generate longer chains of thought even when they still answer correctly. Across mathematical reasoning, code generation, scientific question answering, and agentic tool-use benchmarks, we find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup. To measure this effect, we introduce the CoT Token Inflation Ratio, which compares reasoning length between quantized and full-precision models averaged across all evaluation benchmarks. We further show that token inflation is accompanied by behavioral changes in the reasoning trace, including more intermediate steps and greater semantic repetition. These changes translate into measurable end-to-end real-world serving penalties. Finally, we evaluate mitigation strategies and find that prompting and decoding-time sampling offer inconsistent accuracy-length trade-offs, while quantization-aware training shows more promise in reducing both accuracy degradation and token inflation. Our results suggest that reasoning-token usage should be reported alongside accuracy when evaluating quantized reasoning models. 1 Introduction Recent large language models (LLMs) trained for step-by-step reasoning have achieved strong performance on mathematical reasoning, code generation, and scientific question answering ( 5 ; 20 ) . A key driver of this progress is the ability to spend more test-time computation: instead of producing a short answer directly, reasoning models generate long chains of thought (CoT) that decompose the problem, explore intermediate steps, verify partial results, and sometimes revise earlier conclusions ( 13 ; 5 ; 20 ; 10 ) . This behavior improves accuracy on difficult tasks, but it also makes inference substantially more expensive. For reasoning models, deployment cost is no longer determined only by model size and per-token latency. It also depends on how many reasoning tokens the model chooses to generate. Quantization is one of the most widely used techniques for reducing the inference cost of LLMs. Post-training quantization (PTQ) methods such as GPTQ 7 , AWQ 14 , SmoothQuant 25 , ParoQuant 12 reduce model memory footprint and accelerate decoding with negligible training, and they have shown strong accuracy-efficiency trade-offs on LLM benchmarks. These results have made low-bit quantization a practical default for serving large models. However, most quantization evaluations measure final-answer accuracy and per-token efficiency. This evaluation protocol is incomplete for reasoning models, where the number of generated reasoning tokens is a major component of test-time compute. In this paper, we show that low-bit post-training quantization can introduce a hidden cost for reasoning models: it can make models reason longer. Across mathematical reasoning, code generation, scientific question answering, and agentic tool-use benchmarks, we find that quantized models often preserve final-answer accuracy while generating longer chains of thought than their full-precision counterparts. This effect can offset, and in some cases outweigh, the per-token speedup expected from quantization. In other words, a quantized reasoning model can be faster per decoding step but slower end-to-end because it takes more tokens to reach the answer. We call this phenomenon reasoning-token inflation and introduce the CoT Token Inflation Ratio (CTIR) to measure it. CTIR measures the relative change in reasoning-token count between a quantized model and its full-precision reference, averaged across all evaluation benchmarks. Our analysis further shows that token inflation is not merely harmless verbosity. Quantized models often produce more intermediate reasoning steps and more semantically repetitive traces, revisiting or re-verifying prior reasoning without improving correctness. These behavior changes translate into measurable end-to-end serving cost penalties. We also examine preliminary mitigation strategies, finding that prompting and decoding-time controls provide inconsistent accuracy-length trade-offs, while quantization-aware training (QAT) is a more promising direction for reducing both accuracy degradation and reasoning-token inflation. In summary, this paper makes three contributions: • We study PTQ on SoTA reasoning models ranging from 4B to 30B parameters (dense and MoE) and quantitatively identify reasoning-token inflation as a hidden cost of low-bit post-training quantization. Across reasoning, code, and agentic tool-use benchmarks, quantized models can preserve final-answer accuracy while generating longer chains of thought, offsetting the expected per-token efficiency gains (§ 4 ). • We introduce the CoT Token Inflation Ratio (CTIR), a metric for measuring quantization-induced increases in reasoning length relative to a full-precision reference. CTIR exposes an evaluation gap in standard quantization studies, which typically report accuracy and per-token latency but omit the amount of reasoning generated at inference time (§ 5 ). • We analyze the behavioral consequences of token inflation. Quantized models often produce more intermediate reasoning steps and more semantically repetitive traces, and this increased reasoning length translates into higher end-to-end serving latency and cost (§ 6 ). We further evaluate mitigation strategies, finding that prompting and decoding-time sampling offer inconsistent accuracy-length trade-offs, while quantization-aware training is a promising direction (§ 7 ). 2 Background and Related Work 2.1 Quantization Methods for LLM Deployment Quantization for LLM deployment. Post-training quantization (PTQ) and quantization-aware training (QAT) are two main paradigms for reducing LLM inference cost. PTQ converts a pretrained model to low precision with little or no additional training ( 2 ; 7 ; 14 ; 4 ; 12 ) , while QAT simulates low-precision arithmetic during training or fine-tuning with quantize–dequantize operators ( 6 ; 16 ; 3 ) . Because training or fine-tuning large LLMs is expensive, most prior work has focused on PTQ methods that avoid additional training cost ( 8 ; 1 ; 23 ; 17 ; 24 ; 22 ) . Representative methods include ZeroQuant ( 26 ) , GPTQ ( 7 ) , AWQ ( 14 ) , SmoothQuant ( 25 ) , QuIP# ( 24 ) , and SpinQuant ( 17 ) . These methods reduce memory footprint and memory traffic through reconstruction-based quantization, activation-aware scaling, outlier smoothing, rotations, or low-bit codebooks, and have become attractive for LLM deployment due to their accuracy–efficiency trade-offs. QAT can better adapt weights to low-bit constraints, but its training cost remains substantial at LLM scale, making PTQ the more common practical choice. Quantization for reasoning models. Recent work has begun to study how quantization affects reasoning-heavy tasks. 11 evaluate common PTQ methods, including GPTQ, AWQ, and SmoothQuant, on mathematical reasoning and show that extreme low-bit settings can substantially degrade accuracy. They further propose lightweight fine-tuning for recovery. 15 broaden the study across model families and quantization targets, including weights, activations, and KV cache, and observe that harder reasoning tasks suffer larger quantization-induced degradation. These studies show that reasoning models are more sensitive to quantization than standard LLM benchmarks reveal. Our work studies a different, deployment-critical effect. Prior work primarily evaluates final-answer accuracy, with limited analysis of how quantization changes reasoning length. 15 consider CoT length in a limited setting of distilled models. In contrast, we analyze quantized reasoning models across a range of scales and architectures, including both dense and MoE models, and jointly study accuracy, reasoning-token usage, trace-level behavior, and serving efficiency. We show that PTQ can preserve final-answer accuracy while inflating the reasoning CoT span, creating a hidden