BitCal-TTS : Bit-Calibrated Test-Time Scaling for Quantized Reasoning Models
Sai Babu Patarlapalli, Surya Teja Avvaru · 2026
Read BitCal-TTS : Bit-Calibrated Test-Time Scaling for Quantized Reasoning Models by Sai Babu Patarlapalli, Surya Teja Avvaru (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05561.txt.
Introduction
Shrinking a reasoning model to four bits saves memory but can break its sense of when to stop thinking. Sai Babu Patarlapalli and Surya Teja Avvaru present BitCal-TTS, a runtime controller for that problem. Under a fixed token budget, poor confidence signals can end a trace too early or let it ramble.
The controller blends cheap uncertainty proxies with a confidence rescaling tuned to the bit width. On small GSM8K test shards with Qwen2.5 models, it raised exact-match accuracy at two sizes. Premature stops fell at both sizes as well, while most token savings survived.
Catalog record
| CREATOR | Sai Babu Patarlapalli, Surya Teja Avvaru |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BitCal-TTS : Bit-Calibrated Test-Time Scaling for Quantized Reasoning Models Thanks: Code: https://github.com/Saibabu7770/bitcal-tts Sai Babu Patarlapalli, Surya Teja Avvaru Abstract Post-training quantization makes large reasoning models practical under tight memory and latency budgets, but it can distort the online signals that drive adaptive test-time compute allocation. Under a fixed cap on the number of newly generated tokens, miscalibrated confidence can lead to harmful early halting: the model may surface a plausible final line while the underlying reasoning is still wrong, or the controller may stop before the trace has stabilized. We study this interaction for greedy 4-bit inference and propose BitCal-TTS , a lightweight runtime controller that combines (i) inexpensive online proxies for token-level uncertainty and reasoning-trace stability, (ii) a bit-conditioned confidence rescaling that is conservative at low nominal precision, and (iii) a bit-aware post-marker confirmation horizon designed for GSM8K-style structured outputs. The method requires no fine-tuning of the base model and integrates with standard Hugging Face 4-bit inference using forward hooks for logits and last-layer hidden states [ 1 , 2 ] . On small evaluation shards of GSM8K [ 3 ] with Qwen2.5 Instruct models [ 4 ] , BitCal-TTS improves exact-match accuracy over a non–bit-aware adaptive baseline at the 7B and 14B scales while preserving substantial token savings relative to fixed-budget decoding. At a token cap of B = 512 B{=}512 , on the evaluation shards we report ( N = 54 N{=}54 for 7B and N = 35 N{=}35 for 14B; not the full GSM8K test set), accuracy gains are + 3.7 +3.7 points (7B) and + 2.8 +2.8 points (14B), with the premature-stop rate falling from 14.8 % 14.8\% to 11.1 % 11.1\% on 7B and from 17.1 % 17.1\% to 11.4 % 11.4\% on 14B. We report Wilson 95% confidence intervals throughout and explicitly discuss the limited statistical power of the partial-shard comparisons. We release code and figure-generation scripts to support full reproduction. Keywords: test-time scaling, quantization, adaptive halting, GSM8K reasoning, uncertainty calibration, large language models. 1 Introduction Reasoning-centric large language models (LLMs) often benefit from spending additional compute at inference time. Chain-of-thought style deliberation [ 5 , 6 ] and more recent sequential test-time scaling strategies [ 7 , 8 ] can substantially improve verifiable accuracy on math and logic tasks. In production deployments, however, this compute is almost always bounded : a token cap B controls latency and cost, and product surfaces frequently add early-exit heuristics whenever an answer “looks complete”. This paper focuses on a setting that is increasingly common in deployment: a causal , instruction-tuned LLM served under aggressive post-training quantization (e.g., 4-bit weights via bitsandbytes [ 2 , 9 , 10 ] ) under a hard token budget B . Quantization expands the set of models that fit on consumer GPUs, but it also alters the geometry of logits and hidden states. Online halting cues such as token entropy and trace stability become miscalibrated relative to full precision: the policy may appear confident while the underlying reasoning remains unreliable, increasing the risk of stopping too soon [ 11 , 12 ] . The failure mode is twofold—reduced final accuracy and adaptive compute that is wasted halting on a spurious “final” segment. Research questions. We address three questions: (Q1) How does aggressive 4-bit quantization affect the reliability of adaptive halting signals when the controller treats them as if the model were full precision? (Q2) Can a precision-aware adjustment to confidence and to the post-answer confirmation window recover a meaningful slice of the lost accuracy without training the base model? (Q3) How do these effects scale across reasoning model sizes (3B / 7B / 14B) on a representative math reasoning benchmark? Contributions. • We formalize a budgeted, stepwise inference loop for quantized causal LMs with halting actions \{\texttt{continue},\texttt{stop},\texttt{escalate}\} and contrast (a) fixed-budget decoding, (b) adaptive decoding with a precision- agnostic calibrator, and (c) the proposed BitCal-TTS . • We introduce transparent, implementation-aligned proxies for token entropy, reasoning-trace stability, and last-layer hidden-state drift, and show how a bit-width multiplicative scale makes stop decisions conservative at aggressive quantization. • We propose a GSM8K-oriented post-marker confirmation rule: once the standard #### answer delimiter appears, decoding switches to a bit-conditioned tail budget before termination is permitted. This avoids treating the delimiter as an immediate halt signal, which we find to be brittle under 4-bit noise. • We report GSM8K results on Qwen2.5-3B/7B/14B Instruct under 4-bit inference, including a multi-budget sweep on the 7B model. BitCal-TTS recovers a meaningful slice of the adaptive–vs–fixed accuracy gap on 7B and 14B while preserving large token savings relative to always consuming the full budget. We additionally document a regime—Qwen2.5-3B at 4-bit—in which neither adaptive variant works, and analyze its proximate cause (Section 7 ). Paper organization. Section 2 formalizes adaptive compute under hard caps and reviews how quantization perturbs online halting signals. Section 3 positions this work relative to test-time scaling, adaptive decoding, verification, and quantization reliability. Section 4 specifies the proposed controller. Section 5 describes the experimental protocol. Sections 6 and 7 present and analyze the empirical findings. Section 8 discusses limitations and broader impact. Section 9 concludes. 2 Background and Motivation Adaptive compute under a hard cap. Let B\in\mathbb{N} denote the maximum number of new tokens that may be produced for a single prompt. A fixed policy always requests B tokens (or stops at end-of-sequence), which is safe but often inefficient when many problems admit shorter reasoning chains. An adaptive policy interleaves short generation segments with cheap measurements and may terminate before the budget is exhausted, ideally without sacrificing quality. Quantization changes online signals. Post-training quantization maps weights—and sometimes activations—to low-bit containers while attempting to preserve downstream quality [ 9 , 10 , 13 , 14 ] . A separate question, central to this work, is how quantization affects online halting signals derived from logits and activations during autoregressive decoding. If low-bit inference inflates premature confidence relative to full precision, an adaptive controller will halt earlier than intended, trading away accuracy without realizing commensurate savings relative to a well-tuned fixed budget. Structured final answers in GSM8K. GSM8K [ 3 ] adopts a standard extraction protocol in which the final numeric answer follows a delimiter token, #### . The delimiter is convenient for parsing, but under quantization it can appear in locally fluent yet globally incorrect traces. BitCal-TTS therefore treats delimiter detection as a phase change : after the first occurrence, decoding continues for a precision-conditioned horizon before termination is permitted. 3 Related Work Test-time scaling and adaptive decoding. Inference-time compute can be scaled by allowing longer chains of thought [ 5 ] , by sampling and aggregating multiple candidates [ 6 ] , or by structured search such as Tree-of-Thoughts and ReAct [ 15 , 16 ] . More recent work studies how to scale test-time compute optimally and how small, simple test-time scalers can rival much larger budgets [ 7 , 8 , 17 ] . Closer to our regime, adaptive computation policies expand generation depth based on confidence or estimated difficulty [ 18 , 19 ] . Most published policies are described and tuned for full-precision models; our experiments isolate the additional error introduced by aggressive quantization while keeping the controller skeleton fixed. Verification and process