TARQ: Tail-Aware Reconstruction Quantization for Rare-Word Robust Automatic Speech Recognition
Xinyu Wang, Ziyu Zhao, Ke Bai, Silin Meng, Dongming Shen, Xiao-Wen Chang, Yixuan He · 2026
Read TARQ: Tail-Aware Reconstruction Quantization for Rare-Word Robust Automatic Speech Recognition by Xinyu Wang, Ziyu Zhao, Ke Bai, Silin Meng, Dongming Shen, Xiao-Wen Chang, Yixuan He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27808.txt.
Introduction
Wang and colleagues fix a blind spot in shrinking speech-recognition models for deployment: rare words such as names and numbers get little weight during compression. Their label-free method, TARQ, rebalances calibration toward the vocabulary's tail and corrects the leftover error consistently. Across eight model backbones and six datasets it improved rare-word accuracy without harming overall word error.
Catalog record
| CREATOR | Xinyu Wang, Ziyu Zhao, Ke Bai, Silin Meng, Dongming Shen, Xiao-Wen Chang, Yixuan He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87785 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27808
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TARQ: Tail-Aware Reconstruction Quantization for Rare-Word Robust Automatic Speech Recognition Thanks: Equal contribution. Emails: [email protected] , [email protected] . Xinyu Wang, Ziyu Zhao, Ke Bai, Silin Meng, Dongming Shen, Xiao-Wen Chang, Yixuan He Abstract Data-aware post-training quantization (PTQ) minimizes a per-token reconstruction loss on a small calibration corpus, implicitly weighting positions by their empirical frequency. For A utomatic S peech R ecognition (ASR), this misaligns with tail-sensitive risk: names, numerals, and domain-specific words receive proportionally little calibration mass. We propose Tail-Aware Reconstruction Quantization ( TARQ ), a label-free PTQ framework that shifts calibration toward the lexical tail via rareBAL , a closed-form per-Linear-layer rule equalizing common/tail mass, paired with a metric-consistent residual correction. TARQ requires no entity labels, no curated calibration set, no validation decoding, and no additional training. Across eight ASR backbones and six datasets at W4G128, TARQ improves mean rare- W ord E rror R ate (rare-WER) without an aggregate-WER regression, achieves the lowest cross-corpus rare-WER swing among compared methods, and transfers to entity-rich benchmarks (ProfASR, ContextASR-Speech-En) without entity supervision. 1 Introduction Large-scale end-to-end automatic speech recognition (ASR) models based on Transformer architectures 27 ; 19 ; 21 ; 14 have substantially improved recognition accuracy, but their growing size makes deployment increasingly constrained by memory bandwidth and device capacity. Weight-only post-training quantization (PTQ) is an attractive compression strategy: each Linear-layer full-precision weight matrix W is replaced by a low-bit (e.g., 4-bit integer) approximation \widehat{W} while activations stay in floating point, and PTQ chooses \widehat{W} from a small calibration set to keep the per-layer output \widehat{W}x close to Wx without any gradient updates or training-time data. Most modern low-bit PTQ methods 7 were developed primarily for large language models. Their data-aware variants share a common principle: minimize a reconstruction loss averaged over calibration token positions. This averaging encodes an implicit calibration choice—because every position contributes equally to the average, the effective calibration metric inherits the empirical token-frequency distribution of the corpus, so frequent lexical patterns dominate while infrequent ones contribute only in proportion to how often they appear. This is appropriate when the deployment target is average-token preservation, but it is a design choice rather than a fundamental requirement, and it has not been examined for ASR. This choice can be misaligned with tail-sensitive recognition risk. Names, numerals, technical terms, and domain-specific vocabulary are a recurring focus of ASR adaptation methods such as contextual biasing 12 ; 23 ; 24 , and these positions tend to be lexically rare. Rare positions are plausible candidates for fragility under quantization perturbation: they often correspond to lower-frequency lexical decisions where a small weight change may be more likely to alter the decoded token. Under a frequency-weighted calibration metric, this fragile lexical-tail slice receives only frequency-proportional optimization mass, and aggregate WER further obscures any resulting degradation because its denominator is dominated by common words. An ideal weighting would be derived from per-position loss sensitivity (e.g., gradient or Fisher information), but this requires labels and backward passes incompatible with the one-pass nature of PTQ calibration. We propose Tail-Aware Reconstruction Quantization ( TARQ ), a reconstruction-based PTQ framework that reweights the calibration metric itself, using lexical rarity as a label-free proxy for fragile positions. TARQ has two closed-form components. rareBAL replaces the host PTQ calibration metric with a common/tail-balanced one, where the balancing coefficient is set per Linear layer (i.e., each y=Wx projection inside the model) by trace equalization. A scalar residual correction then keeps the sequential PTQ sweep aligned with the reweighted objective. Both components reuse the activation second-moment statistics that data-aware PTQ already accumulates, adding no extra calibration pass. Main contributions. • We identify an implicit position-weighting choice in reconstruction-based PTQ: calibration metrics inherit the corpus token-frequency distribution, misaligning with tail-sensitive ASR risk in a way that aggregate WER hides. • We introduce TARQ , a PTQ framework with two closed-form components that reuse the second-moment statistics from a single calibration pass: rareBAL , a label-free per-Linear-layer trace-equalization reweighting, and a metric-consistent scalar residual correction. • Across eight ASR backbones, six datasets, and six calibration corpora, TARQ improves rare-WER on the majority of backbones, remains competitive on plain WER, and is the most calibration-stable method; entity-rich evaluations confirm that the rarity-based shift transfers without entity supervision. 2 Related Work Weight-only PTQ for LLMs. Modern PTQ has been developed primarily for large language models. GPTQ 5 minimizes a per-column reconstruction loss under an inverse-Hessian metric; AWQ 13 minimizes a per-channel salience-weighted reconstruction; OmniQuant 20 jointly fits per-channel scales and clipping bounds; SpQR 3 keep a small fraction of weights at higher precision along the weight axis. Despite differences in parameterization and solver, all data-aware families share the same per-token activation reconstruction loss, and are evaluated primarily on LLM perplexity rather than ASR recognition risk. TARQ addresses this objective-level mismatch by rebalancing the calibration loss itself, so that rounding is no longer driven primarily by high-frequency lexical mass. PTQ for ASR. PTQ for ASR remains less explored than LLM PTQ, and existing methods do not resolve low-bit weight-only quantization for modern encoder-decoder ASR backbones. Integer-only zero-shot quantization targets INT 8 CTC/RNN-era models such as QuartzNet, Jasper, and Conformer 11 . StableQuant 9 focuses on CNN front-ends of encoder-only HuBERT/wav2vec 2.0 models and mainly reports W 6 A 6 results. GenPTQ 10 studies mixed-precision PTQ for Whisper and Conformer, but bit allocation acts above the rounding objective and does not correct token-level imbalance. Edge-ASR 4 benchmarks PTQ hosts on Whisper and shows severe low-bit WER degradation, but leaves its cause and remedy open. In contrast, TARQ targets the shared calibration failure itself: uniform reconstruction is frequency-aware but entity-agnostic. By rebalancing common/tail optimization mass and solving the corrected rounding problem, TARQ directly optimizes for the rare-entity errors that prior ASR PTQ methods do not address. 3 Diagnosis: Frequency Inheritance in Data-Aware PTQ Data-aware PTQ averages a per-position reconstruction loss across the calibration corpus. This averaging encodes an implicit position weighting: the calibration metric inherits the empirical token frequency of the corpus, so lexical-tail positions enter the objective only in proportion to how often they appear. We make this implicit weighting explicit in two steps. Section 3.2 decomposes the reconstruction loss into a frequency-weighted common/tail mixture and shows its downstream signature: rare positions incur larger WER degradation than common positions across PTQ solvers. Section 3.3 shows that optimizing the same frequency-weighted objective more aggressively does not close this gap, locating the common/tail asymmetry in the calibration metric itself, not in how tightly that metric is minimized. Figure 1: Diagnosis of frequency-weighted PTQ. (a) Raw Hessian rare-mass share : across all eight backbones, rare-token positions hold only a small fraction of the per-Linear-layer calibration trace mass under the raw second-moment metric H_{\ell} (median across decoder Linear layers). (b) Frequency-skewed damage : across PTQ solvers, rare positions degrade more than common ones, the recognition-level footprint of eq. 4 . Error bars: bootstrap 95\% CI. (c) Tighter optimization does not repair rare-WER : OmniQuant iteration sweep on Whisper-base drops the per-block reconstruction-loss ratio substantially, but plain WER saturates and rare-WER can regress. 3.1 Preliminaries Weight-only uniform quantization. Weight-only PTQ replaces a Linear-layer weight W\in\mathbb{R}^{m\times n} with a low-bit dequantized approximation \hat{W} . We use the standard min-max symmetric scheme at bit-width b