Dynamic Quantization Error Propagation in Encoder-Decoder ASR Quantization Xinyu Wang 1* , Yajie Luo 2* , Yihong Wu 2* , Liheng Ma 1,5* Ziyu Zhao 1 † \dagger , Jingrui Tian 3 , Lei Ding 4 , Yufei Cui 1,5 and Xiao-Wen Chang 1 Abstract Running Automatic Speech Recognition (ASR) models on memory-constrained edge devices requires efficient compression. While layer-wise post-training quantization is effective, it suffers from error accumulation, especially in encoder-decoder architectures. Existing solutions like Quantization Error Propagation (QEP) [ 1 ] are suboptimal for ASR due to the model’s heterogeneity—processing acoustic features in the encoder versus generating text in the decoder. To address this, we propose Fine-grained Alpha for Dynamic Quantization Error Propagation (FADE), which adaptively controls the trade-off between cross-layer error correction and local quantization. Experiments show that FADE significantly improves stability by reducing performance variance across runs, while simultaneously surpassing baselines in mean WER. I Introduction Encoder-Decoder Transformer architectures have become the dominant paradigm in modern Automatic Speech Recognition (ASR). The latest models, such as OpenAI’s Whisper [ 17 ] and the Moonshine family [ 10 ] , demonstrate exceptional capability in handling multilingual speech and diverse acoustic environments. However, widespread deployment on resource-constrained edge devices (e.g., mobile phones, IoT, and wearables) remains challenging due to tight limits on compute, memory, and energy. Deploying these models imposes prohibitive storage and memory bandwidth requirements [ 16 ] , challenging the limited storage capacity of commodity hardware and hampering real-time inference speeds due to the high cost of data movement [ 2 , 11 ] . Previous works have explored various strategies to mitigate these constraints: [ 6 , 3 ] proposes efficient pruning without back-propagation to remove redundant weights; [ 7 ] investigates ultra-low bit quantization via K-Means clustering and mixed-precision allocation. However, these approaches often require altering the model structure or demanding complex hardware support for non-uniform bit-widths. Fig. 1: Visualization of WER v.s. Whisper model size for different quantization techniques. The bubble size represents the standard deviation across runs on 4-bit weight quantization. FADE acquires not only better mean WER but also smaller variance across runs. In the meantime, Post-Training Quantization (PTQ) techniques, particularly fixed-precision Weight-Only Quantization (WOQ), have been widely introduced for Large Language Models (LLMs), which offer a more practical and deployable solution by compressing all weights to uniform low-bit integers (e.g., 3, 4, or 8-bit) without the prohibitive cost of retraining [ 4 , 20 , 2 , 13 , 18 , 19 ] . In particular, Hessian-based methods (e.g., GPTQ [ 4 ] ) minimize layer-wise reconstruction error using second-order information computed from a small calibration set with the objective function: \min_{\widehat{\mathbf{W}}_{l}\in\mathbb{Q}^{n_{l}\times d_{l}}}\|\mathbf{W}_{l}\widehat{\mathbf{X}}_{l}-\widehat{\mathbf{W}}_{l}\widehat{\mathbf{X}}_{l}\|_{F}^{2}, (1) where \widehat{\mathbf{W}} denotes the targeted quantized weight in the quantized space \mathbb{Q}^{n_{l}\times d_{l}} and \widehat{\mathbf{X}}_{l} denotes the activation from the previous quantized layer. Their low computational overhead and hardware-friendly uniform precision make them an ideal foundation for edge deployment. However, the layer-wise quantization might impose exponential accumulation and growth of quantization error on \widehat{\mathbf{X}}_{l} in each layer. Therefore, earlier work [ 1 ] proposes Quantization Error Propagation (QEP) to correct the quantized error via approximating the objective: \min_{\widehat{\mathbf{W}}_{l}\in\mathbb{R}^{n_{l}\times d_{l}}}\|\mathbf{W}_{l}\mathbf{X}_{l}-\widehat{\mathbf{X}}_{l}\widehat{\mathbf{W}}_{l}\|_{F}^{2}, (2) where \mathbf{X}_{l} is the clean activation. In practice, directly solving the Eq. ( 2 ), though effectively reducing the accumulation of quantization error, will lead to severe overfitting to the calibration dataset. To address this issue, QEP introduces a tunable error-correction mechanism that interpolates between the standard PTQ objective in Eq. ( 1 ) and the error-propagation objective in Eq. ( 2 ), enabling a controllable trade-off between fidelity and generalization. While QEP has demonstrated remarkable efficacy on LLMs (Decoder-Only Transformers), directly applying it to ASR models (Encoder-Decoder Transformers) leads to instability (as shown in Fig. 1 ). A key difference is that ASR systems couple heterogeneous modules: an audio encoder operating on acoustic features and a language decoder operating on text-like representations. The different calibration sensitivity and weight outliers in the audio modules and in the language modules necessitate a more fine-grained control on the propagation coefficients in the error correction mechanism beyond QEP. Motivated by this, we introduce Fine-grained Alpha for Dynamic Quantization Error Propagation (FADE) – a diagnostic-driven mechanism that adaptively harmonizes layer-wise quantization with global error correction for each layer. Notably, our method requires neither additional calibration data nor search overhead. We conduct comprehensive experiments on representative Encoder–Decoder ASR models, including Whisper and Moonshine, and demonstrate that the proposed FADE consistently delivers better performance under low-bit weight quantization. In particular, FADE achieves lower word error rates than prior post-training quantization approaches across multiple bit-widths, while substantially reducing performance variance across runs. These results indicate that fine-grained layer-wise error correction coefficients are critical for stable and effective quantization of heterogeneous ASR architectures. II Background II-A Layer-Wise Post-Training Quantization Layer-wise post-training quantization methods perform quantization layer by layer [ 4 , 13 ] . Concretely, given the weight matrix \mathbf{W}_{l}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}} at the l -th layer and the input activation of that layer \widehat{\mathbf{X}}_{l}\in\mathbb{R}^{d_{\text{in}}\times N} from N calibration datapoints, the goal of quantization is to find a low-precision weight matrix \widehat{\mathbf{W}}_{l} to approximate the original weight under given constraints. Mathematically, it is equivalent to solving the following optimization problem: \widehat{\mathbf{W}}_{l}=\underset{\widetilde{\mathbf{W}}_{l}\in\mathbb{Q}^{n_{l}\times d_{l}}}{\arg\min}\|\mathbf{W}_{l}\widehat{\mathbf{X}}_{l}-\widetilde{\mathbf{W}}_{l}\widehat{\mathbf{X}}_{l}\|_{F}^{2}\,, (3) where \mathbb{Q} defines the quantization constraints (e.g., bit-width, grouping). Note that \widehat{\mathbf{X}}_{l} is obtained by running through previous quantized layers following the convention of [ 4 ] with full precision (FP16/BF16). Many existing post-training quantization approaches adopt symmetric uniform group-wise quantization . Let \mathbf{w}\in\mathbb{R}^{g} denote a weight group, typically corresponding to a contiguous subvector of a column of the weight matrix \mathbf{W} . Given a quantization scale s and a target bit-width b , the quantized representation is defined as \widehat{\mathbf{w}}=s\cdot\text{clamp}\!\left(\left\lfloor\frac{\mathbf{w}}{s}\right\rceil,\;-2^{b-1},\;2^{b-1}-1\right), (4) where \lfloor\cdot\rceil denotes round-to-nearest, and \text{clamp}(\cdot) restricts values to the integer range supported by b -bit signed quantization. II-B Hessian-Based Methods The standard least-squares reconstruction loss assigns equal weight to all entries of the weight matrix. By leveraging second-order information, one can enable the identification of structurally important regions within the weight matrices based on calibration data. This principle traces back to Optimal Brain Damage [ 12 ] and has been subsequently adopted and extended by recent Hessian-aware post-training quantization methods, including Optimal Brain Quantization (OBQ) [ 5 ] and GPTQ [ 4 ] . Let \mathcal{L}(\widehat{\mathbf{W}}_{l}):=\|\mathbf{W}_{l}\widehat{\mathbf{X}}_{l}-\widehat{\mathbf{W}}_{l}\widehat{\mathbf{X}}_{l}\|_{F}^{2} and \Delta\mathbf{W}_{l}=\widehat{\mathbf{W}}_{l}-\mathbf{W}_{l} . The Taylor expansion of \mathcal{L} at \mathbf{W}_{l} is \mathcal{L}(\widehat{\mathbf{W}}_{l})=\text{Tr}\left(\Delta\mathbf{W}_{l}\widehat{\mathbf{H}}_{l}\Delta\mathbf{W}_{l}^{\top}\right)+o((\Delta\mathbf{W}_{l})^{3})\,, (5) where \widehat{\mathbf{H}}_{l}=\widehat{\mathbf{X}}\widehat{\mathbf{X}}_{l}^{\top} . Both the 0th-order and 1st-order terms vanish in this expansion. As a result, the quantized weights are obtained via a greedy, Hessian-aware procedure that iteratively quantizes a subset of weights while updating the remaining full-precision weights to compensate for the induced quantization error, leveraging inverse Hessian information [ 8 , 5 , 4 ] . A detailed derivation of this procedure, along with the explicit update algorithm, is provided in Appendix A . II-C Quantization Error Accumulation and Error Correction Consider a model f with m layers, f:=f_{m}\circ f_{m-1}\circ\cdots\circ f_{2}\circ f_{1}(\mathbf{X}), (6) and its quantized counterpart \widehat{f}:=\widehat{f}_{m}\circ\widehat{f}_{m-1}\circ\cdots\circ\widehat{f}_{2}\circ\widehat{f}_{1}(\mathbf{X}). (7) The total quantization error is then defined as \delta=\|f(\mathbf{X})-\widehat{f}(\mathbf{X})\|_{F}^{2}. (8) Due to layer-wise independent quantization, the error introduced at each layer \|f_{i}(\mathbf{X})-\widehat{f}_{i}(\mathbf{X})\|_{F}^{2} propagates to subsequent layers without correction, resulting in an exponential accumulation of error [ 1 ] . Note that the error accumulation not only occurs in decoder-only Transformers like LLMs, but also happens in encoder-decoder Transformers such as ASR models. Quantization Error Propagation (QEP) [ 1 ] propose to correct the optimization target as follows: \displaystyle\mathbf{W}_{l}^{*}(\alpha_{l})=\mathbf{W}_{l}+\alpha_{l}\mathbf{W}_{l}\boldsymbol{\delta}_{l}\widehat{\mathbf{X}}_{l}^{\top}\widehat{\mathbf{H}}_{l}^{-1} (9) \displaystyle\min_{\widehat{\mathbf{W}}_{l}\in\mathbb{Q}^{n_{l}\times d_{l}}}\|\mathbf{W}_{l}^{*}(\alpha_{l})\widehat{\mathbf{X}}_{l}-\widehat{\mathbf{W}}_{l}\widehat{\mathbf{X}}_{l}\|_{F}^{2}, where \boldsymbol{\delta}_{l}:=\mathbf{X}_{l}-\widehat{\mathbf{X}}_{l} and \alpha_{l}\in[0,1] is a predetermined hyperparameter. Setting \alpha_{l}=0 recovers the original objective in Eq. ( 1 ) whereas setting \alpha_{l}=1 corresponds to the fully-corrected objective approximating to Eq. ( 2 ). It is important to note that, although \{\alpha_{l}\} are tunable parameters, identifying their optimal values is non-trivial. The choice of \alpha_{l} depends heavily on the weight distribution of the corresponding layer, the depth of the layer within the network, the type of data modality processed by the layer, as well as the layer’s functional role (e.g., encoder or decoder). In the original QEP paper, a common practice is to set \alpha_{l}=1/2 for all layers, except in certain special cases where layer-specific adjustments are necessary. A systematic method for determining the optimal value of \alpha_{l} has not been established. Fig. 2: Sensitivity study of \alpha\in[0,1] for GPTQ+QEP and FADE using 3-bit weight quantization. We use a fixed \alpha across all layers. The Whisper-Tiny model is calibrated on a subset of 128 samples from LibriSpeech-clean and evaluated on 100 samples from LibriSpeech-other. Bubble size and color gradient represent the standard deviation across 3 independent runs. Fig. 3: Overview of dynamic quantization error propagation in encoder–decoder ASR models. (a) Quantization flow under encoder–decoder asymmetry. The encoder is calibrated using an audio calibration dataset, while the decoder is calibrated using text data together with the quantized encoder outputs. Vanilla calibration quantization refers to calibration-dataset-based PTQ, instantiated as GPTQ [ 4 ] , which optimizes the standard objective in Eq. ( 1 ). To account for error propagation from the encoder to the decoder, corrected quantization introduces a dynamic scaling factor \alpha_{\ell} to adjust the update direction toward the corrected objective in Eq. ( 9 ). The figure provides a schematic view; the same procedure is applied independently within each encoder and decoder block. (b) Dynamic computation of the layer-wise scaling factor \alpha_{\ell} based on error propagation indicators, enabling adaptive quantization. III Methodology As discussed in Section II-C , quantization errors in encoder–decoder ASR models accumulate across successive blocks. While QEP mitigates this effect by compensating for propagated errors, it relies on a single correction strength \alpha across all layers. This uniform approach is suboptimal [ 1 ] , as shown in Fig. 2 . This limitation is especially pronounced in multimodal encoder-decoder Transformers, such as ASR models, where substantial heterogeneity in calibration data and weight distributions leads to markedly different layer-wise sensitivities to quantization noise. Conversely, manually tuning layer-specific coefficients \{\alpha_{l}\} is prohibitively expensive and practically infeasible for large-scale models. The optimal \{\alpha_{l}\} values vary significantly across layers, depending on factors such as the target quantization bit-width, the processed modality, the overall model architecture, and the layer type. Motivated by this insight, we propose a diagnostic-driven, systematic framework to determine layer-specific coefficients \{\alpha_{l}\} at a fine-grained level. TABLE I: Word Error Rate (WER \downarrow ) comparison on LibriSpeech. Calibration on LibriSpeech-clean and test on LibriSpeech-other. FP16 denotes the uncompressed baseline. Best results are in bold . Method Whisper (WER \downarrow ) Moonshine (WER \downarrow ) Tiny Base Small Tiny Base FP16 23.11 12.94 11.54 12.54 9.02 3-bit Weights RTN 238.33 50.02 11.45 189.88 134.93 AWQ [ 13 ] 200.13 \pm 22.13 34.72 \pm 3.47 12.71 \pm 1.32 179.13 \pm 32.13 14.39 \pm 0.83 GPTQ [ 4 ] 120.45 \pm 24.63 33.78 \pm 5.09 11.66 \pm 1.77 171.08 \pm 28.89 12.54 \pm 0.63 GPTQ+QEP [ 1 ] 94.62 \pm 15.92 33.43 \pm 7.46 10.79 \pm 1.21 298.53 \pm 230.77 11.74 \pm 0.70 FADE (Ours) 62.31 \pm 12.61 34.10 \pm 1.89 10.96 \pm 1.04 182.73 \pm 16.94 11.63 \pm 0.59 4-bit Weights RTN 64.49 15.31 12.28 18.45 10.57 AWQ [ 13 ] 35.47 \pm 1.97 20.16 \pm 0.91 13.34 \pm 1.34 22.34 \pm 3.42 9.54 \pm 0.15 GPTQ [ 4 ] 34.67 \pm 5.07 17.33 \pm 2.89 12.69 \pm 0.89 21.37 \pm 4.69 9.48 \pm 0.17 GPTQ+QEP [ 1 ] 32.77 \pm 5.05 16.92 \pm 1.58 11.33 \pm 1.57 21.83 \pm 6.93 9.39 \pm 0.12 FADE (Ours) 29.88 \pm 0.94 16.21 \pm 1.29 11.17 \pm 0.73 16.23 \pm 0.92 9.40 \pm 0.06 III-A QEP with Diagnostic-Driven Adaptation We decompose the layer-wise optimal correction strength into two orthogonal components: Intrinsic Weight Vulnerability and Calibration Reliability . III-A 1 Component I: Intrinsic Weight Vulnerability We first evaluate the quantization difficulty inherent to the weight distribution, independent of calibration data. Let \widehat{\mathbf{W}}_{l}^{\text{rtn}} denote the quantized weights obtained via Round-to-Nearest (RTN) [ 20 ] , which minimizes the local error strictly in the weight space. The normalized RTN reconstruction error is defined as: e_{r}=\frac{\|\mathbf{W}_{l}-\widehat{\mathbf{W}}_{l}^{\text{rtn}}\|_{F}}{\|\mathbf{W}_{l}\|_{F}+\varepsilon}. (10) This metric e_{r} serves as a proxy for the presence of outliers or heavy-tailed distributions often observed in ASR models. A high e_{r} implies that the layer is intrinsically ”vulnerable” to quantization error. We define the Intrinsic Weight Vulnerability term as: \mathcal{V}_{\text{int}}(l)=k_{1}\cdot\log(1+e_{r}). (11) Intuitively, layers with high intrinsic vulnerability require stronger error compensation mechanisms (higher \alpha ) to mitigate the significant local distortion. TABLE II: Cross-dataset evaluation (W3A16&W4A16). WER ( \downarrow ) is reported for Whisper-Base and Moonshine-Base. Method Whisper Base (WER \downarrow ) Moonshine Base (WER \downarrow ) Libri-clean SPGISpeech TED-Lium Libri-clean SPGISpeech TED-Lium FP16 5.04 15.49 17.63 3.87 7.09 17.08 3-bit Weights RTN 23.04 41.18 35.47 7.52 122.5 87.73 AWQ 18.07 \pm 2.15 34.18 \pm 8.39 36.50 \pm 6.98 5.32 \pm 0.41 12.9 \pm 1.21 19.4 \pm 0.89 GPTQ 16.41 \pm 1.43 32.64 \pm 10.15 38.38 \pm 7.33 4.98 \pm 0.18 9.66 \pm 0.96 17.18 \pm 0.84 GPTQ+QEP 16.45 \pm 2.53 26.90 \pm 5.21 35.59 \pm 4.63 4.73 \pm 0.20 9.28 \pm 0.32 18.19 \pm 0.85 FADE (Ours) 14.22 \pm 2.12 27.02 \pm 2.65 32.13 \pm 2.46 3.10 \pm 0.19 9.06 \pm 0.19 17.48 \pm 0.56 4-bit Weights RTN 5.69 16.19 16.55 4.51 13.44 16.41 AWQ 5.33 \pm 0.12 18.27 \pm 3.43 21.23 \pm 1.76 4.31 \pm 0.14 8.34 \pm 0.57 16.29 \pm 0.48 GPTQ 5.50 \pm 0.17 19.33 \pm 3.93 19.39 \pm 1.33 4.26 \pm 0.13 8.72 \pm 0.49 16.10 \pm 0.54 GPTQ+QEP 4.86 \pm 0.05 15.56 \pm 3.17 20.93 \pm 2.20 4.08 \pm 0.18 8.13 \pm 0.59 16.52 \pm 0.85 FADE (Ours) 4.87 \pm 0.01 15.34 \pm 1.31 22.99 \pm 4.41 3.86 \pm 0.03 8.09 \pm 0.26 16.37 \pm 0.38 III-A 2 Component II: Calibration Reliability Score The second component assesses the effectiveness of the data-driven optimization derived from calibration samples. We denote the calibrated quantized weights as \widehat{\mathbf{W}}_{l}^{\text{calib}} . While our framework allows for various calibration strategies (e.g., gradient-based or search-based methods), in this work, we adopt the Hessian-based GPTQ algorithm [ 4 ] as the representative instantiation following previous state-of-the-art techniques [ 4 , 1 ] . Thus, \widehat{\mathbf{W}}_{l}^{\text{calib}} minimizes the output activation error using second-order information. To gauge the reliability of this calibration, we introduce two diagnostic metrics: 1. Calibration Gain ( \Delta ): The relative improvement of the calibrated solution over the RTN baseline, \Delta=\frac{e_{r}-e_{\text{calib}}}{e_{r}+\varepsilon},\quad\text{where }e_{\text{calib}}=\frac{\|\mathbf{W}_{l}-\widehat{\mathbf{W}}_{l}^{\text{calib}}\|_{F}}{\|\mathbf{W}_{l}\|_{F}+\varepsilon}. (12) A positive \Delta indicates that the calibration data successfully captures the loss landscape curvature, effectively reducing error. 2. Solution Stability ( e_{\text{stab}} ): The discrepancy between the data-free (RTN) and data-driven (calibrated) solutions, e_{\text{stab}}=\frac{\|\widehat{\mathbf{W}}_{l}^{\text{rtn}}-\widehat{\mathbf{W}}_{l}^{\text{calib}}\|_{F}}{\|\mathbf{W}_{l}\|_{F}+\varepsilon}. (13) Excessive divergence ( e_{\text{stab}} ) suggests potential overfitting to the specific calibration set or numerical instability in the optimization process (e.g., inverse Hessian instability). We combine these metrics into the Calibration Reliability Score : \mathcal{R}_{\text{calib}}(l)=k_{2}\cdot\max(\Delta,0)-k_{3}\cdot\log(1+e_{\text{stab}}). (14) This term rewards layers where calibration provides a clear gain ( \Delta ) while penalizing instability ( e_{\text{stab}} ). III-A 3 Dynamic \alpha_{l} Synthesis Finally, the adaptive propagation coefficient \alpha_{l} is synthesized by aggregating the intrinsic and reliability components: \displaystyle s_{l} \displaystyle=\mathcal{V}_{\text{int}}(l)+\mathcal{R}_{\text{calib}}(l) (15) \displaystyle=k_{1}\cdot\log(1+e_{r})+k_{2}\cdot\max(\Delta,0)-k_{3}\cdot\log(1+e_{\text{stab}}). The raw score s_{l} is mapped to the valid range [\alpha_{\min},\alpha_{\max}] via a sigmoid projection: \alpha_{l}=\text{clip}\left(\alpha_{\min}+(\alpha_{\max}-\alpha_{\min})\cdot\sigma(s_{l}),\;\alpha_{\min},\;\alpha_{\max}\right). (16) By decoupling intrinsic difficulty from calibration reliability, our method ensures that error propagation is aggressive only when the layer requires it (high \mathcal{V}_{\text{int}} ) and the correction direction is trustworthy (high \mathcal{R}_{\text{calib}} ). The complete procedure is summarized in Algorithm 1 . Algorithm 1 FADE 0: Pre-trained weights \{\mathbf{W}_{l}\}_{l=1}^{L} , Calibration data \mathbf{X} , Bit-width b 0: Quantized model weights \{\widehat{\mathbf{W}}_{l}\}_{l=1}^{L} 1: Initialize: Set scaling factors k_{1}=k_{2}=k_{3}=1 and bounds \alpha_{\min}=0.1,\alpha_{\max}=0.8 2: for l=1 to L do 3: // Step 1: Metric Calculation 4: \widehat{\mathbf{W}}_{l}^{\text{rtn}}\leftarrow\text{Round}(\mathbf{W}_{l}) {Data-free Baseline} 5: Update input activations \widehat{\mathbf{X}}_{l} based on \{\widehat{\mathbf{W}}_{i}\}_{i=1}^{l-1} 6: \widehat{\mathbf{H}}_{l}\leftarrow(\widehat{\mathbf{X}}_{l}^{\top}\widehat{\mathbf{X}}_{l})^{-1} {Compute Inverse Hessian} 7: \widehat{\mathbf{W}}_{l}^{\text{calib}}\leftarrow\text{GPTQ}(\mathbf{W}_{l},\widehat{\mathbf{H}}_{l}) {Data-driven Solution} 8: e_{r}\leftarrow\|\mathbf{W}_{l}-\widehat{\mathbf{W}}_{l}^{\text{rtn}}\|_{F}/(\|\mathbf{W}_{l}\|_{F}+\varepsilon) 9: e_{\text{calib}}\leftarrow\|\mathbf{W}_{l}-\widehat{\mathbf{W}}_{l}^{\text{calib}}\|_{F}/(\|\mathbf{W}_{l}\|_{F}+\varepsilon) 10: e_{\text{stab}}\leftarrow\|\widehat{\mathbf{W}}_{l}^{\text{rtn}}-\widehat{\mathbf{W}}_{l}^{\text{calib}}\|_{F}/(\|\mathbf{W}_{l}\|_{F}+\varepsilon) 11: \Delta\leftarrow(e_{r}-e_{\text{calib}})/(e_{r}+\varepsilon) 12: // Step 2: Dynamic \alpha Synthesis 13: \mathcal{V}_{\text{int}}\leftarrow k_{1}\log(1+e_{r}) 14: \mathcal{R}_{\text{calib}}\leftarrow k_{2}\max(\Delta,0)-k_{3}\log(1+e_{\text{stab}}) 15: s_{l}\leftarrow\mathcal{V}_{\text{int}}+\mathcal{R}_{\text{calib}} 16: \alpha_{l}\leftarrow\text{clip}(\alpha_{\min}+(\alpha_{\max}-\alpha_{\min})\cdot\sigma(s_{l}),\alpha_{\min},\alpha_{\max}) 17: // Step 3: Error Propagation 18: \boldsymbol{\Delta}_{l}\leftarrow\mathbf{W}_{l}\boldsymbol{\delta}_{l}\widehat{\mathbf{X}}_{l}^{\top}\widehat{\mathbf{H}}_{l}^{-1} 19: \widehat{\mathbf{W}}_{l}\leftarrow\arg\min_{\widetilde{\mathbf{W}}_{l}}\|(\mathbf{W}_{l}+\alpha_{l}\boldsymbol{\Delta}_{l})\widehat{\mathbf{X}}_{l}-\widetilde{\mathbf{W}}_{l}\widehat{\mathbf{X}}_{l}\|_{F}^{2} 20: end for 21: return \{\widehat{\mathbf{W}}_{l}\}_{l=1}^{L} The proposed dynamic \alpha can be viewed as balancing the bias-variance tradeoff in quantization. A high \alpha (strong error propagation) acts as an aggressive bias correction, attempting to fully cancel upstream errors. However, if the calibration data is noisy (low reliability), this introduces high variance, causing the model to overfit to specific calibration patterns. Our diagnostic score s_{l} effectively estimates the ”signal-to-noise ratio” of the calibration process, damping the correction ( \alpha\to 0 ) when the variance risk outweighs the bias reduction benefits. IV Experiments We evaluate FADE on the Whisper [ 17 ] (Tiny, Base, Small) and Moonshine [ 10 ] (Tiny, Base), comparing against state-of-the-art LLM quantization methods including RTN (Round-to-Nearest), AWQ [ 13 ] , GPTQ [ 4 ] and GPTQ+QEP [ 1 ] . IV-A Experimental Setup We evaluate on four benchmark datasets, including LibriSpeech [ 15 ] (clean and other), SPGISpeech [ 14 ] , and TED-Lium [ 9 ] , reporting Word Error Rate (WER) as the primary metric. We adopt symmetric per-group quantization with group sizes of 64 for Whisper, 72 for Moonshine-Base, and 52 for Moonshine-Small, focusing on 4-bit (W4A16) and aggressive 3-bit (W3A16) weight quantization. Calibration uses 128 random samples from LibriSpeech-clean. All results are averaged over five runs with different random seeds. Unless otherwise specified, we use a fixed and untuned configuration for the diagnostic weights, setting k_{1}{=}k_{2}{=}k_{3}{=}1 , and constrain the propagation coefficient to \alpha_{\ell}\in[\alpha_{\min},\alpha_{\max}] with \alpha_{\min}=0.1 and \alpha_{\max}=0.8 . Notably, these values are not selected through dataset- or model-specific hyperparameter search. Despite this minimal configuration, FADE already exhibits dynamic, layer-adaptive behavior and consistently outperforms QEP in both stability and average WER. We emphasize that these default settings are intentionally conservative; further tuning of the diagnostic weights or the \alpha range can potentially yield additional performance gains. Fig. 4: Visualization of average FADE \alpha_{l} for each Transformer block in quantized Whisper Models on LibriSpeech IV-B Main Results Table I presents the WER comparison on the LibriSpeech datasets, where models are calibrated on LibriSpeech-clean and evaluated on LibriSpeech-other. Overall, FADE outperforms baseline methods in the majority of settings. Notably, our method achieves not only a lower mean WER but also a reduced standard deviation, indicating superior robustness and stability. This improvement is particularly significant for Whisper-Tiny and Moonshine-Tiny. We hypothesize that smaller models are more sensitive to calibration data, and that FADE effectively identifies optimal \alpha values to stabilize the quantization process. Table II presents a WER comparison where models are calibrated on LibriSpeech-clean and evaluated on LibriSpeech-clean, SPGISpeech, and TED-Lium. These results demonstrate the generalization capability of FADE . This evaluation is particularly challenging due to the distribution shift between the calibration and test datasets. Notably, our method achieves significant improvements in the 3-bit setting, with the Moonshine-Base model exhibiting the most substantial gains. IV-C Visualization of \alpha_{l} in FADE To demonstrate that FADE performs dynamic quantization error correction via fine-grained, layer-wise \alpha_{l} values, we visualize the distribution of \alpha_{l} across layers in Fig. 4 . This observation validates our hypothesis regarding error propagation. The optimal \alpha_{l} values are influenced not only by layer depth but also by the layer’s functional role within the model (encoder or decoder) and the type of signals processed (acoustic signals or language). IV-D Ablation Study of FADE TABLE III: Word Error Rate (WER \downarrow ) over different components in FADE . Calibration on LibriSpeech-clean and test on LibriSpeech-other. Best results are in bold . k_{1} k_{2} k_{3} WER ( \downarrow ) ✗ ✗ ✗ 82.8 \pm 15.4 ✓ ✓ ✗ 93.5 \pm 21.7 ✗ ✗ ✓ 66.1 \pm 16.5 ✓ ✗ ✓ 64.2 \pm 16.2 ✗ ✓ ✓ 63.4 \pm 16.0 ✓ ✓ ✓ 62.3 \pm 12.6 To assess the contribution of FADE ’s three diagnostic components (corresponding to k_{1} , k_{2} , k_{3} in Eq. ( 15 )) to quantization performance, we perform an ablation study (Table III ). Using k_{1} and k_{2} without k_{3} yields no improvement, whereas including k_{3} with either k_{1} or k_{2} reduces WER. The combination of all three components achieves the lowest WER, indicating that each proposed signal provides complementary information. V Conclusion In this work, we show that directly applying QEP to encoder–decoder ASR models is suboptimal due to strong layer- and modality-level heterogeneity. To address this, we proposed FADE , a diagnostic-driven framework that adaptively determines layer-wise error propagation strength without manual tuning or retraining. Experiments on Whisper and Moonshine demonstrate that FADE consistently improves low-bit post-training quantization performance, particularly at 3-bit precision, while substantially reducing performance variance across runs. These results highlight the importance of dynamic fine-grained error control for robust deployment of quantized ASR systems. Appendix A Details of the Hessian-Based Methods In this section, we provide the mathematical details of the Hessian-based methods (e.g., GPTQ). A-A Second-Order Approximation The objective of post-training quantization is to minimize the perturbation of \mathbf{W} in layer output \mathbf{W}\mathbf{X} . Let \Delta\mathbf{W}=\widehat{\mathbf{W}}-\mathbf{W} be the weight matrix perturbation. The error can be approximated via Taylor expansion: \displaystyle E(\widehat{\mathbf{W}}) \displaystyle=\|\mathbf{W}\mathbf{X}-\widehat{\mathbf{W}}\mathbf{X}\|_{F}^{2} \displaystyle=\text{Tr}\left(\mathbf{X}^{\top}\Delta\mathbf{W}^{\top}\Delta\mathbf{W}\mathbf{X}\right) \displaystyle=\text{Tr}\left(\Delta\mathbf{W}\mathbf{H}\Delta\mathbf{W}^{\top}\right), (17) where \mathbf{H}=\mathbf{X}\mathbf{X}^{\top} is the Hessian matrix. Since the rows of \mathbf{W} effectively act independently in the linear layer computation, the problem can be decomposed into d_{\text{out}} independent sub-problems, one for each row \mathbf{w} : \min_{\widehat{\mathbf{w}}}(\mathbf{w}-\widehat{\mathbf{w}})^{\top}\mathbf{H}(\mathbf{w}-\widehat{\mathbf{w}}). (18) A-B Optimal Brain Quantization Optimal Brain Quantization (OBQ) [ 5 ] processes each row of the weight matrix independently. Within each row, OBQ quantizes one weight at a time and compensates for the resulting quantization error by adjusting the remaining weights using Hessian information. Specifically, the algorithm greedily selects the weight w_{q} that minimizes the error metric: w_{q}=\arg\min_{w_{q}}\frac{(w_{q}-\hat{w}_{q})^{2}}{[\mathbf{H}F^{-1}]{qq}},. (19) Subsequently, the remaining unquantized weights are updated by the vector \boldsymbol{\delta}_{F} : \boldsymbol{\delta}_{F}=\frac{w_{q}-\hat{w}_{q}}{[\mathbf{H}^{-1}F]{qq}}\cdot(\mathbf{H}^{-1}F){\cdot,q},, (20) where \mathbf{H}_{F}=\mathbf{X}_{F}\mathbf{X}_{F}^{\top} represents the Hessian matrix corresponding to the subset of currently unquantized channels \mathbf{X}_{F} . A-C Efficient Implementation via Cholesky Directly inverting \mathbf{H} is numerically unstable and computationally expensive. GPTQ addresses this by utilizing the Cholesky decomposition of the Hessian inverse, specifically computing the Cholesky decomposition of \mathbf{H} first (with damping for stability): \mathbf{H}=\mathbf{L}\mathbf{L}^{\top} . Information from \mathbf{H}^{-1} is effectively obtained by solving linear systems involving \mathbf{L} . The algorithm proceeds as follows for each column j in the weight matrix: 1. Quantize : The weight w_{j} is quantized to the nearest grid point \hat{w}_{j}=Q(w_{j}) . 2. Compute Error : The local error is \Delta w_{j}=\hat{w}_{j}-w_{j} . 3. Compensate : The error is propagated to all future columns k>j in the current block/row using the correlations stored in the Cholesky factor (or inverse Hessian): w_{k}\leftarrow w_{k}-\frac{[\mathbf{H}^{-1}]_{kj}}{[\mathbf{H}^{-1}]_{jj}}\Delta w_{j}. (21) This iterative process ensures that quantization error in earlier columns is partially cancelled out by adjustments in later columns, significantly recovering the accuracy loss compared to simple Round-to-Nearest (RTN). The complete procedure is summarized in Algorithm 2 . Algorithm 2 Hessian-Based Weight Quantization (GPTQ-Style) 0: Pre-trained Weight \mathbf{W} , Calibration Inputs \mathbf{X} , Quantizer Q(\cdot) 0: Quantized Weight \hat{\mathbf{W}} 1: Step 1: Hessian Calculation 2: Compute Hessian: \mathbf{H}=\mathbf{X}\mathbf{X}^{\top} 3: Add Damping: \mathbf{H}_{\text{damp}}=\mathbf{H}+\lambda\mathbf{I} {Ensure numerical stability} 4: Step 2: Cholesky Decomposition 5: Compute Cholesky inverse information for \mathbf{H}^{-1} {Via Cholesky \mathbf{L}\mathbf{L}^{\top}=\mathbf{H}_{\text{damp}} } 6: Step 3: Quantization Loop (Column-wise) 7: for j=1 to d_{\text{in}} do 8: Get inverse Hessian diagonal element: d=[\mathbf{H}^{-1}]_{jj} 9: Get current weight column: \mathbf{w}=\mathbf{W}_{:,j} 10: Quantize: 11: \hat{\mathbf{w}}=Q(\mathbf{w}) 12: Calculate quantization error: \delta=\mathbf{w}-\hat{\mathbf{w}} 13: Update Weights: 14: \mathbf{W}_{:,j}=\hat{\mathbf{w}} {Commit quantized weights} 15: Error Compensation: Update remaining weights ( k>j ) 16: \mathbf{W}_{:,j+1:}\leftarrow\mathbf{W}_{:,j+1:}-\frac{\delta}{d}\cdot(\mathbf{H}^{-1})_{j,j+1:} 17: end for 18: return \hat{\mathbf{W}}=\mathbf{W} A-D Scenario Demonstration of \alpha We demonstrate the adaptive mechanism of QEP by analyzing the behavior of the diagnostic score s_{l} and the resulting step-size \alpha_{l} under three distinct regimes. This dynamic adjustment allows the algorithm to distinguish between genuine signal recovery and calibration overfitting. Scenario 1: Effective Calibration (Signal Dominance). In this ideal regime, the Hessian-based optimization achieves a substantial reduction in quantization error, denoted by a high reconstruction gain ( \Delta\gg 0 ). Here, the signal term dominates the instability penalty in the diagnostic score s_{l} . • Mechanism: The positive score ( s_{l}>0 ) drives the sigmoid mapping such that \alpha_{l}\rightarrow\alpha_{\max} (e.g., \alpha_{l}\approx 1.0 ). • Implication: The system identifies high confidence in the update and applies aggressive error propagation, effectively pushing the residual quantization error to subsequent layers for correction. Scenario 2: Calibration Collapse (Overfitting). This scenario occurs when the optimizer significantly adjusts weights to fit the calibration data (high instability penalty e_{\text{stab}} ) but yields negligible improvement in reconstruction (low or negative \Delta ). • Mechanism: The variance term overwhelms the signal, causing the diagnostic score to become negative ( s_{l}<0 ). Consequently, \alpha_{l}\rightarrow\alpha_{\min} (e.g., \alpha_{l}\approx 0.0 ). • Implication: The algorithm suppresses error propagation. This acts as a dampening mechanism, preventing calibration-specific noise from polluting the activations of deeper layers. Scenario 3: Benign Layer (Neutral Zone). For well-conditioned layers, standard quantization methods (e.g., RTN or GPTQ) often achieve near-zero error without complex correction. Both the reconstruction gain and the instability metrics are negligible. • Mechanism: As all diagnostic terms approach zero, s_{l}\approx 0 . The sigmoid mapping produces a neutral value \alpha_{l}\approx 0.45 . • Implication: The method naturally degenerates to a balanced propagation strategy, introducing minimal computational overhead when advanced correction is unnecessary.