Automatic mixed precision for optimizing gained time with constrained loss mean-squared-error based on model partition to sequential sub-graphs
Shmulik Markovich-Golan, Daniel Ohayon, Itay Niv, Yair Hanani · 2025
Read Automatic mixed precision for optimizing gained time with constrained loss mean-squared-error based on model partition to sequential sub-graphs by Shmulik Markovich-Golan, Daniel Ohayon, Itay Niv, Yair Hanani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13060.txt.
Introduction
Shmulik Markovich-Golan and co-authors automated mixed precision for neural nets in two thousand twenty-five. Shrinking numbers compresses models but costs accuracy. Mixed precision varies bit-width by layer. Choosing the mix by hand is slow guesswork. Their method scores each layer's sensitivity first. A Taylor expansion of the loss supplies the metric.
Error in the loss, averaged over calibration data, drives it. Forward and backward passes on few examples suffice. Sensitivity adds cleanly across layers. The optimizer then picks the fastest safe mix. Inference speeds up with bounded accuracy loss. Edge devices gain the most from the tradeoff. Post-training compression becomes a solved routine.
Catalog record
| CREATOR | Shmulik Markovich-Golan, Daniel Ohayon, Itay Niv, Yair Hanani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13060
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Automatic mixed precision for optimizing gained time with constrained loss mean-squared-error based on model partition to sequential sub-graphs Shmulik Markovich-Golan, Daniel Ohayon, Itay Niv, Yair Hanani Abstract Quantization is essential for NN ( NN ) compression, reducing model size and computational demands by using lower bit-width data types, though aggressive reduction often hampers accuracy. MP ( MP ) mitigates this tradeoff by varying the numerical precision across network layers. This study focuses on automatically selecting an optimal MP configuration within PTQ ( PTQ ) for inference. The first key contribution is a novel sensitivity metric derived from a first-order Taylor series expansion of the loss function as a function of quantization errors in weights and activations. This metric, based on the MSE ( MSE ) of the loss, is efficiently calculated per layer using high-precision forward and backward passes over a small calibration dataset. The metric is additive across layers, with low calibration memory overhead as weight optimization is unnecessary. The second contribution is an accurate hardware-aware method for predicting MP time gain by modeling it as additive for sequential sub-graphs. An algorithm partitions the model graph into sequential subgraphs, measuring time gain for each configuration using a few samples. After calibrating per-layer sensitivity and time gain, an IP ( IP ) problem is formulated to maximize time gain while keeping loss MSE below a set threshold. Memory gain and theoretical time gain based on MAC ( MAC ) operations are also considered. Rigorous experiments on the Intel Gaudi 2 accelerator validate the approach on several LLM . 1 Introduction Quantization is a key technique for compressing neural networks by converting high-precision weights and activations to lower-precision formats, significantly reducing model size and computational load ( Rokh et al. 2023 , Gholami et al. 2021 , Nagel et al. 2021 , Guo 2018 , Weng 2021 Lee et al. 2025 ). This is crucial for efficient inference on accelerators and edge devices. The main quantization approaches are QAT ( QAT ), PTQ , and QFT ( QFT ). QAT incorporates quantization during training, maintaining accuracy despite aggressive quantization (e.g., INT4) but requires substantial data and computational resources ( Jacob et al. 2018 , Krishnamoorthi 2018 ). In contrast, PTQ quantizes pre-trained models efficiently but may reduce accuracy, especially in low-precision scenarios ( Migacz 2017 , Banner et al. 2019 ). QFT combines PTQ with fine-tuning to balance accuracy and resource efficiency ( Ashkboos et al. 2024 ). MP has emerged as a key technique for optimizing NN performance across hardware platforms ( Rakka et al. 2024 ). Automatic selection of MP configuration for inference involves search-based methods (e.g., HAWQ ( HAWQ ) and OMPQ ( OMPQ )) and optimization-based approaches (e.g., DNAS ( DNAS ), HAQ ( HAQ ), ReLeQ ( ReLeQ ), AUTOQ). Optimization-based methods tackle the non-differentiability of bit-widths through reinforcement learning and hardware feedback. Pandey et al. 2023 proposed an MP algorithm for post-training scenarios, minimizing data usage and considering hardware limitations. The algorithm first measures layer sensitivity, then reduces bit-width iteratively while maintaining performance. Wu et al. 2025 introduced StruM ( StruM ), tailored for compatible hardware, while Chen et al. 2021 formulated loss minimization as a MCKP ( MCKP ), solved with a greedy search algorithm. In this work, we address the challenge of selecting an MP configuration that optimizes gained time or memory compared to the high-precision model for PTQ , while maintaining a constraint on the MSE of the loss. To solve this problem, we introduce a novel sensitivity metric derived from the MSE of the loss of the quantized model. This metric is formulated by approximating the loss error as a first-order Taylor series expansion of quantization errors from weights and activations. It is estimated using both forward and backward passes of the model at high precision with a calibration dataset. The metric predicts the MSE of the loss for arbitrary MP configurations by considering the loss error components from different quantized layers as statistically independent. The MSE of a given component from a layer is calculated as the product of its sensitivity and the MSE of the quantization error of an individual element. Although the method requires a backward pass, the additional memory requirement is minimal, mainly consisting of stored activations (since weighta optimizer is not required). We also introduce a method for predicting the empirical time gained from a MP configuration, based on the additive execution time of sequentially computed sub-graphs. The model structure is analyzed to find sequential sub-graphs, each potentially comprising multiple layers or a single layer. Time gains for all MP configurations are measured for each group, enabling the prediction of gained time for any configuration. The total time gain is estimated as the sum of the gained times per group. The performance metric can also be expressed as memory gained or theoretical gained time based on the number of MAC operations per layer. To optimize the metric while constraining the MSE of the loss, we employ IP . The paper continues as follows: Sec. 2 presents the proposed method, Sec. 3 details experimental results, and Sec. 4 discusses conclusions. Experiments include results from applying the method on various LLM and validating estimates for loss MSE and gained time. 2 Proposed method In Sec. 2.1 the problem is formulated and the solution is derived based on IP optimizing a generic objective function with constrained loss MSE per group (sequential sub-graph). Then, the sensitivity and the loss MSE are derived in Sec. 2.2 . Various performance metrics which can substitute the generic objective function are defined in Sec. 2.3 . We also discuss the motivation and method for partitioning the model graph to sequential sub-graphs for accurately assessing the time gained by MP . The method is summarized in Sec. A . 2.1 Formulation Let \mathcal{M} , \mathbf{X} and \mathbf{Y} respectively denote a NN , the input and output, such that: \displaystyle\mathbf{Y}\triangleq\mathcal{M}\left(\mathbf{X}\right) (1) and let g\left(\mathcal{M}\left(\mathbf{X}\right),\mathbf{Y}_{\textrm{true}}\right) denote the loss function where \mathbf{Y}_{\textrm{true}} represents the ground-truth target corresponding to \mathbf{Y} . The NN is composed of L linear operations, including both standard linear layers and BGEMM ( BGEMM ) layers. Assume that the underlying hardware accelerator supports F distinct numerical formats. A per-layer MP configuration \mathcal{I}^{\textrm{layer}} is defined by a set of L\times F binary indicators (one per each combination of layer and numerical format): \displaystyle\mathcal{I}^{\textrm{layer}}\triangleq\left\{i_{\ell,f}^{\textrm{layer}}\in\left\{0,1\right\}\right\}_{\ell\in\left[0,L-1\right],f\in\left[0,F-1\right]} (2) where \ell indexes the layers and f indexes the numerical formats. Each layer is assigned exactly one numerical format, enforcing the constraint \sum_{f}i_{\ell,f}^{\textrm{layer}}=1 . The numerical formats are assumed to be various floating-point representations, differentiated by their mantissa bit widths, denoted m_{f} . Now, suppose the model is partitioned into J disjoint groups of layers, \left\{V_{j}\right\}_{j=0}^{J-1} , such that layers within a group exhibit dependent performance characteristics, while different groups are independent. Define each group as the set of layer indices comprising it, i.e., V_{j}\triangleq\left\{\ell_{j,0},\ldots,\ell_{j,L_{j}-1}\right\} , where L_{j} is the number of layers in group j . Let \mathbf{Q}_{j}\in\mathbb{Z}^{L_{j}\times F^{L_{j}}} be the matrix enumerating all per-layer possible quantization configurations for group j , where each column specifies a choice of numerical formats for the group layers, and each of its elements is in the range \left[0,F-1\right] . We extend the standard per-layer