ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs Xinlin Li, Timothy Chou, Josh Fromm, Zichang Liu, Yunjie Pan, Christina Fragouli Abstract Post-training weight quantization is crucial for reducing the memory and inference cost of large language models (LLMs), yet pushing the average precision below 4 bits remains challenging due to highly non-uniform weight sensitivity and the lack of principled precision allocation. Existing solutions use irregular fine-grained mixed-precision with high runtime overhead or rely on heuristics or highly constrained precision allocation strategies. In this work, we propose ScaleBITS, a mixed-precision quantization framework that enables automated, fine-grained bitwidth allocation under a memory budget while preserving hardware efficiency. Guided by a new sensitivity analysis, we introduce a hardware-aligned, block-wise weight partitioning scheme, powered by bi-directional channel reordering. We formulate global bitwidth allocation as a constrained optimization problem and develop a scalable approximation to the greedy algorithm, enabling end-to-end principled allocation. Experiments show that ScaleBITS significantly improves over uniform-precision quantization (up to +36%) and outperforms state-of-the-art sensitivity-aware baselines (up to +13%) in ultra-low-bit regime, without adding runtime overhead. 1 Introduction Figure 1 : ScaleBITS yields a smooth accuracy–compression trade-off beyond the budgets supported by existing methods. The rapid scaling of large language models (LLMs) has made post-training weight-only quantization an essential technique for reducing memory footprint and inference cost. While standard 4-bit formats (e.g., INT4 and FP4) are often sufficient to preserve model quality, pushing weight quantization below 4 bits remains challenging, as accuracy degrades rapidly. Recent solutions increasingly rely on unstructured mixed-precision or complex vector quantization schemes, which improve quantization quality at the cost of additional runtime overhead. In this work, we investigate whether more principled mixed-precision allocation can better exploit simple ultra-low-bit representations, without sacrificing runtime efficiency. A key difficulty in the ultra-low-bit regime is the highly non-uniform sensitivity of model weights. While most weights tolerate aggressive quantization, errors on a small subset of critical weights can severely degrade model behavior ( 50 ; 41 ) . Existing approaches address this through either uniform or mixed-precision. Uniform-precision methods maintain a fixed bitwidth while reducing quantization error on sensitive weights through equivalent transformations ( 31 ) , error compensation ( 15 ) , adaptive rounding ( 5 ) , or improved quantization grids ( 25 ; 24 ) . Although effective at moderate precision, these methods fundamentally saturate in the ultra-low-bit regime, where the limited number of representation levels makes it difficult to balance quantization error across heterogeneous weights. Moreover, uniform-precision methods are restricted to a small set of discrete operating points (e.g., 2, 3, or 4 bits), which severely limits the granularity of achievable compression–accuracy trade-offs, especially when the gap between adjacent bitwidths is large. Mixed-precision quantization offers a more direct way to address non-uniform weight sensitivity by assigning higher precision to more important components. However, designing a practical mixed-precision solution for LLMs remains challenging due to two requirements: (i) hardware-friendly granularity, and (ii) scalable precision allocation strategy. At one extreme, coarse, layer-wise precision assignment is computationally efficient and has been explored extensively in earlier work ( 46 ; 12 ; 7 ) . However, it is inadequate for LLMs, where each layer contains millions of parameters and the sensitive weights are sparsely distributed. At the other extreme, recent fine-grained schemes ( 11 ; 25 ; 52 ) preserve a small fraction of sensitive weight elements in full precision, achieving strong accuracy but introducing irregular sparsity, index overhead and poor hardware utilization. Intermediate granularities, such as channel-grouped partitions ( 26 ; 22 ) , have been explored, but their designs inevitably trade off adaptivity for runtime regularity. Beyond partition granularity, the effectiveness of mixed-precision quantization critically depends on how precision levels are determined for each weight group. This allocation problem is challenging due to the combinatorial search space, which grows rapidly with model size, partition granularity, and the number of candidate precisions. Classic mixed-precision methods ( 4 ; 46 ; 51 ) —based on differentiable search, reinforcement learning, or evolutionary algorithms—are computationally prohibitive at LLM scale and typically require retraining or extensive evaluation. As a result, most LLM quantization pipelines rely on sensitivity-guided heuristics, manually specified precision ratios, or severely restricted search spaces ( 11 ; 25 ; 21 ; 22 ) . In this work, we revisit mixed-precision quantization from a unified optimization perspective, framing it as a discrete resource allocation problem. Under mild assumptions, this formulation admits a classic greedy solution with provable approximation guarantees. However, applying such a greedy strategy directly at the scale and granularity of modern LLMs is impractical, as it would require an excessive number of sequential evaluations to obtain the marginal gains of precision adjustments. Our key insight is that greedy decisions are driven primarily by the relative ordering of these marginal gains, which can be efficiently approximated using a lightweight sensitivity estimate computed around a progressively quantized model. Our sensitivity analysis further reveals a bi-directional structural pattern: sensitive weights are concentrated in a small subset of both output (row) and input (column) channels, forming localized high-sensitivity regions within weight matrices. This structure indicates that suitably reordered, block-wise partitions can preserve high expressivity. Building on these insights, we introduce ScaleBITS, a scalable mixed-precision quantization framework for LLMs, which achieves a continuous Pareto frontier, reaching operating points that are inaccessible to existing uniform or locally constrained mixed-precision schemes (as shown in Figure 1 ). ScaleBITS first applies a bi-directional channel reordering strategy to cluster sensitive weights into contiguous regions within each matrix. It then employs hardware-aligned, block-wise partitions that map efficiently to optimized matrix-multiplication kernels, enabling uniform execution within each block. Finally, we propose a scalable approximation to the classic greedy algorithm, allowing automated precision allocation under an arbitrary global memory budget. Our main contributions are: • A progressive-quantization sensitivity analysis that exposes both the dynamic behavior of sensitivities and a previously underexplored bi-directional structure across weight channels. • A hardware-aligned, block-wise weight partitioning strategy via bi-directional channel reordering, jointly optimizing expressiveness and runtime efficiency. • A scalable, automated precision search algorithm that enables fine-grained mixed-precision allocation for large language models without retraining. • Extensive experiments demonstrating that our framework allows even a simple round-to-nearest uniform quantizer to outperform state-of-the-art sensitivity-aware and mixed-precision methods in the ultra-low-bit regime, with no additional inference overhead. 2 Problem Formulation Notation. This paper focuses on weight-only quantization. Given a pre-trained model, we use {\bm{w}} to denote the full set of model weights and \Pi_{\bm{w}}=\{{\bm{w}}_{i}\}_{i=1}^{N} to denote a partition of {\bm{w}} , where {\bm{w}}_{i}\in\mathbb{R}^{M_{i}} is a vectorization of all weights in component i , N is the total number of components and M_{i} is the size of each. For example, {\bm{w}}_{i} may correspond to weights in a linear layer or a channel group (i.e., row/column in a weight matrix). The finest partition corresponds to M_{i}=1 , where each component reduces to a scalar weight element w_{i}\in\mathbb{R} . When the partition \Pi_{\bm{w}} is clear from context, we use {\bm{w}} in expressions for simplicity. We assume that all weights in the same component share the same precision (e.g., 2 bits per weight). We use {\bm{b}}=(b_{1},\ldots,b_{N})\in{\mathcal{B}}^{N} to denote the precision assigned to each component, where {\mathcal{B}} is the set of candidate precisions. In the most general case {\mathcal{B}}=\mathbb{Z}_{\geq 0}^{N} . We note that if b_{i}=0 , the component {\bm{w}}_{i} is effectively pruned. We use Q(\cdot,\cdot) to denote a quantization scheme (or quantizer), and Q({\bm{w}},{\bm{b}}) is the quantized model under precision {\bm{b}} . Let \ell({\bm{x}};{\bm{w}}) be the per-example loss. We denote the population loss as L({\bm{w}})=\mathbb{E}_{{\bm{x}}}[\ell({\bm{x}};{\bm{w}})] and use L_{{\mathcal{D}}}({\bm{w}}) for the empirical loss evaluated on a calibration dataset {\mathcal{D}} . 2.1 Problem Formulation Mixed-precision quantization involves two core problems: (i) designing the weight partition \Pi_{\bm{w}} and (ii) optimizing the bit allocation {\bm{b}} . The weight partition \Pi_{{\bm{w}}} determines the mixed-precision granularity and must balance quantization quality and runtime efficiency. From a quality perspective, an ideal partition groups weights with similar sensitivity, reducing sensitivity variation within each component and enabling appropriate bitwidth sharing. Finer-grained partitions are more expressive, as they allow precision to adapt to local sensitivity differences; in the extreme, per-element precision assignment offers maximum flexibility. However, unstructured partitions typically incur additional overhead in metadata storage, control flow, and memory access, preventing efficient runtime execution. The core challenge is therefore to design \Pi_{{\bm{w}}} that maximizes sensitivity clustering while maintaining compatibility with efficient hardware execution. Given a weight partition \Pi_{\bm{w}} and a total memory budget B (bits per weight), finding an optimal bit allocation {\bm{b}}^{*} can be formulated as the following integer optimization problem \displaystyle\min_{\bm{b}} \displaystyle L(Q({\bm{w}},{\bm{b}})) (1) \displaystyle\text{s.t.} \displaystyle\frac{1}{N}\sum_{i}^{N}b_{i}\leq B, \displaystyle{\displaystyle\bm{b}}\in\mathbb{Z}_{\geq 0}^{N}. Directly solving this problem is challenging due to the non-convex objective function and the discrete yet vast search space. Under some natural assumptions (see Appendix B ), it becomes optimizing a monotone submodular function over the integer lattice with a knapsack constraint, which is a well-studied NP-hard problem ( 34 ) . A classic greedy algorithm provides a simple solution with (1-1/e) approximation guarantee to the optimal {\bm{b}}^{*} ( 33 ) and this bound is tight, meaning no polynomial-time algorithm can achieve a better worst-case guarantee ( 14 ) . For completeness, we restate the classic greedy search in Algorithm 2 for mixed-precision allocation, which has been shown effective in the layer precision search for small CNNs ( 7 ) . Despite its simple concept and strong guarantee, classic greedy search is prohibitively expensive at scale. It starts from the minimal precision and iteratively increases the precision of a single weight component {\bm{w}}_{i} that produces the largest marginal decrease in loss. To reach the target budget B , the number of required evaluations (each involving forward passes over a calibration dataset) scales as O(N^{2}) . In practice, N becomes extremely large due to the combined effect of larger model size and fine partition. For example, partitioning the Gemma2-9B model into blocks of size M=64\times 64 results in approximately N\approx\frac{9\times 10^{9}}{4096}\approx 2.2\times 10^{6} . This motivates the need for scalable approximations that preserve the greedy decision structure without explicitly evaluating all marginal losses. Method MP Granularity Sensitivity Metric LLM-MQ layer 1 \begin{aligned} |g^{(w_{i})}|\,|\Delta w_{i}|\end{aligned} TACQ element \hbox to11.37pt{\vbox to11.37pt{\pgfpicture\makeatletter\hbox{\hskip 5.68657pt\lower-5.68657pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} \lxSVG@begingroup@{stroke} \lxSVG@begingroup@{fill} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width} \lx@inpgf@ignorespaces\nullfont\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin} { {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin} {{}{{{}}}{{}}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}{}{}{}{}{}{{}\lxSVG@stroke\lxSVG@drawpath@unclipped{M 7.59 0 C 7.59 4.19 4.19 7.59 0 7.59 C -4.19 7.59 -7.59 4.19 -7.59 0 C -7.59 -4.19 -4.19 -7.59 0 -7.59 C 4.19 -7.59 7.59 -4.19 7.59 0 Z M 0 0}{fill:none} \lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-2.5pt}{-3.22221pt}\lxSVG@begingroup@{transform} \pgfsys@hbox{60}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope {{{}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}\begin{aligned} \;|g^{(w_{i})}|\,|\Delta w_{i}|\,|w_{i}|\end{aligned} SqueezeLLM element 3 \begin{aligned} \;{\mathcal{F}}_{ii}\,\Delta w_{i}^{\,2}\end{aligned} SpQR, PB-LLM element 4 \begin{aligned} \;\frac{(\Delta w_{i})^{2}}{[{\bm{X}}{\bm{X}}^{\top}]^{-1}_{ii}}\end{aligned} OWQ, SlimLLM channel (group) Table 1 : Comparison of commonly used sensitivity metrics for a single weight w_{i}\in\mathbb{R} . Here {\mathcal{F}}_{ii}\approx\mathbb{E}[(\frac{\partial\ell({\bm{w}})}{\partial w_{i}})^{2}] , and {\bm{X}} is the batched input activation to the linear layer. For a large weight component {\bm{w}}_{i}\in\mathbb{R}^{M_{i}} with M_{i}>1 , sensitivity is typically aggregated by summation or averaging. Figure 2 : Weight sensitivity distribution in Gemma2-9B model. 3 Revisit Weight Sensitivity To scale the greedy mixed-precision optimization in Section 2 , we require an efficient proxy for the marginal loss changes induced by adjusting the precision. In this section, we revisit weight sensitivity from this optimization perspective, and study how it can be estimated reliably under progressive quantization. Beyond optimization, the spatial structure of the sensitivity distribution also inspires our design of hardware-aligned weight partitions. 3.1 Sensitivity Estimation The sensitivity/saliency of a weight component {\bm{w}}_{i} measures the degradation of model quality (final loss) induced by modifying that component, for example, through quantization or pruning. At fine granularity, enumerating all component-wise changes and obtaining the exact loss difference is computationally infeasible. As a result, the Taylor series is commonly used to approximate the impact of small perturbations \Delta{\bm{w}}_{i} . Most existing methods expand the loss around the full-precision model (with weights {\bm{w}} ): \displaystyle|L({\bm{w}}+\Delta{\bm{w}}_{i})-L({\bm{w}})| (2) \displaystyle\approx \displaystyle|{{\bm{g}}^{({\bm{w}}_{i})}}^{\top}\Delta{\bm{w}}_{i}+\frac{1}{2}\Delta{\bm{w}}_{i}^{\top}H^{({\bm{w}}_{i})}\Delta{\bm{w}}_{i}|, where {\bm{g}}^{({\bm{w}}_{i})}=\nabla_{{\bm{w}}_{i}}L({\bm{w}}) and H^{({\bm{w}}_{i})}=\nabla_{{\bm{w}}_{i}}^{2}L({\bm{w}}) denote the gradient and Hessian with respect to {\bm{w}}_{i} . Table 1 summarizes representative sensitivity metrics derived from this approximation. Some approaches retain only the first-order term ( 29 ; 48 ) , while others assume the pre-trained model is near a local minimum (i.e., {\bm{g}}^{({\bm{w}}_{i})}\approx{\bm{0}} ) and rely solely on the second-order information ( 25 ; 11 ; 52 ; 26 ; 22 ) . Because computing the full Hessian H^{({\bm{w}}_{i})} is intractable for LLMs, the curvature is further approximated using local activation statistics or the diagonal Fisher information matrix. These approaches implicitly assume that derivatives evaluated at the full-precision model remain informative after quantization. However, weight sensitivity is inherently dynamic: the impact of perturbing one component {\bm{w}}_{i} depends on the precision states of the entire model. This dependency becomes especially important in the ultra-low-bit regime. When most weights are quantized aggressively, the resulting model can be far from the full-precision one, and the loss landscape may change substantially. As a result, sensitivity measured at the full-precision model may no longer reflect the true effect of additional perturbation. To address this issue, we propose to estimate sensitivity with respect to a quantized reference point, which is naturally available when the model is progressively quantized (as in Algorithm 2 and 1 ). Specifically, instead of using the full-precision model, we expand the loss around the quantized model (with weights {\bm{w}}^{Q} ) and define the sensitivity of a component {\bm{w}}_{i} as s_{i}:=~|L({\bm{w}}^{Q}+\Delta{\bm{w}}_{i})-L({\bm{w}}^{Q})|\approx~|{{\bm{g}}^{({\bm{w}}^{Q}_{i})}}^{\top}\Delta{\bm{w}}_{i}| (3) where {\bm{g}}^{({\bm{w}}^{Q}_{i})}=\nabla_{{\bm{w}}^{Q}_{i}}L({\bm{w}}^{Q}) . This choice offers two advantages. First, since {\bm{w}}^{Q} is closer to the current quantized model, the local derivatives better capture the true loss behavior under quantization. Second, unlike the full-precision model, {\bm{w}}^{Q} are not at a local optimum with respect to the model loss; as a result, the first-order term dominates and can be estimated reliably without resorting to expensive second-order approximation. Figure 3 validates this design by comparing layer-wise sensitivities (where {\bm{w}}_{i} corresponds to a decoder layer) obtained from gradients evaluated at the quantized model (Eq. 3 ) versus the full-precision one ( 1 in Table 1 ). The ground-truth loss changes are computed by restoring a single layer to higher precision while keeping the rest quantized. We found that the estimate using the quantized model effectively preserves the correct ordering of layer sensitivities, although it tends to overestimate the magnitude. In contrast, the full-precision estimate fails to produce meaningful rankings. Comparison with other metrics is provided in Appendix C . Figure 3 : Estimated layer sensitivity using first-order Taylor expansion around a quantized model (top) and the full-precision model (bottom). Figure 4 : Overview of the proposed mixed-precision framework. (a) Sensitivity distribution of the original weight matrix. (b) Sensitivity distribution after bi-directional reordering of input and output channels based on sensitivity. (c) Resulting block-wise precision allocation over the reordered weight matrix. 3.2 Spatial Distribution of Sensitive Weights Having established the reliability of our sensitivity estimate at the layer level, we focus on individual weight elements ( w_{i}\in\mathbb{R} ) and study the spatial distribution of sensitive weights within each matrix. Using our sensitivity metric (Eq. 3 ), we visualize the spatial distribution of weight sensitivities in each weight matrix. Figure 2 shows examples from Gemma2-9B model, where sensitive weights concentrate in a few columns and rows (see more examples in Appendix C.2 ). This pattern can be explained by expanding the gradient term in Eq. 3 . Given a linear layer with output {\bm{y}}={\bm{W}}^{Q}{\bm{x}} , where {\bm{W}}^{Q} is the quantized weights in matrix form, the gradient of each entry W_{ij}^{Q} is calculated as g^{({\bm{W}}^{Q})}_{ij}:=\frac{\partial L}{\partial W_{ij}^{Q}}=g_{i}^{({\bm{y}})}x_{j}, (4) where g_{i}^{({\bm{y}})} is the output gradient of row i and x_{j} is the input activation of column j . Plugging this into the sensitivity metric, we have s_{ij}=|g^{({\bm{W}}^{Q})}_{ij}\Delta W_{ij}|=|g_{i}^{({\bm{y}})}|\;|x_{j}|\;|\Delta W_{ij}|, (5) which consists of a row-wise factor |g_{i}^{({\bm{y}})}| , a column-wise factor |x_{j}| and a local weight distortion |\Delta W_{ij}| . This two-dimensional pattern inspires our partition scheme presented in Section 4.1 . Note that prior work, such as SlimLLM ( 22 ) reports only column-wise clustering, likely because their sensitivity metric 4 overlooks the inter-layer dependency captured in output gradients {\bm{g}}^{({\bm{y}})} . 4 Methodology In this section, we introduce a hardware-aligned mixed-precision quantization framework that builds on the sensitivity analysis in Section 3 and addresses the two problems stated in Section 2 . As shown in Figure 4 , we first reorder the channels of weight matrices by sensitivity, clustering the most sensitive weight toward the top-left corner. We then partition the reordered weight matrices into small blocks that match efficient matrix-multiplication kernels. Finally, precision allocation among these blocks is optimized through a scalable automated search algorithm. 4.1 Hardware-aligned Weight Partition Block-wise Weight Partition. In the deployment of weight-only quantization, weights are dequantized on the fly, and matrix multiplication is performed in high precision. For efficient inference, it is desirable to fuse dequantization with matrix multiplication so that dequantized weights are consumed immediately, avoiding intermediate storage and additional memory traffic. Modern accelerators achieve high throughput by decomposing large matrix multiplications into fixed-size tiles (e.g., 64\times 64 blocks) that are executed in parallel, with each tile following a uniform and predetermined sequence of operations. Specialized units such as Tensor Cores are designed to operate most efficiently under this tiled execution structure. Motivated by this execution behavior, we adopt weight blocks as the basic unit for precision assignment. In particular, we choose the weight partition \Pi_{{\bm{w}}}=\{{\bm{w}}_{i}\}_{i=1}^{N} such that {\bm{w}}_{i}=\text{vec}({\bm{W}}_{i}) with {\bm{W}}_{i}\in\mathbb{R}^{m\times n} being a submatrix of some weight matrix in the model. Each block is assigned a single precision level, ensuring that dequantization and subsequent matrix multiplication within a tile follow a uniform execution pattern. In practice, the block size can be chosen to match the underlying matrix multiplication kernel and may be coarsened by grouping multiple tiles together, but it cannot be finer than the kernel’s computation granularity. Bi-directional Channel Reordering. As observed in Section 3 , sensitive weights tend to concentrate along a small subset of input and output channels distributed across the weight matrix. Under a block-wise layout, such a structure may be diluted unless channels with similar sensitivity are grouped together. To better capture this structure under a block-wise layout, we propose to reorder both rows (output channels) and columns (input channels) of each weight matrix based on sensitivity. Specifically, for each channel, we aggregate element-wise sensitivity using the \ell_{1} norm, which emphasizes the presence of highly sensitive weight elements within the channel rather than canceling them out. After reordering, sensitive weights concentrate toward the top-left region of the matrix, where higher precision is preferentially allocated, as shown in Figure 4 (b). This reordering is a one-time preprocessing step performed before quantization and introduces no inference-time overhead. To preserve functional equivalence, channel reordering must be applied consistently across connected layers. For example, the output channel order of one linear layer must match the input channel order of the next. Empirically, channels linked by these constraints tend to have similar sensitivity statistics, making joint reordering effective. Further details are provided in Appendix D . Although (input) channel reordering ( 30 ; 15 ) and block-wise quantization ( 23 ; 20 ) have been explored independently for other purposes, our key insight is that reordering in both directions is essential for making coarse, hardware-aligned weight partitions expressive enough to support mixed-precision allocation in large models. 4.2 Scalable Precision Search Given a fixed weight partition \Pi_{\bm{w}} , the remaining problem is to allocate per-block precisions {\bm{b}}\in\mathbb{Z}_{\geq 0}^{N} under a global bit budget B . As discussed in Section 2 , this problem admits a greedy solution with theoretical guarantees. However, directly applying the classic greedy search (Algorithm 2 ) is computationally infeasible for such fine-grained, block-wise precision allocation in large models. In this section, we construct a scalable approximation that preserves the core structure of greedy search while dramatically reducing its computational cost. The resulting scalable search procedure is summarized in Algorithm 1 . At each iteration, the classic greedy search algorithm can be viewed as solving the following optimization problem: \displaystyle\min_{{\bm{b}}^{(t)}} \displaystyle L(Q({\bm{w}},{\bm{b}}^{(t)}) (6) \displaystyle\text{s.t.} \displaystyle\frac{1}{N}\sum_{i}^{N}b_{i}^{(t)}\leq B, \displaystyle\|{\bm{b}}^{(t)}-{\bm{b}}^{(t-1)}\|_{1}\leq 1, \displaystyle\|{\bm{b}}^{(t)}-{\bm{b}}^{(t-1)}\|_{\infty}\leq 1,\hskip 9.24994pt\hskip 9.24994pt{\bm{b}}\in\mathbb{Z}_{\geq 0}^{N}. The inefficiency comes from evaluating the objective function and the single-update enforced by the second constraint. To obtain a scalable alternative, we introduce two relaxations. First, we replace the exact loss in the objective function with a first-order approximation based on sensitivity estimates. This relies on the observation that the greedy search procedure depends primarily on the relative ordering of marginal gains, rather than their exact values. As shown in Section 3, our sensitivity estimate derived from quantized gradients reliably preserves this ordering in practice. Second, we relax the feasible set to allow batched updates, in which multiple blocks may change precision simultaneously. These relaxations yield the following approximate subproblem at iteration t : \displaystyle\min_{{\bm{b}}^{(t)}} \displaystyle L^{(t-1)}-\sum_{i=1}^{N}({\bm{s}}_{\text{up}})_{i}\mathbb{I}(b_{i}^{(t)}=b_{i}^{(t-1)}+1) (7) \displaystyle+\sum_{i=1}^{N}({\bm{s}}_{\text{down}})_{i}\mathbb{I}(b_{i}^{(t)}=b_{i}^{(t-1)}-1) \displaystyle\text{s.t.} \displaystyle\frac{1}{N}\sum_{i}^{N}b_{i}^{(t)}\leq B, \displaystyle\|{\bm{b}}^{(t)}-{\bm{b}}^{(t-1)}\|_{1}\leq rN, \displaystyle\|{\bm{b}}^{(t)}-{\bm{b}}^{(t-1)}\|_{\infty}\leq 1,\hskip 9.24994pt\hskip 9.24994pt{\bm{b}}\in\mathbb{Z}_{\geq 0}^{N}. Here, ({\bm{s}}_{\text{up}})_{i} and ({\bm{s}}_{\text{down}})_{i} approximate the marginal loss change of increasing or decreasing the precision of block i, respectively, and \gamma\in(0,1] controls the batch size. Algorithm 2 implements this relaxed subproblem in iterations, incorporating several principled modifications to the classic greedy search to ensure scalability and numerical stability. Algorithm 1 Scalable Greedy Search 1: Input: weight partition \Pi_{{\bm{w}}} , bit budget B , calibration dataset {\mathcal{D}} , quantizer Q(\cdot) , update ratios \gamma_{0} and \gamma_{T} . 2: initialize b_{i}^{(0)}=\lfloor B\rfloor\ \forall i , t=0 , k=\lfloor\gamma_{0}N\rfloor 3: while k\geq\lfloor\gamma_{T}N\rfloor do 4: sample {\mathcal{D}}^{(t)}\sim{\mathcal{D}} 5: update {\bm{s}}_{\text{up}}^{(t)} and {\bm{s}}_{\text{down}}^{(t)} on {\mathcal{D}}^{(t)} 6: if \frac{1}{N}\sum_{i=1}^{N}b_{i}^{(t)}L_{{\mathcal{D}}^{(t)}}(Q({\bm{w}},{\bm{b}}^{(t)})) then 12: {\bm{b}}^{(t+1)}\leftarrow{\bm{b}}^{(t)},k\leftarrow k/2 13: end if 14: end if 15: t\leftarrow t+1 16: end while 17: Return {\bm{b}}^{(t)} Two-stage batched updates. The optimal solution of the problem ( 7 ) admits two update rules, depending on whether the bit budget is active. If the current allocation under-utilizes the budget, i.e., \frac{1}{N}\sum_{i}b_{i}^{(t)}