LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models

Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz, Ying Hong Tham, Yuanyi Lin, Wenyi Fang, Fan Wu, Philipp Petersen · 2026

Read LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models by Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz, Ying Hong Tham, Yuanyi Lin, Wenyi Fang, Fan Wu, Philipp Petersen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21623.txt.

Introduction

Stanislav Budzinskiy and seven co-authors speed up language model inference with mixed numerical precision. Modern AI runs on a blend of number formats, some cheap and coarse, some costly and fine. The blend is a hallmark of current systems and helps models run locally.

The paper studies the floating-point math behind functions built from many composed parts. It focuses on transformer inference, where such compositions pile up. A look-ahead method chooses precision step by step with the road ahead in view. Smarter precision choices can cut cost without letting small errors grow into wrong answers.

Catalog record

CREATORStanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz, Ying Hong Tham, Yuanyi Lin, Wenyi Fang, Fan Wu, Philipp Petersen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21623

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz, Ying Hong Tham, Yuanyi Lin, Wenyi Fang, Fan Wu, Philipp Petersen Abstract Mixed-precision computations are a hallmark of the current stage of AI, driving the progress in large language models towards efficient, locally deployable solutions. This article addresses the floating-point computation of compositionally-rich functions, concentrating on transformer inference. Based on the rounding error analysis of a composition f ⁡ ( g ⁡ ( 𝐱 ) ) f(g(\bm{\mathrm{x}})) , we provide an adaptive strategy that selects a small subset of components of g ⁡ ( 𝐱 ) g(\bm{\mathrm{x}}) to be computed more accurately while all other computations can be carried out with lower accuracy. We then explain how this strategy can be applied to different compositions within a transformer and illustrate its overall effect on transformer inference. We study the effectiveness of this algorithm numerically on GPT-2 models and demonstrate that already very low recomputation rates allow for improvements of up to two orders of magnitude in accuracy. 1 Introduction Transformer deep neural networks (DNNs), originally introduced for sequence modeling in natural language processing ( Vaswani et al. 2017 ) , have become a standard computational paradigm across a wide range of domains, including large-scale language understanding ( Devlin et al. 2019 ) as well as vision and multimodal learning ( Dosovitskiy 2021 ) . From a computational perspective, a transformer can be viewed as a deep composition of simple operators, obtained by iterating attention-based mappings and pointwise nonlinear transformations across layers. In practice, the evaluation of such deep compositions ( inference ) is performed in floating-point (FP) arithmetic, low-precision formats being routinely employed to improve performance and energy efficiency ( Gupta et al. 2015 ; Micikevicius et al. 2018 ; Kalamkar et al. 2019 ) . From a numerical analysis standpoint, FP evaluation introduces rounding errors at every stage of the computation; the cumulative effect of these local errors hinges on how they propagate through successive compositions of operators ( Higham 2002 ) . The bulk of operators in transformers are matrix products, and the existing approaches to mixed-precision inference largely address them based on two key principles: the input is quantized to low precision, and the output is accumulated in high precision ( Xiao et al. 2023 ) . Typically, the quantization precision is uniform across the whole input, and the accumulation precision is uniform across the whole output. While there are also mixed-precision quantization strategies ( Dettmers et al. 2022 ) , we are not aware of developments in mixed-precision accumulation for transformer inference. In this work, we address this gap by proposing a mathematically principled mixed-precision inference strategy that is explicitly aware of compositional effects. Instead of treating the output of an intermediate computation uniformly, we follow a look-ahead strategy by flagging and recomputing, using higher-precision accumulation, those computations whose round-off errors will be most strongly amplified by the ensuing operator. Our method has a rigorous theoretical foundation and delivers strong empirical results in numerical experiments. We describe our contribution in detail in Subsection 1.2 . 1.1 Rounding error analysis To place our contribution in context, we briefly review how rounding error analysis has been applied to function evaluation, and why DNNs fall outside this classical setting. Rounding errors have been analyzed primarily for matrix computations ( Higham 2002 ; Connolly et al. 2021 ) and “basic” nonlinear functions such as the elementary functions : n th powers and roots, exponentials and logarithms, trigonometric and hyperbolic functions ( Muller 2016 ) . Special functions ( Gil et al. 2007 ) are “basic” in mathematical physics, defined as solutions to specific differential equations or integrals. The complexity of their evaluation stems from the need to discretize the differential equation or integral to sufficient (typically very high) precision, after which rounding effects of FP arithmetic become noticeable ( Lauter & Mezzarobba 2015 ) . DNNs, including transformers, can be considered ‘‘basic’’ in the field of AI, and they differ from the aforementioned ‘‘basic’’ functions in two aspects: the target accuracy 1 1 1 How many unit round-offs of error are tolerable, rather than the value of the unit round-off. and the source of evaluation complexity. First, the applications of DNNs do not seem to require the accuracy to be very high. Second, the evaluation complexity of DNNs is due to their extremely rich compositional structure , where the “building blocks” are either elementary functions or simple compositions of elementary functions ( Blanchard et al. 2021 ; El Arar et al. 2024 ) . Therefore, the algorithms of DNN inference need to pay specific attention to compositions . Rounding error analysis applied to DNNs leads to worst-case bounds that grow exponentially with depth ( El Arar et al. 2025 ; Budzinskiy et al. 2025 ) . While the global rounding error appears to be difficult to tame, the local rounding error at each composition is easier to control—the core idea of our work. 1.2 Contributions We propose a novel adaptive inference strategy with mixed-precision accumulation for various DNN architectures, including transformers, and establish its theoretical basis. The unit of our numerical analysis is a composition of two arbitrary functions, for which we develop a rigorous strategy to select the components of the inner function that need to be recomputed more accurately to ensure the numerical stability of the composition. The details are given in Section 2 . A theoretical investigation of the nonlinearities inherent to the transformer architecture allows us to prove that the aforementioned selection procedure can be carried out with simple greedy algorithms , overcoming the otherwise combinatorial nature of the problem, see Section 3 . The proposed method showcases highly convincing performance in numerical experiments with the GPT-2 XL model. In Figure 1 , the key-query inner products are accumulated using \mu mantissa bits, and we recompute about 8.3% of adaptively selected inner products in FP32. 2 2 2 This corresponds to \tau=1.2 in our method; see Section 2 . One illustrative observation is that BF16 accumulation ( \mu=7 ) with adaptive recomputation deviates from uniform FP32 accumulation just as much as uniform TF32 accumulation ( \mu=10 ). Meanwhile, the effective number of mantissa bits used per key-query inner product is about 8.9 for the former against 10 for the latter. 3 3 3 1\cdot 7+0.083\cdot 23=8.909 Additionally, Figure 1 shows that our method does indeed select critical-for-performance inner products, since the same number of random recomputations has no effect. Find a detailed description of our experiments and more numerical results in Section 4 . Figure 1 : Performance of the proposed adaptive mixed-precision inference strategy with \mu mantissa bits used for the low-precision accumulation of key-query inner products. The mixed precision strategy recomputes approximately 8.3% of the key-query inner products. The Kullback–Leibler divergence is measured against a reference model with uniform FP32 accumulation. 1.3 Impact and limitations We demonstrate that numerical rounding effects arising from the compositional structure of DNNs can be mitigated at inference through a theoretically-grounded compositionality-aware mixed-precision strategy. Our empirical results hint at the possibility of accumulating key-query inner products in low-bitwidth formats on GPUs, packing multiple inner products in a single 32-bit register, and recovering the prediction accuracy

Read the complete text