Reliable Evaluation Protocol for Low-Precision Retrieval
Kisu Yang, Yoonna Jang, Hwanseok Jang, Kenneth Choi, Isabelle Augenstein, Heuiseok Lim · 2025
Read Reliable Evaluation Protocol for Low-Precision Retrieval by Kisu Yang, Yoonna Jang, Hwanseok Jang, Kenneth Choi, Isabelle Augenstein, Heuiseok Lim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03306.txt.
Introduction
Kisu Yang and co-authors fix a measurement flaw in low-precision search systems in this 2025 study. Search engines rank documents by score, and running models with fewer bits per number makes ranking faster and cheaper. But coarse numbers create many tied scores, and how ties are broken can swing the final grade of a system, making tests unreliable.
Their protocol has two parts. First, recompute the final scoring step in higher precision, which unties most near-equal documents at small cost. Second, report metrics that state the expected score, the range, and the bias caused by ties. Tests across models and data sets showed the first step sharply cut score swings. The second step recovered true expected values. Together they give low-precision search a steadier yardstick.
Catalog record
| CREATOR | Kisu Yang, Yoonna Jang, Hwanseok Jang, Kenneth Choi, Isabelle Augenstein, Heuiseok Lim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03306
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reliable Evaluation Protocol for Low-Precision Retrieval Kisu Yang, Yoonna Jang, Hwanseok Jang, Kenneth Choi, Isabelle Augenstein, Heuiseok Lim Abstract Lowering the numerical precision of model parameters and computations is widely adopted to improve the efficiency of retrieval systems. However, when computing relevance scores between the query and documents in low-precision, we observe spurious ties due to the reduced granularity. This introduces high variability in the results based on tie resolution, making the evaluation less reliable. To address this, we propose a more robust retrieval evaluation protocol designed to reduce score variation. It consists of: (1) High-Precision Scoring (HPS), which upcasts the final scoring step to higher precision to resolve tied candidates with minimal computational cost; and (2) Tie-aware Retrieval Metrics (TRM), which report expected scores, range, and bias to quantify order uncertainty of tied candidates. Our experiments test multiple models with three scoring functions on two retrieval datasets to demonstrate that HPS dramatically reduces tie-induced instability, and TRM accurately recovers expected metric values. This combination enables a more consistent and reliable evaluation system for lower-precision retrievals. 1 1 1 The source code is available at https://github.com/kisuyang/lowprec-retrieval-eval . 0 0 footnotetext: † Corresponding authors. 1 Introduction Recent studies on low-precision techniques have been widely explored (e.g., quantization and compression) to enhance the efficiency and scalability of neural networks while reducing computational cost Nagel et al. ; Kurtic et al. 2024 ; Zhu et al. 2024 ; Hao et al. 2025 . Without sacrificing performance, these methods aim to lower the numerical precision of model weights, gradients, and activations in training and inference, along with the retrieval stage Choi et al. 2024 ; Lee et al. 2025 of retrieval-augmented generation (RAG) Wang et al. 2024 ; Zhang et al. 2024 ; Zhang et al. 2025 . To generate informative responses, retrieving accurate candidates is crucial; otherwise, the following stages may be negatively affected and result in incoherent outputs Chen et al. 2024b ; Yadav et al. 2024 ; Sharma 2025 . Figure 1: Example of tie-induced instability in evaluation metric. Three documents share the same score ( G_{2} ); two of them are relevant to the query. A tie - oblivious evaluation arbitrarily breaks the tie, so the reported R@3 depends on a random internal ordering. Instead, the tie - aware formulation deterministically reports the expectation over all permutations within the tie. In neural retrieval systems, however, lowering numerical precision (e.g, FP32 to FP16 ) inevitably reduces the granularity of representable floating point numbers Shen et al. 2024 ; Hu et al. 2025 (see Appendix A ); this coarser grid produces spurious ties among candidates by forcing many distinct relevance scores to quantize to the same value. Though resolving this issue can significantly affect evaluation scores (Figure 1 ), current mainstream retrieval evaluation systems (e.g., MTEB 2 2 2 https://github.com/embeddings-benchmark/mteb Muennighoff et al. 2023 ) do not provide any principled mechanism for handling ties. Instead, they truncate the ranked list based on an arbitrary order (e.g., document IDs), which increases variances in results. Thus, we propose a reliable evaluation protocol for low-precision retrieval. It is composed of (1) High-Precision Scoring (HPS) and (2) Tie-aware Retrieval Metrics (TRM). HPS upcasts the last scoring function into higher precision, to collapse spurious ties (§ 2.2 ). TRM is an expectation-based evaluation augmented with extrema (i.e., maximum and minimum achievable scores) to quantify the order uncertainty of tied candidates (§ 2.3 ). In our experiments, we demonstrate that evaluating low-precision models using conventional tie-oblivious metrics leads to misleading outcomes as shown in Figure 1 . Adopting HPS significantly reduces score range variability, reducing MRR@10 range by 36.82%p. Meanwhile, TRM exposes biases inherent in tie-oblivious metrics, highlighting systematic overestimation by up to +9.08%p in BF16 evaluations. By contrast, our combined approach recovers near-FP32 stability and ordering, offering a consistent and discriminative framework for evaluating retrieval models in low-precision settings. 2 Reliable Evaluation Protocol We first formalize the vulnerability of the current tie-oblivious evaluation, and then present High-Precision Scoring and Tie-aware Retrieval Metric . (See Appendix A for preliminaries.) 2.1 Spurious Ties in Low-Precision Evaluation Let z denote the output of the linear layer after the last hidden state h . If the scoring function \phi is softmax or sigmoid, then the cross-encoder takes the concatenated query and i -document pair (q;d_{i}) as input and produces the logits z_{i} : two scalar values for softmax, or a single scalar value for sigmoid. If \phi is a pairwise product, z_{i} denotes the pair of embeddings (h_{q},h_{d_{i}}) obtained by encoding the query q and the document d_{i} independently with a bi - encoder. We denote the query-document relevance score \tilde{s}_{i} as: \tilde{s}_{i}=\phi^{(B)}(z_{i}) (1) where \phi^{(B)} indicates that \phi is operated entirely in a B -bit mantissa format. Applied with low-precision inference (e.g., BF16 Burgess et al. 2019 , FP16 , etc), this maps theoretically continuous values onto a discrete set of representable scores; distinct true scores may collide: \tilde{s}_{i}=\tilde{s}_{j} even with z_{i}\neq z_{j} , creating a tie . After sorting by \tilde{s} , we obtain ordered tie groups G_{n} consisting of scores s_{i} equivalent to \nu_{n} : G_{n}=\{i\mid\tilde{s}_{i}=\nu_{n}\}. (2) If the relevant document at cutoff rank k falls inside a tie group G_{n} where |G_{n}|\geq 2 , Any evaluation that disregards ties ( tie-oblivious ) may become stochastic and yield unpredictable results, as shown in Figure 1 . 2.2 High-Precision Scoring (HPS) Scoring functions such as softmax, sigmoid, and pairwise product compress logits into a narrow range. This effect is exacerbated under lower-precision formats due to fewer representable values resulting in coarser bucketization in (0,1) range (See examples in Appendix B ). For lower-precision models, HPS upcasts only the final scoring operation to FP32 , leaving other layers unchanged. Concretely we replace the low-precision scoring function (Equation 1 ) with a higher-precision scoring function: \hat{s}_{i}=\phi(\operatorname{upcast}(z_{i})), (3) and retain a more fine-grained score \hat{s}_{i} for document candidate sorting. This significantly reduces the probability of tie collisions while preserving latency, since only a small logits tensor is upcast, requiring no re-training. Advantages. HPS (i) leaves the forward pass intact and upcasts logits right before scoring, (ii) adds negligible memory and time overhead, (iii) collapses large tie groups, and (iv) restores alignment with deterministic, high-precision production sorting. Models FP32 BF16 BF16 \rightarrow FP32 ( + HPS ) \boldsymbol{M} \boldsymbol{M_{obl}} \boldsymbol{\mathbb{E}M} Range ( \blacktriangledown ) Bias ( \blacktriangledown ) \boldsymbol{M_{obl}} \boldsymbol{\mathbb{E}M} Range ( \blacktriangledown ) Bias ( \blacktriangledown ) MIRACLReranking , \boldsymbol{M=nDCG@10} Qwen3-Reranker-0.6B ♣ 73.53 75.04 68.38 25.59 6.66 73.59 73.35 1.13 0.24 bge-reranker-v2-m3 ♢ 74.61 75.59 74.54 3.90 1.05 74.63 74.57 0.16 0.06 gte-multilingual-reranker-base ♢ 74.14 74.48 74.22 0.97 0.26 74.39 74.34 0.14 0.05 Qwen3-Embedding-0.6B ♠ 63.94 64.52 63.98 1.90 0.54 64.01 64.01 0.00 0.00 multilingual-e5-large-large ♠ 64.78 65.70 64.81 4.62 0.89 64.80 64.80 0.00 0.00 MIRACLReranking , \boldsymbol{M=MRR@10} Qwen3-Reranker-0.6B ♣ 77.48 78.45 69.37 38.03 9.08 77.43 77.22 1.21 0.21 bge-reranker-v2-m3 ♢ 79.58 80.68 79.17 6.72 1.51 79.66 79.56 0.19 0.10 gte-multilingual-reranker-base ♢ 79.39 79.75 79.47 0.85 0.28 79.59 79.52 0.18 0.07 Qwen3-Embedding-0.6B ♠ 68.97 69.54 68.91 2.23 0.63 69.02 69.02 0.00 0.00 multilingual-e5-large-large ♠ 71.37 71.84 71.28 4.61 0.56 71.18 71.18 0.00 0.00 AskUbuntuDupQuestions , \boldsymbol{M=MAP@3} Qwen3-Reranker-0.6B ♣ 31.20 33.28 31.13 4.03 2.15 31.58 31.29 0.57 0.29 bge-reranker-v2-m3 ♢ 31.91 32.26 31.83