Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers

Zhiyuan Peng, Ting-ruen Wei, Tingyu Song, Yilun Zhao, Yi Fang · 2025

Read Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers by Zhiyuan Peng, Ting-ruen Wei, Tingyu Song, Yilun Zhao, Yi Fang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06223.txt.

Introduction

Search tools often ask a large language model to re-sort a first list of results, and the re-sorting works well but costs a lot of compute. Zhiyuan Peng and co-authors argue in their 2025 paper that the field judges that cost with the wrong yardsticks. Latency, pass counts, and token counts shift with hardware and run settings, and they hide the size of the model itself.

The team offers two yardsticks tied to FLOPs, the raw count of math steps. One scores ranking quality per petaFLOP, and the other counts queries handled per petaFLOP. A companion estimator predicts a reranker's FLOPs without running any test at all. Trials across many reranker designs then map where extra compute still buys better rankings and where it stops paying off, giving builders a fairer way to weigh speed against quality.

Catalog record

CREATORZhiyuan Peng, Ting-ruen Wei, Tingyu Song, Yilun Zhao, Yi Fang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers Zhiyuan Peng, Ting-ruen Wei, Tingyu Song, Yilun Zhao, Yi Fang Abstract Large Language Models (LLMs) have recently been applied to reranking tasks in information retrieval, achieving strong performance. However, their high computational demands often hinder practical deployment. Existing studies evaluate the efficiency of LLM-based rerankers using proxy metrics such as latency, the number of forward passes, input tokens, and output tokens. However, these metrics depend on hardware and running-time choices ( e.g., parallel or not, batch size, etc), and often fail to account for model size, making it difficult to interpret and obscuring the evaluation of the efficiency-effectiveness tradeoff. To address this issue, we propose E 2 R-FLOPs , for LLM-based rerankers: ranking metrics per PetaFLOP (RPP) for relevance per compute and queries per PetaFLOP (QPP) for hardware-agnostic throughput. Companied with the new metrics, an interpretable FLOPs estimator is built to estimate the FLOPs of an LLM-based reranker even without running any experiments. Based on the proposed metrics, we conduct comprehensive experiments to evaluate a wide range of LLM-based rerankers with different architecture, studying the efficiency-effectiveness trade-off and bringing this issue to the attention of the research community. 1 Introduction A typical search system balances efficiency and quality with a two-stage pipeline: a lightweight retriever retrieves hundreds of documents from a vast corpus, prioritizing efficiency, and then a more powerful but computationally expensive reranker refines their order. Thanks to the rapid progress of LLMs Brown et al. 2020 ; Grattafiori et al. 2024 ; Team et al. 2023 , LLM-based rerankers have achieved impressive gains in reranking metrics, such as NDCG; however, these gains often come at the cost of substantial computational expense, making them difficult to deploy at scale in production. This underscores the need for evaluation metrics that consider not only reranking quality but also computational efficiency. Existing studies evaluate the efficiency of LLM-based rerankers using proxies such as latency Jin et al. 2025 , the number of LLM calls ( i.e., forward passes) Zhuang et al. 2024 , and input and output token usage Chen et al. 2025 . However, these metrics lack computational granularity needed to distinguish differences in internal compute per token or per model call. Specifically, latency is heavily dependent on hardware and runtime choices (GPU vs. CPU, batch size, parallelism), making it an inconsistent basis for comparing algorithms across studies. The number of LLM calls ignores the model size: a single call to a 70B LLM costs orders of magnitude more compute than a call to a 3B model, yet both appear identical under this metric. Similarly, token usage overlooks the model size and is difficult to interpret as the cost of the input token and the output token can be different. Inspired by the scaling law in LLMs that studies the connection between total compute and performance Kaplan et al. 2020 , we employ floating-point operations (FLOPs) as a fundamental measure of cost for each forward pass or LLM call. The total number of FLOPs required by a model to rerank documents is a hardware-agnostic, intrinsic metric of computational work Sukthanker et al. 2024 . Building on this insight, we introduce E 2 R-FLOPs , E fficiency- E ffectiveness R eranking FLOPs for LLM-based rerankers: ranking metrics per PetaFLOP (RPP) for relevance per compute and queries per PetaFLOP (QPP) for hardware-agnostic throughput. The proposed metrics thus enable fair comparisons between methods that might utilize different LLMs, reranking algorithms, and running-time choices. Companied with the proposed metrics, an interpretable FLOPs estimator was built to estimate the FLOPs of an LLM-based reranker even without running any experiments. Based on the proposed metrics, we conduct comprehensive experiments to evaluate a wide range of LLM-based rerankers, examining the efficiency-effectiveness trade-off and drawing attention to this issue within the research community. Our key contributions include: • We derive a closed-form, interpretable formula for the FLOPs of LLM-based rerankers and provide an open-source calculator covering up-to-date models and decoding settings. • We propose two efficiency-effectiveness metrics: RPP for relevance per compute and QPP for hardware-agnostic throughput. • We conduct the first large-scale study of the efficiency–effectiveness trade-off in LLM-based rerankers, bring this issue to the attention of the research community. • All code, data, and the FLOPs estimator are publicly released for reproducible research on computationally efficient rerankingEfficiency-Effectiveness Reranking FLOPs 1 1 1 https://github.com/zhiyuanpeng/EER-FLOPs . . 2 Related Work 2.1 LLM-based Rerankers Based on how the documents are compared with each other, LLM-based rerankers can be categorized as pointwise, pairwise and listwise. Pointwise methods primarily compute the query-document relevance score by either the likelihood of generating the query conditioned on the document Ponte and Croft 2017 ; Zhuang and Zuccon 2021 ; Zhuang et al. 2021 ; Peng et al. 2024 or the normalized possibility of generating the “Yes” when prompting the LLM whether the query-document pair is relevant or not Liang et al. 2023 ; Nogueira et al. 2020 . The ranking can be easily accomplished by sorting the relevance score of each document. Pairwise methods compare the relevance of a pair of documents to a given query and output the document ID of the more relevant one. To rank a list of documents, sorting Qin et al. 2024 and sampling Gienapp et al. 2022 ; Mikhailiuk et al. 2020 methods are proposed. Sorting uses the pairwise comparison to replace the comparison operation in sorting algorithms like bubble sorting and heap sorting, while the sampling method. Soring method is more efficient to get the top-K document as it does not need to compare all the pairs. Setwise Zhuang et al. 2024 extends pairwise by comparing more than two documents in one LLM call. Listwise methods directly output a ranked list of document IDs. Most of the existing listwise methods are zero-shot Ma et al. 2023 or few-shot Sun et al. 2023 ; Ma et al. 2023 ; Pradeep et al. 2023 prompting methods. Recently, researchers Zhang et al. 2025 have resorted to adopting reinforcement learning to fine-tune LLMs to generate reasoning followed by ranked document IDs to tackle reasoning-intensive tasks like BRIGHT Su et al. 2025 . To get a full rank list, listwise methods usually adopt strategies like sliding-window Sun et al. 2023 and tournament ranking Chen et al. 2025 to get a full rank list with a limited window size. 2.2 FLOPs Calculation Several FLOPs profilers exist for deep learning models, but most are limited to standard forward passes and do not support token-level generation with KV-cache, which is essential for accurate LLM inference estimation. PyPAPI LOISON 2023 measures CPU-level FLOPs for general Python code but is not designed for PyTorch or GPU workloads. ptflops Sovrasov 2018 and fvcore facebookresearch 2019 compute FLOPs by running a model’s “forward” function, but do not support autoregressive decoding. DeepSpeed’s FLOPs profiler Rasley et al. 2020 and calflops MrYxJ 2023 both support the FLOPs of the decoding process, but they also require full forward execution. All existing tools require model execution and lack closed-form support for generation-aware FLOPs estimation. For reranking, prior studies utilize coarse FLOP estimates, e.g., double the total parameter count Shao et al. 2025 or open-source tooling Abdallah et al. 2025 , which lack interpretability regarding the specific facts that

Read the complete text