Efficient Evaluation of Large Language Models via Collaborative Filtering

Xu-Xiang Zhong, Chao Yi, Han-Jia Ye · 2025

Read Efficient Evaluation of Large Language Models via Collaborative Filtering by Xu-Xiang Zhong, Chao Yi, Han-Jia Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08781.txt.

Introduction

Xu-Xiang Zhong, Chao Yi, and Han-Jia Ye cut the cost, in a 2025 preprint, of grading large language models on big test sets. Benchmarks hold thousands of questions, and each model answers slowly. Running every model on every item burns time and money. The authors ask whether a model's full score can be estimated from a small slice of the test. They borrow a tool from shopping sites: collaborative filtering, which predicts a user's taste from other users.

Here models play the role of users and test items the role of products. Stage one picks items that best separate strong models from weak ones. Stage two predicts how the target model would do on the items it never saw. Tests show the estimates track the true scores closely. The trick slashes the number of costly model calls. Benchmark runners could screen new models far more cheaply this way.

Catalog record

CREATORXu-Xiang Zhong, Chao Yi, Han-Jia Ye
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08781

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Evaluation of Large Language Models via Collaborative Filtering Xu-Xiang Zhong, Chao Yi, Han-Jia Ye Abstract With the development of Large Language Models (LLMs), numerous benchmarks have been proposed to measure and compare the capabilities of different LLMs. However, evaluating LLMs is costly due to the large number of test instances and their slow inference speed. In this paper, we aim to explore how to efficiently estimate a model’s real performance on a given benchmark based on its evaluation results on a small number of instances sampled from the benchmark. Inspired by Collaborative Filtering (CF) in Recommendation Systems (RS), we treat LLMs as users and test instances as items and propose a two-stage method. In the first stage, we treat instance selection as recommending products to users to choose instances that can easily distinguish model performance. In the second stage, we see performance prediction as rating prediction problem in RS to predict the target LLM’s behavior on unselected instances. Experiments on multiple LLMs and datasets imply that our method can accurately estimate the target model’s performance while largely reducing its inference overhead. † † affiliationnotice: Equal contribution 1 Introduction Large Language Models (LLMs) have garnered widespread attention, with numerous LLMs ( Bai et al. 2023 ; Touvron et al. 2023a ; Touvron et al. 2023b ; Zeng et al. 2023 ) being released and rapidly iterated. Due to their powerful general capabilities, these LLMs are expected to perform a diverse and broad range of tasks ( Zhou et al. 2023 ; Pham et al. 2024 ; Gao et al. 2023 ; Qin et al. 2023 ) . To fairly assess and compare different LLMs’ capabilities, many benchmarks for evaluating LLMs have emerged and developed continuously ( Liu et al. 2024 ; Hendrycks et al. 2021 ; Li et al. 2024 ; Zhong et al. 2024 ; Liang et al. 2023 ) . To comprehensively and accurately assess the diverse capabilities of LLMs, LLM benchmarks typically contain a variety of tasks (or scenarios), with each task corresponding to a sub-dataset. Furthermore, even within a specific task, such as code generation ( Lu et al. 2021 ) , there may be multiple programming languages, and each language has its own sub-dataset. So, a well-designed benchmark often contains many test instances. Figure 1: Comparison between Methods and Problem Setting. On the left, the red dashed line represents the real performance of a new model, and the gray area indicates the gap between the estimated performance of our method and the real performance, which is smaller. On the right is the problem setup, where the goal is to extract a subset from each task, use the new model’s evaluation on it to predict performance on each task and minimize the gap between estimated and real performance. Given the large number of test instances and the relatively slow inference speed of LLMs, it is impractical to let LLMs infer on all instances to obtain their performance. For example, when selecting the best model for a target task from an LLM Zoo, evaluating all models on a benchmark would result in substantial time and computational overhead. Furthermore, inference with certain closed-source large models incurs fees, making it costly to run these models on a large volume of test samples. In summary, performing inference on all instances of the target benchmark for each LLM in a large model zoo is not viable in resource-constrained scenarios. We hope to accurately evaluate the capabilities of an LLM in one specific task at a low cost. Some previous researches ( Bommasani et al. 2021 ; Prabhu et al. 2024 ; Polo et al. 2024 ) have proposed their solutions to this problem. They only need to conduct inference on a small number of test instances to estimate the target models’ performance. However, most of the previous researches focus only on the overall performance and ranking of LLMs across the entire benchmark, without considering the task-specific performance and ranking of LLMs on individual tasks within the benchmark. Considering only overall performance cannot comprehensively and fairly evaluate the capabilities of large language models (LLMs), as different LLMs with similar overall performance and rankings may exhibit significant performance differences between tasks. And in practical applications, we often focus more on an LLM’s task-level capabilities rather than overall capabilities. Therefore, a low-cost method is urgently needed to evaluate the capabilities of large models in various tasks. To meet the above need, we propose a two-stage method inspired by Collaborative Filtering (CF) in recommendation systems ( Goldberg et al. 1992 ; Schafer et al. 2007 ; Su and Khoshgoftaar 2009 ) . We apply collaborative filtering in both stages, treating LLMs as users and instances as items. Processes in two stages focus on determining the value of items for a new user. They utilize the interaction history of other users with items and the new user’s interaction history with some items. And Figure 1 is a conceptual diagram that integrates different tasks. We can observe that our method is closer to real performance compared to other methods. In the first stage, we treat instance selection as recommending products to users. And we use CF to select a given number of instances, such as 10% for each scenario according to the importance score to get the real evaluation results of the target model. In the second stage, we view performance prediction as rating prediction problem in RS to predict the target LLM’s behavior on unselected instances. Specifically, we predict performance using CF, based on the results of similar LLMs on remaing 90% unselected instances and the results of the target LLM on selected instances and synthesized instances by optimal transport ( Peyré and Cuturi 2019 ) . Our contributions are as follows: • We propose an efficient evaluation method based on the idea of collaborative filtering, which can efficiently give the performance of LLMs on different tasks. • We analyze the similarities and differences between efficient evaluation methods and Recommendation Systems (RS), which inspire us to apply the methods of RS to address the efficient evaluating problem. • Experiments on benchmarks composed of various LLMs and tasks demonstrate the effectiveness of our method. 2 Related Work Efficient Benchmarking of PFMs. With the development of Pre-trained Foundation Models (PFMs), multiple benchmarks are introduced to quantify PFM’s abilities and compare different PFMs. However, the continuous growth in the size of models and datasets has increased evaluation costs. Some researchers focus on designing efficient benchmarks to accommodate the costs. Perlitz et al. 2024 find that while diversity across datasets is crucial for evaluation in HELM ( Liang et al. 2023 ) , the quantity of examples presently utilized is unnecessarily large. They also design a coarse-to-fine tournament algorithm to get LLM’s ranking. Vivek et al. 2024 suggest grouping evaluation samples according to LLM’s confidence in the correct class to accelerate evaluation processes for classification tasks. TinyBenchmarks ( Polo et al. 2024 ) leverages models of educational assessments from psychometrics to accurately assess the capabilities of LLMs with a fraction of the test instances in standard benchmark datasets. Lifelong Benchmarks ( Prabhu et al. 2024 ) focuses on vision models. It proposes a method called Sort&Search and this method leverages

Read the complete text