Spark-LLM-Eval: A Distributed Framework for Statistically Rigorous Large Language Model Evaluation
Subhadip Mitra · 2026
Read Spark-LLM-Eval: A Distributed Framework for Statistically Rigorous Large Language Model Evaluation by Subhadip Mitra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28769.txt.
Description
Mitra presents Spark-LLM-Eval, a distributed framework for evaluating large language models at scale. Older tools handle thousands of cases but struggle at hundreds of thousands or millions. It runs on Apache Spark and treats eval as a data-parallel problem. Every metric includes bootstrap confidence intervals. Model pairs are compared with fit significance tests for each metric type. A content-addressable cache backed by Delta Lake avoids rerunning inference when metric definitions change. Benchmarks show linear scaling with cluster size. The code is open source.
Catalog record
| CREATOR | Subhadip Mitra |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35414 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28769