SuiteEval: Simplifying Retrieval Benchmarks

Andrew Parry, Debasis Ganguly, Sean MacAvaney · 2026

Read SuiteEval: Simplifying Retrieval Benchmarks by Andrew Parry, Debasis Ganguly, Sean MacAvaney (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-18107.txt.

Description

Andrew Parry and two co-authors built SuiteEval, a single tool for testing search systems the same way each time. Their 2026 paper starts from a known mess: test sets, scoring choices, and setup details vary so much that results are hard to compare. That problem bites hardest for new embedding models, which must work well on topics they were not trained on.

SuiteEval runs the whole test chain on its own, from loading data and building indexes to scoring and adding up results. It reuses stored indexes to save disk space and ships with support for five major test sets. A user only has to supply a way to build a pipeline, and a new test set can be added in one line. The goal is tests that other teams can rerun and trust.

Catalog record

CREATORAndrew Parry, Debasis Ganguly, Sean MacAvaney
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-18107

Browse the full library · About the Tower of Babel