GenTREC : The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systems

Mehmet Deniz Türkmen, Mucahid Kutlu, Bahadir Altun, Gokalp Cosgun · 2025

Read GenTREC : The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systems by Mehmet Deniz Türkmen, Mucahid Kutlu, Bahadir Altun, Gokalp Cosgun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02408.txt.

Description

Mehmet Deniz Turkmen, Mucahid Kutlu, Bahadir Altun, and Gokalp Cosgun build GenTREC in 2025, a search test set made in full by a language model. Hand-made relevance calls make test sets slow and dear to build. Their bet is that a text made from a prompt counts as on-topic for that prompt alone.

The set holds about 96,000 texts, 300 topics, and near 19,000 calls, with off-topic texts made to add grit. System ranks on the set track ranks on classic TREC sets for top grades. The path cuts cost for future search tests.

Catalog record

CREATORMehmet Deniz Türkmen, Mucahid Kutlu, Bahadir Altun, Gokalp Cosgun
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-02408

Browse the full library · About the Tower of Babel