mt RAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

Yannis Katsis *, Sara Rosenthal *, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee · 2025

Read mt RAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems by Yannis Katsis *, Sara Rosenthal *, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03468.txt.

Description

Katsis and co-authors present MTRAG in 2025, a benchmark for retrieval-augmented chat systems. Such systems fetch documents before answering, and multi-turn talks add strain that single questions miss. Later turns, vague follow-ups, and unanswerable asks all trip real deployments.

The set holds one hundred ten human-written conversations across four domains, averaging near eight turns each. That yields over eight hundred tasks for testing the full pipeline. Synthetic data and model-based judging are explored as cheaper scoring paths. Even top systems struggled, above all on later turns and questions with no answer. The benchmark is public for builders to test against.

Catalog record

CREATORYannis Katsis *, Sara Rosenthal *, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-03468

Browse the full library · About the Tower of Babel