mt RAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
Yannis Katsis *, Sara Rosenthal *, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee · 2025
Read mt RAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems by Yannis Katsis *, Sara Rosenthal *, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03468.txt.
Description
Katsis and co-authors present MTRAG in 2025, a benchmark for retrieval-augmented chat systems. Such systems fetch documents before answering, and multi-turn talks add strain that single questions miss. Later turns, vague follow-ups, and unanswerable asks all trip real deployments.
The set holds one hundred ten human-written conversations across four domains, averaging near eight turns each. That yields over eight hundred tasks for testing the full pipeline. Synthetic data and model-based judging are explored as cheaper scoring paths. Even top systems struggled, above all on later turns and questions with no answer. The benchmark is public for builders to test against.
Catalog record
| CREATOR | Yannis Katsis *, Sara Rosenthal *, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03468