Argument-Based Comparative Question Answering Evaluation Benchmark
Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin, Chris Biemann · 2025
Read Argument-Based Comparative Question Answering Evaluation Benchmark by Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin, Chris Biemann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14476.txt.
Description
Irina Nikishina and co-authors build a yardstick for answers that compare two things, in a 2025 paper. Comparative questions, such as which phone or policy is better, need answers that argue both sides fairly. The team wrote fifteen criteria for judging such summaries, then scored answers from people and from six large language models across two datasets.
Human and model judges agreed closely under several setups. Llama-3 70B proved the best judge of summaries, while GPT-4 wrote the best comparative answers. All data and code are public. The benchmark gives builders of question-answering systems a shared way to grade a tricky answer style.
Catalog record
| CREATOR | Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin, Chris Biemann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14476