Argument-Based Comparative Question Answering Evaluation Benchmark

Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin, Chris Biemann · 2025

Read Argument-Based Comparative Question Answering Evaluation Benchmark by Irina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin, Chris Biemann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14476.txt.

Description

Irina Nikishina and co-authors build a yardstick for answers that compare two things, in a 2025 paper. Comparative questions, such as which phone or policy is better, need answers that argue both sides fairly. The team wrote fifteen criteria for judging such summaries, then scored answers from people and from six large language models across two datasets.

Human and model judges agreed closely under several setups. Llama-3 70B proved the best judge of summaries, while GPT-4 wrote the best comparative answers. All data and code are public. The benchmark gives builders of question-answering systems a shared way to grade a tricky answer style.

Catalog record

CREATORIrina Nikishina, Saba Anwar, Nikolay Dolgov, Maria Manina, Daria Ignatenko, Viktor Moskvoretskii, Artem Shelmanov, Tim Baldwin, Chris Biemann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14476

Browse the full library · About the Tower of Babel