Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehension

Yulong Wu, Viktor Schlegel, Riza Batista-Navarro · 2025

Read Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehension by Yulong Wu, Viktor Schlegel, Riza Batista-Navarro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16523.txt.

Description

Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro test reading models against natural text edits in 2025. Machine reading systems now rival humans on clean benchmarks, but real text is messy. Synthetic typos may not reflect how real writing actually changes.

Their frame swaps benchmark passages for their real edited versions drawn from Wikipedia history. Across question sets and model types, natural edits clearly degraded scores. Even top chat models inherited the same errors. Training on perturbed examples, natural or synthetic, recovered part of the loss. A gap to clean-text scores remained, marking robustness as unfinished work.

Catalog record

CREATORYulong Wu, Viktor Schlegel, Riza Batista-Navarro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-16523

Browse the full library · About the Tower of Babel