Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehension
Yulong Wu, Viktor Schlegel, Riza Batista-Navarro · 2025
Read Pay Attention to Real World Perturbations! Natural Robustness Evaluation in Machine Reading Comprehension by Yulong Wu, Viktor Schlegel, Riza Batista-Navarro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16523.txt.
Description
Yulong Wu, Viktor Schlegel, and Riza Batista-Navarro test reading models against natural text edits in 2025. Machine reading systems now rival humans on clean benchmarks, but real text is messy. Synthetic typos may not reflect how real writing actually changes.
Their frame swaps benchmark passages for their real edited versions drawn from Wikipedia history. Across question sets and model types, natural edits clearly degraded scores. Even top chat models inherited the same errors. Training on perturbed examples, natural or synthetic, recovered part of the loss. A gap to clean-text scores remained, marking robustness as unfinished work.
Catalog record
| CREATOR | Yulong Wu, Viktor Schlegel, Riza Batista-Navarro |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-16523