Out of Style: RAG’s Fragility to Linguistic Variation

Tianyu Cao, Neel Bhandari, Akhila Yerukola, Akari Asai, Maarten Sap · 2025

Read Out of Style: RAG’s Fragility to Linguistic Variation by Tianyu Cao, Neel Bhandari, Akhila Yerukola, Akari Asai, Maarten Sap (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08231.txt.

Description

Tianyu Cao and four co-authors present a 2025 study of how the wording of a question affects search-aided AI systems. These systems, called RAG systems, look up documents and then write an answer. The team rewrote questions along four dimensions: formality, reading level, politeness, and grammar. They then ran two search models and nine language models, of many sizes, on four question-answering data sets.

Rewording alone caused large drops in quality. Less formal questions cut search recall by up to about 40 percent, and grammar errors cut answer scores by up to about 39 percent. The full search-aided systems suffered more than models answering alone, because errors in search fed errors in writing. The authors call for design work that makes these systems steadier under real-world wording.

Catalog record

CREATORTianyu Cao, Neel Bhandari, Akhila Yerukola, Akari Asai, Maarten Sap
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE134 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08231

Browse the full library · About the Tower of Babel