Finding Flawed Fictions : Evaluating Complex Reasoning in Language Models via Plot Hole Detection
Kabir Ahuja, Melanie Sclar, Yulia Tsvetkov · 2025
Read Finding Flawed Fictions : Evaluating Complex Reasoning in Language Models via Plot Hole Detection by Kabir Ahuja, Melanie Sclar, Yulia Tsvetkov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-11900.txt.
Introduction
A plot hole is a small crack in a story's logic, and spotting one takes real reasoning: tracking characters, rules, and cause and effect across a whole tale. The authors propose plot hole detection as a hard test for language models. Their generator algorithm inserts carefully built inconsistencies into human-written stories, and human filtering keeps the benchmark clean and resistant to contamination. Models must then find the planted flaws in stories of growing length.
Even the strongest current models struggle, and their accuracy falls steeply as stories lengthen. The study also turns the lens on the models' own writing. Machine summaries of stories carry over 50 percent more detectable plot holes than the originals, and machine-generated stories more than double the rate. The authors present the bench as a sharper measure of narrative understanding than surface comprehension tests, and as a warning about long-form generation quality.
Catalog record
| CREATOR | Kabir Ahuja, Melanie Sclar, Yulia Tsvetkov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 171 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-11900