The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic
Dominika Długosz, Arlindo Oliveira, Natalia Díaz-Rodríguez · 2026
Read The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic by Dominika Długosz, Arlindo Oliveira, Natalia Díaz-Rodríguez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28700.txt.
Description
Dominika Długosz, Arlindo Oliveira, and Natalia Díaz-Rodríguez re-evaluate GSM-Symbolic, which reported drops across 25 models on GSM8K variants. Re-testing 20 open-weight models with mixed models with per-question effects, only eight show significant changes under the original prompts. They find the dataset's integers run larger than in GSM8K, against original claims, and controlling for this large-number effect removes significance in half the rest. Models with real deltas show distinct profiles: fragile variable binding, arithmetic limits, and dual-task interference.
Catalog record
| CREATOR | Dominika Długosz, Arlindo Oliveira, Natalia Díaz-Rodríguez |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99639 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28700