The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic

Dominika Długosz, Arlindo Oliveira, Natalia Díaz-Rodríguez · 2026

Read The Importance of Being Statistically Earnest: A Critical Re-evaluation of GSM-Symbolic by Dominika Długosz, Arlindo Oliveira, Natalia Díaz-Rodríguez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28700.txt.

Description

Dominika Długosz, Arlindo Oliveira, and Natalia Díaz-Rodríguez re-evaluate GSM-Symbolic, which reported drops across 25 models on GSM8K variants. Re-testing 20 open-weight models with mixed models with per-question effects, only eight show significant changes under the original prompts. They find the dataset's integers run larger than in GSM8K, against original claims, and controlling for this large-number effect removes significance in half the rest. Models with real deltas show distinct profiles: fragile variable binding, arithmetic limits, and dual-task interference.

Catalog record

CREATORDominika Długosz, Arlindo Oliveira, Natalia Díaz-Rodríguez
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99639
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28700

Browse the full library · About the Tower of Babel