FOL-Pretrain: A complexity annotated corpus of first-order logic
Isabelle Lee, Sarah Liaw, Dani Yogatama · 2025
Read FOL-Pretrain: A complexity annotated corpus of first-order logic by Isabelle Lee, Sarah Liaw, Dani Yogatama (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14932.txt.
Description
Judging whether a language model truly reasons, rather than pattern-matching, needs tests whose answers can be checked mechanically. Isabelle Lee, Sarah Liaw, and Dani Yogatama built such a testbed in this 2025 paper: FOL-Traces, a large set of first-order logic reasoning chains verified by program. Two diagnostic tasks probe whether models know the rules and can carry a proof forward.
Five reasoning models were tried, and none shone. The best reached about forty-six percent at guessing a masked operation and only about twenty-seven percent at completing two proof steps. The authors offer the dataset as a rigorous yardstick for structured inference, separating real deduction from fluent imitation.
Catalog record
| CREATOR | Isabelle Lee, Sarah Liaw, Dani Yogatama |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14932