FOL-Pretrain: A complexity annotated corpus of first-order logic

Isabelle Lee, Sarah Liaw, Dani Yogatama · 2025

Read FOL-Pretrain: A complexity annotated corpus of first-order logic by Isabelle Lee, Sarah Liaw, Dani Yogatama (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14932.txt.

Description

Judging whether a language model truly reasons, rather than pattern-matching, needs tests whose answers can be checked mechanically. Isabelle Lee, Sarah Liaw, and Dani Yogatama built such a testbed in this 2025 paper: FOL-Traces, a large set of first-order logic reasoning chains verified by program. Two diagnostic tasks probe whether models know the rules and can carry a proof forward.

Five reasoning models were tried, and none shone. The best reached about forty-six percent at guessing a masked operation and only about twenty-seven percent at completing two proof steps. The authors offer the dataset as a rigorous yardstick for structured inference, separating real deduction from fluent imitation.

Catalog record

CREATORIsabelle Lee, Sarah Liaw, Dani Yogatama
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14932

Browse the full library · About the Tower of Babel