Logic Haystacks: Probing LLMs Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding)

Damien Sileo · 2025

Read Logic Haystacks: Probing LLMs Long-Context Logical Reasoning (Without Easily Identifiable Unrelated Padding) by Damien Sileo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-17169.txt.

Description

Damien Sileo tests long-context logic in language models with a harder kind of haystack in a paper from 2025. Models now claim windows near a million tokens, but tests often hide a simple fact in padding that is easy to spot and skip. Sileo instead fills long texts with uniform, simplified English statements backed by logic rules. Distractors look much like the real evidence yet provably do not interfere with it.

The task is to retrieve evidence and detect contradictions across up to two thousand clauses. Performance crumbled early, already at about one hundred twenty-eight clauses. Sileo concludes the effective context window is far smaller than advertised. The test offers a sterner yardstick for long-document reasoning.

Catalog record

CREATORDamien Sileo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-17169

Browse the full library · About the Tower of Babel