Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models

Tharindu Madusanka, Ian Pratt-Hartmann, Riza Batista-Navarro1. Department of Computer Science, University of Manchester;2. Instytut Informatyki, Uniwersytet Opolski · 2025

Read Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models by Tharindu Madusanka, Ian Pratt-Hartmann, Riza Batista-Navarro1. Department of Computer Science, University of Manchester;2. Instytut Informatyki, Uniwersytet Opolski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17153.txt.

Description

Tharindu Madusanka and two co-authors study in 2025 how well transformer language models learn to judge whether natural language statements can all be true at once. Satisfiability is the bedrock logic task beneath most reasoning benchmarks. Yet its difficulty shifts with the language fragment used, from simple forms to ones that are formally hard.

The team maps how real problem sets spread across those classes and grammar shapes. They then test how class and wording affect a model's ability to learn the inference rules. The study warns that benchmark scores can hide blind spots: a model strong on easy fragments may fail where logic gets dense. Fairer evaluation needs problems sampled with those differences in mind.

Catalog record

CREATORTharindu Madusanka, Ian Pratt-Hartmann, Riza Batista-Navarro1. Department of Computer Science, University of Manchester;2. Instytut Informatyki, Uniwersytet Opolski
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17153

Browse the full library · About the Tower of Babel