Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models
Tharindu Madusanka, Ian Pratt-Hartmann, Riza Batista-Navarro1. Department of Computer Science, University of Manchester;2. Instytut Informatyki, Uniwersytet Opolski · 2025
Read Natural Language Satisfiability: Exploring the Problem Distribution and Evaluating Transformer-based Language Models by Tharindu Madusanka, Ian Pratt-Hartmann, Riza Batista-Navarro1. Department of Computer Science, University of Manchester;2. Instytut Informatyki, Uniwersytet Opolski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17153.txt.
Description
Tharindu Madusanka and two co-authors study in 2025 how well transformer language models learn to judge whether natural language statements can all be true at once. Satisfiability is the bedrock logic task beneath most reasoning benchmarks. Yet its difficulty shifts with the language fragment used, from simple forms to ones that are formally hard.
The team maps how real problem sets spread across those classes and grammar shapes. They then test how class and wording affect a model's ability to learn the inference rules. The study warns that benchmark scores can hide blind spots: a model strong on easy fragments may fail where logic gets dense. Fairer evaluation needs problems sampled with those differences in mind.
Catalog record
| CREATOR | Tharindu Madusanka, Ian Pratt-Hartmann, Riza Batista-Navarro1. Department of Computer Science, University of Manchester;2. Instytut Informatyki, Uniwersytet Opolski |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17153