JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models

Michael K. Chen, Xikun Zhang, Dacheng Tao · 2025

Read JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models by Michael K. Chen, Xikun Zhang, Dacheng Tao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14851.txt.

Introduction

Michael Chen, Xikun Zhang, and Dacheng Tao build a sterner logic test for language models. Their 2025 paper notes that older deduction benchmarks are too easy, leak world knowledge, and shrug at error patterns. Their synthetic set, JustLogic, spins varied wording and argument shapes on demand. Since facts are invented, only pure deduction can score.

Top reasoning models matched or beat the average human, yet fell well short of the best humans. Plain chat models still trailed the human average. The set also splits errors by reasoning depth and argument form, showing where models crack. Code and data are public. The authors offer JustLogic as a cleaner yardstick for machine logic.

Catalog record

CREATORMichael K. Chen, Xikun Zhang, Dacheng Tao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14851

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

JustLogic: A Comprehensive Benchmark for Evaluating Deductive Reasoning in Large Language Models Michael K. Chen, Xikun Zhang, Dacheng Tao Abstract Logical reasoning is a critical component of Large Language Models (LLMs), and substantial research efforts in recent years have aimed to enhance their deductive reasoning capabilities. However, existing deductive reasoning benchmarks, which are crucial for evaluating and advancing LLMs, are inadequate due to their lack of task complexity, presence of prior knowledge as a confounder, and superficial error analysis. To address these deficiencies, we introduce JustLogic, a synthetically generated deductive reasoning benchmark designed for rigorous evaluation of LLMs. JustLogic is (i) highly complex, capable of generating a diverse range of linguistic patterns, vocabulary, and argument structures; (ii) prior knowledge independent, eliminating the advantage of models possessing prior knowledge and ensuring that only deductive reasoning is used to answer questions; and (iii) capable of in-depth error analysis on the heterogeneous effects of reasoning depth and argument form on model accuracy. Our experimental results on JustLogic reveal that (i) state-of-the-art (SOTA) reasoning LLMs perform on par or better than the human average but significantly worse than the human ceiling, and (ii) SOTA non-reasoning models still underperform the human average. All code and data are available at https://github.com/michaelchen-lab/JustLogic 1 Introduction Deductive reasoning is a crucial capability for large language models (LLMs). It refers to the process of creating logically valid arguments, where conclusions necessarily follow from the premises. In other words, if an argument’s premises are true, the conclusion must also be true. Recent state-of-the-art (SOTA) LLMs ( Achiam et al., 2023 ; Dubey et al., 2024 ; Jiang et al., 2023 ) have exhibited outstanding performance and consistent improvement across various reasoning benchmarks, including HelloSwag ( Zellers et al., 2019 ) , ARC Challenge ( Clark et al., 2018 ) and WinoGrande ( Sakaguchi et al., 2021 ) . However, we argue that the existing benchmarks are insufficient and often ineffective for evaluating LLMs’ true deductive reasoning capabilities. Paragraph: • It is a fact that either relics are artifacts or dogs are capable of barking. [ p\lor q ] • If relics are artifacts, then fertilizers contain phosphorus. [ p\rightarrow r ] • Assuming dogs are capable of barking, we know that fertilizers contain phosphorus. [ q\rightarrow r ] Statement: Fertilizers contain phosphorus. [ r ] Question: Is the statement true, false, or uncertain? Answer: True Figure 1: Example of a question adapted using JustLogic’s dataset construction algorithm. Formal notations are included for illustrative purposes and are not provided to models. We identify three major problems with the existing benchmarks. First , they lack complexity, as measured on two dimensions: natural language complexity, i.e. how arguments are linguistically expressed, and argument complexity, i.e. the structure of the argument itself. Manually curated datasets, such as FOLIO ( Han et al., 2022 ) and LogiQA 2.0 ( Liu et al., 2020 ; Liu et al., 2023a ) exhibit high natural language complexity but low argument complexity, while synthetic datasets like CLUTRR ( Sinha et al., 2019 ) and ProofWriter ( Tafjord et al., 2020 ) exhibit the opposite. Simplicity in either dimension makes these benchmarks prone to overfitting and memorization, thus allowing models to perform well despite underlying weaknesses in logical reasoning. A more detailed analysis can be found in Section 3.4 . Second , existing benchmarks often fail to test deductive reasoning in isolation, as models can benefit from prior knowledge. To empirically validate this claim, we developed a novel test for prior knowledge independence, which measures the influence of prior knowledge on reasoning benchmarks. As detailed in Section 5.1 , prior knowledge can substantially increase accuracy, even in datasets not intended to require commonsense or domain knowledge, e.g. FOLIO and LogiQA 2.0. Thus, high accuracy may not reflect strong reasoning capabilities. Third , many existing benchmarks provide superficial error analysis, leaving key questions unanswered: At what reasoning depth does the model start to fail? How does the model compare to humans at different argument depths? Which argument forms is the model particularly weak at? These insights are essential for understanding the depth and robustness of a model’s deductive reasoning, yet many benchmarks provide them. Section 5.3 demonstrates the importance and usefulness of comprehensive error analysis. Due to these issues, LLMs’ deductive reasoning abilities remain ambiguous. In response to the critical need for a reliable benchmark to support ongoing research efforts, we present JustLogic, a novel natural language deductive reasoning benchmark. Each instance in JustLogic contains a paragraph of premises and a statement. The task is to determine whether the statement is true, false, or uncertain, based solely on the premises, and assuming they are all true. An example is shown in Figure 1 . JustLogic’s construction ensures it is (i) complex, (ii) prior knowledge independent, and (iii) capable of in-depth error analysis. First , to achieve high argument and natural language complexity, JustLogic is code-generated rather than manually curated. This allows the generation of a theoretically infinite number of unique argument structures. Natural language sentences are then drawn from GenericsKB-Best ( Bhakthavatsalam et al., 2020 ) , a database of 1M+ unique real-world sentences, and inserted into the argument structures, introducing high natural language complexity. Second , since sentences are randomly sampled from the entire GenericsKB-Best dataset, the generated arguments generally do not align with real-world knowledge, thereby eliminating prior knowledge as a confounder. Finally , in-depth error analysis is enabled by the programmatic generation process, which enables the inspection of each question’s properties, such as reasoning depth and argument form, to investigate their impact on model performance. A comparison between JustLogic and four similar logical reasoning benchmarks (CLUTRR, ProofWriter, LogiQA 2.0, and FOLIO) is presented in Table 1 , with further details on dataset construction provided in Section 3 . Table 1: Comparison of JustLogic with other deductive reasoning datasets. The symbol \sim suggests the feature is present but to a limited extent. High NL Complexity High Arg. Complexity Prior Knowledge Independence In-Depth Error Analysis CLUTRR ✗ ✓ ✓ ✓ ProofWriter ✗ ✓ ✓ \sim LogiQA 2.0 ✓ ✗ ✗ \sim FOLIO ✓ ✗ ✗ \sim JustLogic ✓ ✓ ✓ ✓ Using JustLogic, we conducted comprehensive experiments to evaluate the deductive reasoning capabilities of current LLMs. First, our novel prior knowledge independence test demonstrated that prior knowledge enables LLMs to bypass deductive reasoning on existing datasets, resulting in artificially high accuracies. This is not observed in JustLogic. Second, we benchmarked the performance of SOTA LLMs and human participants using JustLogic. Most SOTA LLMs performed significantly lower than the average human accuracy (73.0%). Only DeepSeek R1 performed substantially better (80.9%), but still fell short of the human ceiling (100.0%). Finally, enabled by JustLogic’s code-generated nature, our thorough error analysis examined the heterogeneous impact of argument structure and reasoning depth on model performance. These experiments show that the JustLogic benchmark is a reliable test of deductive reasoning and reveals significant room for improvement in LLMs. In summary, our contributions are threefold. First, we evaluate the limitations of existing benchmarks. Second, we introduce the JustLogic benchmark, a synthetic dataset to evaluate deductive reasoning, that addresses the aforementioned limitations. Third, our experiments using JustLogic demonstrate that most SOTA models perform significantly worse than humans. We

Read the complete text