DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models

Tsz Ting Chung, Lemao Liu, Dit-Yan Yeung · 2025

Read DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models by Tsz Ting Chung, Lemao Liu, Dit-Yan Yeung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15587.txt.

Introduction

Tsz Ting Chung leads a team that tests how well language models reason with logic. Logic skill is a core sign of smarts, yet popular tests mix many skills at once. Such mixes can give a false read on pure logic. Tests also skew toward few languages and odd spreads of questions.

Their new test, DivLogicEval, uses plain sentences built from varied statements in surprising ways. It rests on classical logic, so each item has a firm answer. They add a scoring rule that damps bias and chance in model replies. Their runs show how much true logic each item needs. They also rank popular models on the test. The work gives a cleaner gauge of logic skill.

Catalog record

CREATORTsz Ting Chung, Lemao Liu, Dit-Yan Yeung
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15587

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models Tsz Ting Chung, Lemao Liu, Dit-Yan Yeung Abstract Logic reasoning in natural language has been recognized as an important measure of human intelligence for Large Language Models (LLMs). Popular benchmarks may entangle multiple reasoning skills and thus provide unfaithful evaluations on the logic reasoning skill. Meanwhile, existing logic reasoning benchmarks are limited in language diversity and their distributions are deviated from the distribution of an ideal logic reasoning benchmark, which may lead to biased evaluation results. This paper thereby proposes a new classical logic benchmark DivLogicEval, consisting of natural sentences composed of diverse statements in a counterintuitive way. To ensure a more reliable evaluation, we also introduce a new evaluation metric that mitigates the influence of bias and randomness inherent in LLMs. Through experiments, we demonstrate the extent to which logical reasoning is required to answer the questions in DivLogicEval and compare the performance of different LLMs in conducting logical reasoning. 1 Introduction Large Language Models (LLMs) have been evaluated across diverse reasoning capabilities, including narrative reasoning Yu et al. 2025a ; Zhou et al. 2025 , mathematical reasoning Hendrycks et al. 2021 ; Cobbe et al. 2021 , inductive reasoning Yu et al. 2025b ; Chollet 2019 , and logical reasoning Saparov et al. 2023 ; Yu et al. 2020 . Among these, logical reasoning has long been regarded as a key indicator of human intelligence, frequently employed in contexts such as academic admissions, employment screening, and civil service recruitment. Many globally recognized standardized tests, such as the LSAT, GMAT, civil service examinations, and general aptitude tests, include a substantial number of logic-based questions. Diversity Logic-centered ReClor ✓ ✗ LogiQA2 ✓ ✗ RuleTaker ✗ ✓ LogicNLI ✗ ✓ FOLIO ✗ ✓ RobustLR ✗ ✓ PrOntoQA-OOD ✗ ✓ DivLogicEval ✓ ✓ Table 1: Comparison between the proposed DivLogicEval and existing logic reasoning benchmarks. DivLogicEval demonstrates great language diversity and provides a logic-centered evaluation that isolates the impact of logical reasoning from factors like commonsense reasoning and pretraining shortcuts. Figure 1: Illustration with the construction process. Although some popular benchmarks such as ReClor Yu et al. 2020 and LogiQA Liu et al. 2020 exist for evaluating logical reasoning Yu et al. 2020 ; Liu et al. 2020 ; Liu et al. 2023a ; Huang et al. 2023 , they do not solely concentrate on logic reasoning. In fact, various reasoning (e.g., commonsense reasoning and logic reasoning) skills may entangle in these benchmarks and some other reasoning skills can make non-negligible contributions to solving the task. For example, as shown in experiments (see §3.1), by simply reducing the model’s ability to utilize logical reasoning through prompting, LLM surprisingly yields better performance on ReClor and LogiQA. Consequently, such benchmarks may overestimate the logic reasoning abilities of LLMs. Meanwhile, some efforts have been made to create datasets based on classical logic Hahn et al. 2021 ; Pi et al. 2022 ; Sanyal et al. 2022 . For example, FOLIO is equipped with human annotations of classical logic Han et al. 2022 but it only contains about two hundred instances due to annotation cost. Other datasets are synthetically generated by predefined templates over classical logic expressions Clark et al. 2020 ; Tian et al. 2021 ; Saparov and He 2023 . Unfortunately, the language variety within these benchmarks is limited due to the high expenses associated with manually annotating diverse sentences, while their predefined templates are unable to ensure sentence diversity. Therefore, they inherently suffer from an important limitation in distribution bias. As quantified in Section 3.2, for the most diverse synthetic dataset, its vocabulary size is only about two hundred, and their data distribution is largely deviated from the distribution of natural language. As a result, such test datasets may induce a biased evaluation result according to the statistic sampling principle Cochran 1977 ; Lohr 2021 . In this paper, we thereby propose a Div erse Logic al Reasoning Eval uation (DivLogicEval) benchmark, which includes a diverse evaluation dataset with a rich vocabulary. Specifically, as illustrated in Figure 1 , the key idea to constructing DivLogicEval includes two steps (§2.1): we first sample a classical logic expression, which is verifiable by an external logic solver from a predefined set of symbolic logic propositions; and then the logic expression is transformed into natural language by instantiating its variables with diverse natural sentences and concatenating them with counterintuitive sentence connectives. Moreover, we propose a new evaluation metric for DivLogicEval which posses additional benefits over the existing evaluation metrics (§2.2). Our further analysis demonstrates the effectiveness of DivLogicEval: DivLogicEval not only concentrates more on the logic reasoning skill than ReClor and LogiQA but also surpasses existing Logic -aware benchmarks in language diversity (§3), as illustrated in Table 1 . Finally, we evaluate popular LLMs include open-sourced and closed-sourced LLMs on DivLogicEval with some interesting findings (§4). 2 DivLogicEval Benchmark 2.1 Dataset Construction Overview. DivLogicEval is a multiple-choice MRC dataset for ease and effectiveness in evaluation, and it consists of three components: content, passage, and (four) options, with only one of them being correct. DivLogicEval is a synthetic dataset constructed based on verifiable propositional logic in a counterintuitive manner. The construction method comprises several steps as illustrated in Figure 1 . We first sample a propositional logic expression that can be verified by an external logic solver, thereby creating a predefined set of symbolic logic propositions. Subsequently, the logic expression is transformed into natural language by instantiating its variables with diverse natural sentences. Finally, these sentences are concatenated with connectives in a counterintuitive manner. The detailed construction process is described in the rest of this subsection. Formation of Symbolic Logical Propositions. Two to three logic atom units are sampled from a set of eight possible variables (i.e., ‘A’, ‘B’, …, ‘H’) iteratively while the probability of sampling the variables will drop in case it is being selected. The selected variables are then incorporated into the three implication rules that are commonly used in previous research to address Wang et al. 2022 ; Li et al. 2022 ; Zhao et al. 2022 or generate Clark et al. 2020 ; Sanyal et al. 2022 logical reasoning benchmarks. The sampled propositions are concatenated to form the content of the MCQA. \displaystyle((A\rightarrow B) \displaystyle\rightarrow(\neg B\rightarrow\neg A)) (1a) \displaystyle((\neg(A\land B)\rightarrow C) \displaystyle\rightarrow(\neg A\rightarrow C)) (1b) \displaystyle(((A\lor B)\rightarrow C) \displaystyle\rightarrow(A\rightarrow C)) (1c) Formation of Potential Answer Set. The potential answer set is constructed with all possible pairs of variables with inference relation as well as the single atom variable, including the consideration of the negated variables. With an external logic validator, the options set can be divided into the entailment group e and the non-entailment group n . To ensure at least two propositions in the content are necessary to derive the correct answer, the variable pairs in the entailment group e that can be directly derived from a single proposition q_{i} are filtered out. Formation of Question Types. DivLogicEval is designed with three similar question types to the renowned GMAT examination in this domain. The content part for different question types is normally as the premise, except for “Missing Premise”. 3c1e: The content fails to imply three of the options while

Read the complete text