Human-Level Reasoning? A Comparative Study of Large Language Models on Logical and Abstract Reasoning
Benjamin Grando Moreira 1 · 2025
Read Human-Level Reasoning? A Comparative Study of Large Language Models on Logical and Abstract Reasoning by Benjamin Grando Moreira 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24435.txt.
Introduction
Benjamin Grando Moreira compares the logical and abstract reasoning of many large language models against human performance, working alone. Judging reasoning matters because it shows whether models truly understand information and draw valid conclusions, beyond fluent language performance. He tests models including GPT, Claude, DeepSeek, Gemini, Grok, Llama, Mistral, Perplexity, and Sabiá on eight custom reasoning questions.
The same questions were given to human participants as a benchmark. Results reveal significant differences between models and humans, with models struggling on deduction tasks that require strict logical steps. The study highlights where current models fall short of human level reasoning despite strong language skills.
Catalog record
| CREATOR | Benjamin Grando Moreira 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24435
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Human-Level Reasoning? A Comparative Study of Large Language Models on Logical and Abstract Reasoning Benjamin Grando Moreira 1 Abstract Resumo Evaluating reasoning ability in Large Language Models (LLMs) is important for advancing artificial intelligence, as it transcends mere linguistic task performance. It involves understanding whether these models truly understand information, perform inferences, and are able to draw conclusions in a logical and valid way. This study compare logical and abstract reasoning skills of several LLMs - including GPT, Claude, DeepSeek, Gemini, Grok, Llama, Mistral, Perplexity, and Sabiá - using a set of eight custom-designed reasoning questions. The LLM results are benchmarked against human performance on the same tasks, revealing significant differences and indicating areas where LLMs struggle with deduction. 1 Introduction Large Language Models (LLMs) have recently garnered significant attention as a transformative technology within the field of Artificial Intelligence, particularly due to their demonstrated proficiencies in natural language processing and text generation. Models such as GPT, BERT, and their successors have exhibited considerable efficacy across a spectrum of applications, ranging from automated translation to content synthesis and programming support. The capacity of these models to comprehend and generate text in a coherent and contextually relevant manner signifies a notable advancement, rendering them valuable assets across diverse domains [ Brown et al. 2020 , Devlin et al. 2019 ] . However, with the increasing deployment of LLMs, a critical imperative emerges to rigorously evaluate their performance in tasks demanding logical and abstract reasoning acumen. Such tasks are of fundamental importance, as they present challenges that extend beyond mere lexical association, necessitating a deeper understanding and the ability to establish logical interconnections between concepts. A comparative assessment of LLM performance in logical and abstract challenges is, therefore, essential to delineate their inherent limitations and potential capabilities, thereby informing the development of future models capable of surmounting these constraints. Despite the demonstrated capabilities of LLMs in tasks predicated on natural language processing, such as translation and text generation, their performance in tasks requiring logical and critical reasoning remains suboptimal. Prior research [ Marcus and Davis 2019 , Bender et al. 2021 ] suggests that while these models adeptly capture statistical patterns within language, they often exhibit inconsistencies in the application of logical principles. This limitation is particularly salient in scenarios where problem resolution hinges on the capacity to comprehend intricate premises and derive valid inferences therefrom. In light of these considerations, the present study undertakes a comparative analysis of the performance of several LLMs across eight distinct reasoning challenges. The primary objective is to ascertain the extent to which each model can effectively address these challenges. The significance of this analysis lies in its potential to elucidate the relative strengths and weaknesses of each model, thereby providing valuable insights for researchers and developers engaged in efforts to enhance the reasoning capabilities of LLMs. Existing literature indicates that, notwithstanding recent advancements, numerous models continue to encounter substantial difficulties in tasks necessitating intricate logical reasoning capabilities [ Marcus and Davis 2019 , Bender et al. 2021 , Malik 2024 ] . Addressing these logical challenges constitutes one of the most complex and intellectually stimulating domains within the broader field of artificial intelligence. Such challenges demand that models transcend superficial linguistic comprehension and exhibit the capacity to perform inferences, establish interrelations between concepts, and apply logical rules with precision. This inherent complexity poses a significant impediment to LLMs, which, despite their notable progress, still manifest considerable limitations when confronted with problems requiring sophisticated logical reasoning. This paper adopts a personalized assessment approach, creating its own challenges and analyzing the responses qualitatively. The paper does not use formal metrics such as Perplexity, BLEU or ROUGE. The assessment is done qualitatively, analyzing whether the LLMs’ answers are correct, whether the judgment is correct (even if the answer is wrong) or whether the answer was not answered. It is an approach that focuses more on analyzing the content of the answer than on statistical statistics. The paper also does not use existing calculation benchmarks such as Winograd Schema Challenge or ARC. Instead, a set of 8 logical reasoning challenges specific to this study is created. This allows for greater control over the type of calculation evaluated, although it is difficult to directly compare with other studies that use standardized benchmarks. The methodology employed in this study incorporates a customized assessment paradigm, wherein unique challenges were devised, and responses were subjected to qualitative analysis. In contrast to employing standardized quantitative metrics (such as Perplexity, BLEU, or ROUGE), the assessment focused on a qualitative evaluation of the LLMs’ responses, determining the correctness of answers, the validity of the reasoning employed (even in instances of incorrect answers), and instances where no response was provided. This approach prioritizes the in-depth analysis of answer content over reliance on statistical metrics. Furthermore, the study diverges from the use of established computational benchmarks like the Winograd Schema Challenge or ARC. Instead, a bespoke set of eight logical reasoning challenges, tailored specifically for this investigation, was created. This affords enhanced control over the nature of the evaluations conducted, albeit at the expense of direct comparability with studies utilizing standardized benchmarks. The evaluation framework integrates a multifaceted assessment strategy, encompassing: (1) a problem-solving paradigm wherein LLMs are tasked with resolving logical problems; (2) an analysis of response accuracy, involving verification of answer correctness; and (3) a reasoning analysis, focusing on the validity of the judgment employed to arrive at the answer, even in cases where the final response is erroneous. This approach underscores a focus on elucidating the cognitive processes underlying the models’ responses. In total, fifteen distinct LLMs were evaluated in this study, each tasked with addressing eight logical and abstract reasoning challenges. The findings of this investigation underscore the difficulties encountered by several LLMs in responding to the questions posed and reveal notable disparities in performance relative to the application of these challenges to a cohort of human participants. 2 Logical and abstract reasoning Logical and abstract reasoning are central components of intelligence, especially so-called fluid intelligence, which involves an ability to solve novel problems and think flexibly. Logical reasoning refers to the ability to deduce correct conclusions from premises using formal rules of logic. It is a core ability for human cognition and is often used as a measure of intelligence in psychometric tests. Abstract reasoning involves identifying patterns, relationships, and rules in complex or novel information, often without relying on prior knowledge [ Yu et al. 2021 ] . Logical reasoning and intelligence are closely related but distinct cognitive abilities, with strong evidence of a positive relationship between the two, especially in analytical and mathematical domains. Logical reasoning is the process of deriving conclusions from established premises using formal rules and principles to ensure the validity of inferences, and is fundamental for critical analysis, problem-solving and informed decision-making. In the context of AI, the importance of logical reasoning lies in the system’s ability to analyze data in a structured manner, formulating informed responses or decisions. Abstract reasoning involves understanding and manipulating concepts that go beyond concrete experiences or objects, allowing the individual to identify complex patterns, make generalizations and deal with subtle and symbolic relationships [ Mitchell 2021 ] . In AI, this