Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective
Qingchuan Ma, Yuhang Wu, Xiawu Zheng, Rongrong Ji · 2025
Read Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective by Qingchuan Ma, Yuhang Wu, Xiawu Zheng, Rongrong Ji (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23833.txt.
Introduction
Qingchuan Ma and co-authors (2025) build a benchmark that tests whether language models truly reason in the abstract, or merely memorize surface patterns. They start from a mathematical definition: abstraction means pulling the essential pattern out of a problem regardless of how it is written, then applying rules to that pattern consistently. Two scores follow. One measures plain reasoning accuracy. The other measures how much a model's success depends on the particular symbols used, a telltale of memorization.
The test itself remaps symbols systematically inside rule-based tasks. If a model learned the rule, renaming every symbol should not hurt it. If it memorized token patterns, performance collapses. The authors ran the benchmark across many models, from commercial systems to open ones between seven and seventy billion parameters. Their framework gives the field a principled way to separate genuine abstraction from fluent mimicry.
Catalog record
| CREATOR | Qingchuan Ma, Yuhang Wu, Xiawu Zheng, Rongrong Ji |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23833
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective Qingchuan Ma, Yuhang Wu, Xiawu Zheng, Rongrong Ji Abstract In this paper, we aim to establish a simple, effective, and theoretically grounded benchmark for rigorously probing abstract reasoning in Large Language Models (LLMs). To achieve this, we first develop a mathematic framework that defines abstract reasoning as the ability to: (i) extract essential patterns independent of surface representations, and (ii) apply consistent rules to these abstract patterns. Based on this framework, we introduce two novel complementary metrics: Γ \Gamma measures basic reasoning accuracy, while Δ \Delta quantifies a model’s reliance on specific symbols rather than underlying patterns - a key indicator of true abstraction versus mere memorization. To implement this measurement, we design a benchmark: systematic symbol remapping in rule-based tasks, which forces models to demonstrate genuine pattern recognition beyond superficial token matching. Extensive LLM evaluations using this benchmark (commercial API models, 7B-70B, multi-agent) reveal:1) critical limitations in non-decimal arithmetic and symbolic reasoning; 2) persistent abstraction gaps despite chain-of-thought prompting; and 3) Δ \Delta ’s effectiveness in robustly measuring memory dependence by quantifying performance degradation under symbol remapping, particularly highlighting operand-specific memorization. These findings underscore that current LLMs, despite domain-specific strengths, still lack robust abstract reasoning, highlighting key areas for future improvement. † † affiliationnotice: Equal contribution 1 Introduction Abstract reasoning, a cornerstone of human-level intelligence ( Holyoak & Morrison, 2012 ; Penn et al., 2008 ; Holyoak, 2012 ; Chollet, 2019 ; Bober-Irizar & Banerjee, 2024 ; Xiong et al., 2024 ) , remains a critical yet elusive capability for large language models (LLMs). As philosophically defined, abstract reasoning involves two core processes: abstraction, the extraction of common features from concrete entities ( Murphy, 2004 ) , and reasoning, the derivation of new knowledge from existing information ( Holyoak & Morrison, 2012 ) . Abstraction provides the fundamental units and organizational structure for cognition, while reasoning operates and infers relationships between these abstractions. Their synergy empowers systems to understand the world and solve problems. Llama-3.3-70B deepseek v3 Qwen2.5-72B Qwen2.5-Math-7B Llama-3.1-Nemotron gemin-2.0 OM2-Llama70B gemin-1.5 gpt-4o-mini Qwen2.5-7B agentchat OM2-Llama8B Llama-3.1-8B gemma-2-9b llm debate react glm-4-9b internlm2-7b Yi-1.5-9B openmath2-Llama70B Marco-o1 0 0.2 0.4 0.6 0.8 1 Score GSM8K Our Benchmark Figure 1: Models often excel on GSM8K (blue line) but show significantly lower performance on our abstract reasoning benchmark (red line), suggesting that domain-specific math tasks may not probe deeper abstract skills. 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.1 0.2 0.3 0.4 0.5 0.6 Memory Dependence Score Average Abstract Reasoning Score 7B 32B 70B API Agents Figure 2: Memory Dependence vs. Average Score (Small Scale). The corresponding full-size version is available in Fig. 6 of the Appendix. While large language models (LLMs) such as GPT series ( Radford et al., 2019 ; Achiam et al., 2023 ) , Llama seris ( Touvron et al., 2023a ; Touvron et al., 2023b ; Dubey et al., 2024 ) , and PaLM-2 ( Anil et al., 2023 ) have exhibited remarkable achievements in various benchmarks, high performance does not necessarily imply abstract generalization . Indeed, many benchmark tasks can be tackled via pattern matching or memorized heuristics, failing to assess true reasoning. Numerous benchmarks attempt to assess reasoning, but often fall short for rigorously evaluating abstract reasoning in LLMs. ARC ( Chollet, 2019 ) ’s 2D visual format is fundamentally misaligned with LLMs’ text-based nature, hindering direct evaluation of their reasoning. Benchmarks like GSM8K ( Cobbe et al., 2021 ) and MATH ( Hendrycks et al., 2021 ) , while symbolic, risk measuring memorization of problem-solving heuristics rather than true abstract mathematical understanding. Even BIG-Bench Hard ( Suzgun et al., 2023 ) , with its broad task range, lacks a focused theoretical grounding for abstraction, potentially allowing solutions via pattern matching without genuine abstraction. These limitations underscore the urgent need for benchmarks that are both LLM-compatible and theoretically designed to isolate and measure abstract reasoning, moving beyond surface patterns and memorization ( Wu et al., 2024 ; Jiang et al., 2024 ; Mirzadeh et al., 2024 ) . Therefore, in this paper, we aim to bridge this gap by constructing a mathematically rigorous theoretical framework to formally define abstract reasoning ( Chollet, 2019 ; Morris et al., 2023 ) to formally define abstract reasoning as a composite process: first, abstraction as extracting essential patterns from concrete inputs through a mapping function, and second, reasoning as applying consistent rules to these abstract patterns to derive conclusions. Based on this framework, we introduce two complementary metrics to rigorously probe LLMs’ abstract reasoning abilities. Specifically, \Gamma measures the basic accuracy of reasoning, reflecting how well models apply rules, and \Delta quantifies a model’s dependence on specific symbols rather than underlying abstract patterns, thus serving as a crucial indicator of genuine abstraction versus mere memorization. Based on these rigorous definitions, we further derive three theorems. This theorem further motivates the core principle for our benchmark design: systematic symbol remapping in rule-based tasks , which compels models to demonstrate genuine pattern recognition beyond superficial token matching. Consequently, we design our domain-general benchmark, which, unlike domain-specific, math-centric benchmarks like GSM8K, offers a more discerning evaluation through symbolic tasks and abstract rules. This design effectively reduces reliance on memorization and domain knowledge, directly targeting genuine abstract reasoning. We then conduct extensive evaluations of various LLMs (7B-70B, multi-agent) using this benchmark, revealing key limitations and the diagnostic value of \Delta . Our contributions are threefold: (1) a theoretically grounded framework formalizing abstraction and reasoning, and introducing the \Gamma and \Delta metrics; (2) a novel symbolic benchmark with diverse tasks and remapping variants to rigorously test abstract reasoning; and (3) empirical insights from extensive LLM evaluations, revealing persistent challenges in non-decimal arithmetic, symbolic transformations, and function inference, even with Chain-of-Thought and multi-agent methods showing only partial gains. These findings underscore the need for refined benchmarks like ours to guide the development of truly abstractly reasoning AI systems and highlight key areas for future improvement. 2 Related Work Reasoning Benchmarks. Existing abstract reasoning benchmarks are often domain-specific, limiting scope and theoretical interpretability ( Chollet, 2019 ; Barrett et al., 2018 ; Mirzadeh et al., 2024 ; Frohberg & Binder, 2022 ; Majumder et al., 2024 ; Li et al., 2024a ; Yuan et al., 2023 ) . Visual benchmarks like ARC ( Chollet, 2019 ) and PGM ( Barrett et al., 2018 ) focus on 2D spatial tasks, misaligned with LLMs’ text nature. Mathematical benchmarks (e.g., GSM8K ( Cobbe et al., 2021 ) , MATH ( Hendrycks et al., 2021 ) ), while symbolic, may allow memorization of domain patterns without deep abstract reasoning. BIG-Bench Hard ( Suzgun et al., 2023 ) offers broader tasks but lacks a unified theoretical abstraction framework. Our domain-general, theoretically grounded benchmark with symbolic tasks addresses these limitations for rigorous abstract reasoning evaluation in LLMs. Multi-Agent Systems and CoT Prompting. Chain-of-thought (CoT) ( Wei et al., 2022 ) and multi-agent frameworks (ReAct ( Yao et al., 2023 ) , AutoGen ( Wu et al., 2023 ) , LLM Debate ( Du et al., 2023 ) ) aim to enhance LLM reasoning, but evaluations often lack abstract reasoning focus. While CoT/multi-agent methods improve performance on some reasoning tasks, they don’t fundamentally