SLR : An Automated Synthesis Framework for Scalable Logical Reasoning
Lukas Helff, Ahmad Omar, Felix Friedrich, Wolfgang Stammer, Antonia Wüst, Tim Woydt, Rupert Mitchell, Patrick Schramowski, Kristian Kersting · 2025
Read SLR : An Automated Synthesis Framework for Scalable Logical Reasoning by Lukas Helff, Ahmad Omar, Felix Friedrich, Wolfgang Stammer, Antonia Wüst, Tim Woydt, Rupert Mitchell, Patrick Schramowski, Kristian Kersting (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15787.txt.
Introduction
Testing whether language models truly reason, rather than parrot patterns, needs tasks with known right answers at many difficulty levels. Lukas Helff and co-authors present SLR, a framework that builds such tasks on its own. From a short task description it writes the prompt, a checking program that scores answers, and the hidden rule behind the task, with no human labeling.
Their benchmark holds 19,000 prompts across 20 levels of rising difficulty. Today's models often produce rules with valid form but wrong logic, and even strong reasoning models can cost over 300 dollars per thousand prompts. Training one mid-size model on the graded tasks doubled its score, matching a far costlier rival. The gains also carried over to other reasoning tests, the team reports in 2025.
Catalog record
| CREATOR | Lukas Helff, Ahmad Omar, Felix Friedrich, Wolfgang Stammer, Antonia Wüst, Tim Woydt, Rupert Mitchell, Patrick Schramowski, Kristian Kersting |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-15787
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SLR : An Automated Synthesis Framework for Scalable Logical Reasoning Lukas Helff, Ahmad Omar, Felix Friedrich, Wolfgang Stammer, Antonia Wüst, Tim Woydt, Rupert Mitchell, Patrick Schramowski, Kristian Kersting Abstract We introduce SLR , an end-to-end framework for systematic evaluation and training of Large Language Models (LLMs) via S calable L ogical R easoning. Given a user’s task specification, SLR enables scalable, automated synthesis of inductive reasoning tasks with precisely controlled difficulty. For each task, SLR synthesizes (i) a latent ground-truth rule, (ii) an executable validation program used by a symbolic judge to deterministically verify model outputs, and (iii) an instruction prompt for the reasoning task. Using SLR , we create SLR-Bench , a benchmark comprising over 19k prompts spanning 20 curriculum levels that progressively increase in relational, arithmetic, and recursive complexity. Large-scale evaluation reveals that contemporary LLMs readily produce syntactically valid rules, yet often fail at correct logical inference. Recent reasoning LLMs do somewhat better, but incur substantial increases in test-time compute, sometimes exceeding 15k completion tokens. Finally, logic-tuning via SLR doubles Llama-3-8B accuracy on SLR-Bench , achieving parity with Gemini-Flash-Thinking at a fraction of computational cost. SLR is fully automated, requires no human annotation, ensures dataset novelty, and offers a scalable environment for probing and advancing LLMs’ reasoning capabilities. 1 Introduction ⟦!htb⟧ width=⟧figs/overview/main.pdf Figure 1: Overview of the SLR Framework, including formal task specification, automated task synthesis, training, and evaluation. Left (blue): Language defines vocabulary and grammar for automated task generation; Task Config specifies hyperparameters for background and rule generation. Middle (green): The ILP Task Synthesiser automatically generates ground-truth rules, validation programs, and instruction prompts according to the defined language and config. Right (purple): Training LLMs on logic tasks, with updates provided via either cross-entropy loss for SFT or symbolic judge feedback for RL. Right (orange): Evaluating LLMs on synthesized tasks, using the symbolic judge to provide automated scoring of LLM hypotheses. Arrows denote data and control flow through task synthesis, prompting, LLM evaluation, and downstream training loops. Logical reasoning is a fundamental aspect of intelligence, yet state-of-the-art AI systems still lag in tasks requiring robust reasoning and systematic generalization delfosse2025deepreinforcementlearningagents ; Kostikova2025LLLMsAD ; Woydt2025FodorAP ; wuest2025bongard ; helff2025vlol ; sinha2019clutrr . Current benchmarks largely focus on deductive reasoning, where conclusions are necessarily entailed by premises. This includes math problems hendrycksmath2021 and logic puzzles zebralogic2025 ; xie2025logicrlunleashingllmreasoning ; liu2025synlogic . In inductive reasoning, general rules or patterns are inferred from specific examples or observations, which has been shown to be especially challenging and underexplored for LLMs luo2024logiglue ; xie2024memorization (also see Tab. 1 ). Current evaluation frameworks commonly employ constrained formats (e.g., multiple-choice) or rely on other LLMs to serve as judges patel2024aimeaioptimizationmultiple ; Lin_ZeroEval_A_Unified_2024 ; lin2024wildbench , making it difficult to assess whether models genuinely understand logical structure or are merely exploiting superficial patterns in the data. Moreover, as training sets grow, benchmark items or their paraphrases increasingly overlap with pre-training data, making apparent reasoning abilities potentially just memorization shojaee2025illusion ; xie2024memorization . This paper introduces SLR (Scalable Logical Reasoning), an open-source framework for evaluating and training models in logical reasoning. From a user-defined logic task (Fig. 1 , left), the task synthesiser (center) automatically generates novel inductive logic programming (ILP) tasks of controllable complexity. Each task comes with (i) a latent ground-truth rule, (ii) an executable validation program, and (iii) an instruction prompt for the reasoning task. The ground-truth rule serves as the reference answer, while the validation program deterministically evaluates any candidate hypothesis. SLR supports both systematic model evaluation (Fig. 1 , top right) and downstream model training, either via supervised finetuning or reinforcement learning with rewards provided by the symbolic judge (Fig. 1 , bottom right). Because both generation and validation are symbolic and automated, SLR requires no human annotation and avoids overlap with existing corpora. Leveraging SLR , we present a first instantiation, SLR-Bench (Fig. 2 ), a 19k task benchmark that forms a twenty-level curriculum of increasing logical complexity. These levels are further organized into four curriculum tiers: basic , easy , medium , and hard . Each task is unique, with a precise and systematic assessment of logical reasoning skills. Running a large-scale evaluation, we find that LLMs almost always produce syntactically valid rules, yet struggle with complex reasoning. Most models readily solve the initial levels, indicating that contemporary LLMs possess the very basic capabilities for logical inference. However, as we move towards more logically complex tasks, we observe a steady decline in model performance. Beyond evaluation, SLR enables logic-tuning, allowing Llama-3-8B trained on SLR-Bench to outperform much larger reasoning models while generating significantly fewer tokens. In sum, our contributions are: (i) SLR , an open framework for automated synthesis and symbolic evaluation of logical reasoning in LLMs. (ii) SLR-Bench , a 19k task benchmark that forms a 20-level curriculum of increasing logical complexity, enabling training and evaluating for logical reasoning across a controlled complexity spectrum. (iii) We evaluate LLMs on SLR-Bench , highlighting key insights and trade-offs. (iv) We logic-tune LLMs with SLR , boosting reasoning ability. ⟦t!⟧ Table 1: Comparison of logic reasoning benchmarks and frameworks. Reasoning Type : Principal logical inference type(s) (deduction, induction, abduction, or mix). Creation : How the dataset is created (synthetic, human-annotation, DS collection). Evaluation : Output scoring (symbolic, MCQA/Auto, LLM, exact match (EM)). Task Synthesis : Can the framework generate new tasks? Custom Tasks : Can users create custom tasks (e.g., define a new language, grammar, or task setup)? Curriculum Learning : Is a staged, fine-grained progression of difficulty/curriculum available? Scalable Difficulty : Can the complexity of generated tasks be scaled arbitrarily? ( ✓ : fully supported, ✗ : not supported, (✓) : partially/limited) Reasoning Data Evaluation Task Custom Curriculum Scalable Dataset Type Creation Methodology Synthesis Tasks Learning Complexity LogiQA liu2020logiqa Deduction Human MCQA/Auto ✗ ✗ ✗ ✗ LogiQA 2.0 Hanmeng2023logicqa2 Mix Human MCQA/Auto ✗ ✗ ✗ ✗ FOLIO han2022folio Deduction Human EM ✗ ✗ ✗ ✗ AbductionRules young2022abductionrules Abduction Synthetic/Human EM ✗ ✗ ✗ ✗ FineLogic zhou2025FineLogic Deduction DS collection Symbolic/LLM ✗ ✗ ✗ ✗ HLE phan2025humanitysexam Mix DS collection LLM ✗ ✗ ✗ ✗ Big-Bench phan2025humanitysexam Mix DS collection Mix ✗ ✗ ✗ ✗ LogiGLUE luo2024logiglue Mix DS collection MCQA/Auto ✗ ✗ ✗ ✗ Multi-LogiEval patel2024multilogieval Deduction Synthetic Symbolic ✗ ✗ ✗ ✗ CLUTRR sinha2019clutrr Induction Synthetic EM ✗ ✗ ✗ ✗ KOR-Bench ma2024korbenchbenchmarkinglanguagemodels Mix Synthetic/Human EM/Symbolic (✓) ✗ ✗ ✗ PrOntoQA PrOntoQA Deduction Synthetic EM ✓ ✗ ✗ ✗ bAbI Weston2015TowardsAQ Mix Synthetic EM ✓ ✗ ✗ ✗ SynLogic liu2025synlogic Deduction Synthetic EM ✓ ✗ ✗ ✗ FLD morishita202FLD Deduction Synthetic Symbolic ✓ ✓ ✗ (✓) K&K xie2025logicrlunleashingllmreasoning ; xie2024memorization Deduction Synthetic EM ✓ ✗ (✓) (✓) ZebraLogic zebralogic2025 Deduction Synthetic EM ✓ ✗ (✓) (✓) SLR (ours) Induction Synthetic Symbolic ✓ ✓ ✓ ✓ 2 Related Work Evaluating LLMs’ Logical Reasoning. Tab. 1 summarizes several logical reasoning benchmarks and frameworks, contrasting their goals, design choices, and trade-offs. LogiQA/2.0 liu2020logiqa ; Hanmeng2023logicqa2 and FOLIO han2022folio provide human-authored tasks testing primarily deductive reasoning (via multiple-choice or proof formats). AbductionRules young2022abductionrules focuses on abduction; models must generate plausible missing facts (explanations) that account for the observation. bAbI Weston2015TowardsAQ and CLUTRR sinha2019clutrr are synthetic datasets that consist of diverse reasoning tasks, while bAbI focuses