GEAR: A G eneral E valuation Framework for A bductive R easoning

Kaiyu He, Peilin Wu, Mian Zhang, Kun Wan, Wentian Zhao, Xinya Du, Zhiyu Chen · 2025

Read GEAR: A G eneral E valuation Framework for A bductive R easoning by Kaiyu He, Peilin Wu, Mian Zhang, Kun Wan, Wentian Zhao, Xinya Du, Zhiyu Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24096.txt.

Introduction

Abduction means proposing a plausible cause for what was observed, a skill that tests of language models rarely target. Kaiyu He and 6 co-authors built GEAR, an automated way to score such hypotheses with no gold answers. It grades a set of hypotheses on three points: each must fit the facts, the set must predict new cases well, and its members must differ from each other.

The team ran nine models on four benchmarks with well over a thousand problems and tens of thousands of candidate hypotheses. Clear gaps between models emerged that gold-answer tests had hidden. A training plan based on GEAR scores, paced by how fast the model learns, improved all three scores and carried over to standard benchmarks.

Catalog record

CREATORKaiyu He, Peilin Wu, Mian Zhang, Kun Wan, Wentian Zhao, Xinya Du, Zhiyu Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GEAR: A G eneral E valuation Framework for A bductive R easoning Kaiyu He, Peilin Wu, Mian Zhang, Kun Wan, Wentian Zhao, Xinya Du, Zhiyu Chen Abstract Since the advent of Large Language Models (LLMs), research has primarily focused on improving their instruction-following and deductive reasoning abilities. Yet a central question remains: can these models truly discover new knowledge, and how can we evaluate this ability? In this work, we address this gap by studying abductive reasoning—the process of generating plausible hypotheses to explain observations. We introduce G eneral E valuation for A bductive R easoning ( GEAR ), a new general-purpose, fully automated, transparent, and label-free evaluation paradigm that overcomes limitations of prior approaches. GEAR evaluates a set of hypotheses using three metrics: consistency (each hypothesis correctly explains the given observations), generalizability (consistent hypotheses make meaningful predictions on unseen inputs), and diversity (the set of hypotheses covers many distinct predictions and patterns). Built this way, GEAR is scalable (no human gold answers needed), reliable (transparent, deterministic scoring aligned with classical abduction), and open-ended (scores improve only when models produce new, plausible hypotheses, unlike existing static benchmarks that saturate once accuracy is high). Using GEAR , we conduct a fine-grained study of nine LLMs on four popular abduction benchmarks (1,500 problems), generating 50,340 candidate hypotheses. GEAR reveals model differences and insights that are obscured by prior gold-answer–based or purely human evaluations. We further propose a momentum-based curriculum training strategy that dynamically adjusts GEAR -derived training data by learning velocity: it begins with what the model learns faster and shifts toward harder objectives such as generating diverse hypotheses once the model is confident on foundational objectives (e.g., instruction following and consistency). Without gold-label supervision, this strategy improves all three GEAR objectives—consistency, generalizability, and diversity—and these gains transfer to established abductive-reasoning benchmarks. Taken together, GEAR provides a principled framework that not only evaluates abduction but also supplies label-free, scalable training signals that help LLMs produce more diverse and reliable hypotheses. Our code and data are available at: https://github.com/KaiyuHe998/GEAR-Abduction_evaluation . 1 Introduction In the current AI community, there are many competing definitions of abductive reasoning. The most widely adopted one is Harman’s view of abduction as inference to the best explanation (IBE) ( Harman 1965 ; Douven 2021 ) . Although this definition is simple and intuitive, it suffers from key limitations when applied to real-world settings, making benchmarks and evaluations built on it problematic. First , IBE does not specify what counts as “best,” and the criteria vary across contexts. In some cases, simplicity is prioritized; in others, novelty or explanatory power is preferred. As a result, IBE-based benchmarks often select a single “gold” hypothesis according to annotators’ subjective judgments, yielding heterogeneous and unreliable labels ( Zhao et al. 2023 ; Okasha 2000 ; Cabrera 2023 ) . Second , multiple plausible hypotheses typically coexist. Real-world observations can often be explained in several ways, depending on how the data are conceptualized. This limitation appears across popular abductive benchmarks such as MINI-ARC ( Kim et al. 2022 ) , ACRE ( Zhang et al. 2021 ) , List Functions ( Rule 2020 ) , and ARC-2025 ( Chollet 2019 ) . For example, in Figure 1 , the annotated “gold” hypothesis and other valid alternatives may all be well supported by sound abductive reasoning. However, existing evaluation usually excludes these alternatives by enforcing agreement with a single annotated “gold” hypothesis under a vague “best” standard. Philosophical accounts of abduction emphasize that hypotheses are often evidentially underdetermined : given finite data and background assumptions, more than one hypothesis may be well supported ( Quine 2014 ; Stanford 2023 ) , the ability to generate other plausible, well-supported hypotheses should also count as abductive success. Scientific progress relies on maintaining a diverse pool of feasible hypotheses. Although many will ultimately be falsified, they guide the design of experiments to discriminate, verify, or refute competing explanations, thereby enriching our knowledge. Even in current mature fields, credible alternative hypotheses often remain. Figure 1: Underdetermination in abductive reasoning: a logically sound, observation-consistent hypothesis need not coincide with the annotated “gold” hypothesis; both can fit the seen data yet disagree on held-out predictions. Single-gold labeling can mask the plurality of valid explanations. To address these limitations, we use Peirce’s original definition of abduction, which frames it as a more general task of generating hypotheses from given observations ( Frankfurt 1958 ; Burks 1946 ; Minnameier 2004 ; Peirce 1974 ) . Building on this foundation, we propose GEAR , a new framework for systematically evaluating abductive reasoning. GEAR is grounded in three classical criteria for good scientific hypotheses: (1) Consistency. A hypothesis must not contradict observed facts, ensuring compatibility with existing evidence. (2) Generalizability. A good hypothesis extends beyond the observed data by making testable predictions on unseen cases. In Popper’s terms, better hypotheses carry higher empirical content: they make riskier, more precise claims and thus invite more opportunities for refutation ( Popper 2005 ; Popper 2014 ) . We operationalize generalizability as the coverage of unseen inputs on which a hypothesis yields determinate predictions; larger coverage indicates greater generality. Accordingly, when two hypotheses fit the observed data, we prefer the broader (more falsifiable) one; if it withstands more severe tests, it is more strongly corroborated and more robust. (3) Diversity. A hypothesis should contribute a genuinely new perspective rather than echo existing ones, to avoid premature convergence on a single explanation. In the spirit of Chamberlin’s multiple working hypotheses and Platt’s strong inference ( Chamberlin 1965 ; Platt 1964 ) , we favor sets of hypotheses that articulate competing mechanisms testable by critical evidence. We quantify diversity with two complementary measures: \gamma -diversity , the average number of unique predictions per input across the hypotheses set (set-level variety), and \beta -diversity , the dissimilarity of prediction patterns between hypothesis pairs (dispersion). Higher diversity indicates broader causal coverage and multiple viewpoints on the observations; points of disagreement highlight decisive experiments and help accelerate scientific progress. With GEAR , we re-examine four popularly used abduction benchmarks MINI-ARC , ACRE , List Functions , and ARC-2025 across nine LLMs—spanning API-access models (GPT-o1 ( OpenAI 2024 ) , GPT-4.1-mini ( OpenAI 2025a ) , GPT-o4-mini ( OpenAI 2025b ) ) and open-source models ( Llama-3.3-70B , Llama-3.1-8B ( Grattafiori et al. 2024 ) , Qwen-2.5-72B , Qwen-2.5-7B ( Qwen et al. 2025 ) , Gemma-2-9B ( Team et al. 2024 ) , NextCoder-7B ( Aggarwal* et al. 2025 ) )—and we find that— (1), Consistency remains hard, 70B-class models produce only 20% consistent hypotheses; (2) Consistency shows no significant correlation with the size of the initial observation set; (3), Model size is weakly related to abductive diversity—larger models do not necessarily generate more diverse hypotheses; and (4)existing gold-answer evaluations overlook the underdetermination inherent to abduction, around 80% of equally plausible hypotheses are labeled incorrect, and even the “accepted” hypotheses can differ substantially. Unlike prior frameworks that depend on gold answers or human raters, GEAR is label-free and fully automated, yielding dense, scalable signals that directly train models to generate consistent, generalizable, and diverse hypothesis sets. We convert these signals into optimization targets for preference-based RL and fine-tune base models with LoRA, so

Read the complete text