Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models

Blanca Calvo Figueras, Rodrigo Agerri · 2025

Read Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models by Blanca Calvo Figueras, Rodrigo Agerri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11341.txt.

Introduction

Blanca Calvo Figueras and Rodrigo Agerri build the first large test set for machines that ask critical questions, in a 2025 paper. A critical question probes the soft spots of an argument: its hidden premises and weak links. Such questions train sharp thought in class and court. No shared dataset or scoring rule existed to grade machines at the task.

Their set holds about five thousand hand-marked questions. They also weigh how to score machine output on its own. Comparing a model's questions to gold ones tracked human judgment best. Eleven large models were then run with no examples given. Scores set a strong base line, yet the gap to good human work stayed wide. A public board invites fresh entries, and the authors hope schools will test the classroom payoff too.

Catalog record

CREATORBlanca Calvo Figueras, Rodrigo Agerri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11341

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking Critical Questions Generation: A Challenging Reasoning Task for Large Language Models Blanca Calvo Figueras, Rodrigo Agerri Abstract The task of Critical Questions Generation (CQs-Gen) aims to foster critical thinking by enabling systems to generate questions that expose underlying assumptions and challenge the validity of argumentative reasoning structures. Despite growing interest in this area, progress has been hindered by the lack of suitable datasets and automatic evaluation standards. This paper presents a comprehensive approach to support the development and benchmarking of systems for this task. We construct the first large-scale dataset including 5K manually annotated questions. We also investigate automatic evaluation methods and propose a reference-based technique using large language models (LLMs) as the strategy that best correlates with human judgments. Our zero-shot evaluation of 11 LLMs establishes a strong baseline while showcasing the difficulty of the task. Data and code 1 1 1 https://github.com/hitz-zentroa/Benchmarking_CQs-Gen plus a public leaderboard are provided to encourage further research not only in terms of model performance, but also to explore the practical benefits of CQs-Gen for both automated reasoning and human critical thinking. 1 Introduction Recent research has documented significant pedagogical concerns regarding the proliferation of Artificial Intelligence conversational interfaces. Specifically, it suggests that these systems might be reinforcing superficial learning processes while simultaneously diminishing users’ capacities for critical thinking Hadi Mogavi et al. 2024 . In this context, researchers have proposed Critical Questions Generation (CQs-Gen) as a method to leverage LLMs to automatically generate critical questions that systematically expose evidential weaknesses or structural flaws (such as fallacies) embedded within argumentative discourse. The ultimate goal is to promote deeper analytical engagement by developing systems capable of generating meaningful critical questions with respect to argumentative texts Calvo Figueras and Agerri 2024 . (a) Argument from analogy Argument: "I did not use sunscreen yesterday and I was fine, so I don’t think you need it today." Analogy CQs: Is the weather today going to be similar to the one yesterday? And our sun-exposure time? Are your skin and mine similar? Experience CQs: Has there been any other day similar to today in which I did happen to need sunscreen? (b) Argument from expert opinion Argument: "Dr. Smith says sunscreen is not necessary in May, so you don’t need it today." Source CQs: Is Dr. Smith an expert in skin care? Do other experts in skin care agree with Dr. Smith? Is Dr. Smith a trustworthy source? Might Dr. Smith be biased? Misinterpretation CQs: What were the literal words of Dr. Smith? Can his words be checked? Evidence CQs: Is his claim consistent with the known evidence about the effects of sun in May? Figure 1: Examples of two argumentation schemes and critical questions they should raise. While these are synthetic examples, the instances of our dataset are naturally occurring arguments. Critical Questions (CQs) are inquiries that may be posed to assess the acceptability of an argument. These questions are closely tied to the specific argumentation schemes underlying the construction of a given discourse Walton et al. 2008 . As illustrated in Figure 1 , an argument can be built based on different schemes, such as argument from analogy or argument from expert opinion , each triggering a different set of critical questions. Thus, the task of Critical Questions Generation is formulated as a generative task, in which systems are expected to produce questions that expose the underlying assumptions in the arguments’ premises and challenge their inferential structure. Although interest in these types of tasks is growing Musi et al. 2023 ; Favero et al. 2024 ; Ruiz-Dolz and Lawrence 2023 , the lack of datasets and standardized evaluation methods hinders the advancement of research for this particular task. In this work, we introduce the required infrastructure to address these gaps. First, we present the first large-scale dataset including 5K manually annotated critical questions. The dataset, collected from multiple sources, includes several critical questions for each text, which are labeled according to a set of validated annotation guidelines. Second, we explore various evaluation strategies for the task, measuring their correlation to human judgments. Finally, we provide a benchmarking by experimenting with 11 LLMs in zero-shot settings, analyzing the diversity of the generated questions, and providing a public leaderboard to encourage further research on Critical Questions Generation. The main contributions of this work are the following: • The first manually annotated dataset for the task of Critical Questions Generation. • An extensive investigation on the best automatic evaluation methods for this task, establishing that reference-based methods using LLMs correlate best with human judgments. • A first benchmarking of the task with 11 state-of-the-art LLMs, which showcases substantial room for improvement, accompanied by a public leaderboard to encourage further research on this particular task. 2 2 2 https://huggingface.co/spaces/HiTZ/Critical_Questions_Leaderboard In the rest of the paper, we first present relevant previous work (Section 2 ), explain the dataset creation (Section 3 ), and experimental settings for evaluation and task benchmarking (Section 4 ). We then report our experimental findings, concluding that reference-based metrics are the most suitable for this task (Section 5.1 ), which are then applied to evaluate 11 LLMs (Section 5.2 ). We finish by analyzing the results in terms of diversity and presenting our leaderboard. 2 Previous Work The automatic generation of Critical Questions is a relatively new task, introduced and motivated by Calvo Figueras and Agerri 2024 . Their work demonstrated that theoretical critical questions, such as those proposed by Walton et al. 2008 , and questions generated by LLMs are complementary, as they tend to target different aspects of argumentation. They also identified common failure modes in LLM-generated critical questions, including: (a) producing overly generic or unrelated questions, (b) introducing concepts not present in the source text, (c) flawed reasoning, and (d) generating non-critical questions, such as those resembling reading comprehension tasks. In parallel, a shared task on Critical Questions Generation has been organized to further stimulate research in this area Figueras et al. 2025 . Moreover, previous work has shown that critical questions are useful for fighting misinformation, since they help users identify fallacious reasoning Musi et al. 2023 , and also for predicting the grades of argumentative essays, since they reveal the quality of the arguments Song et al. 2014 . The potential of using Critical Questions in computational applications has been discussed at length Reed and Walton 2001 ; Macagno et al. 2017 ; Ruiz-Dolz and Lawrence 2023 . One of the big challenges of generative tasks is evaluation, and CQs-Gen is no different. Thus, machine translation researchers have long been investigating how to use human-generated references to evaluate new text generations using metrics such as BLEURT Sellam et al. 2020 , chrF Popović 2015 , and COMET Bosselut et al. 2019 . Similarity metrics such as Semantic Text Similarity (STS) Reimers and Gurevych 2019 have also been used to compare the new generations to the reference outputs Aynetdinov and Akbik 2024 . Finally, recent work has shown the effectiveness of using LLMs to evaluate various downstream tasks Zhong et al. 2022 ; Ke et al. 2022 ; Jones et al. 2024 ; Zubiaga et al. 2024 , either by using general-purpose models Wang et al. 2023 , or specialized ones Kim et

Read the complete text