Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration

Ante Wang, Yujie Lin, Jingyao Liu, Suhang Wu, Hao Liu, Xinyan Xiao, Jinsong Su · 2025

Read Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration by Ante Wang, Yujie Lin, Jingyao Liu, Suhang Wu, Hao Liu, Xinyan Xiao, Jinsong Su (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23407.txt.

Introduction

Ante Wang and co-authors teach AI models to ask for missing facts in a 2025 paper. Sound judgment means not just spotting bad inputs, but asking what would fix them. Most past work stops at refusal, which leaves the user's task stuck.

The team built two test sets from a school math bank. In one, each problem drops a key value, so the model must spot and request it. The other adds decoy details to test focus. Top models did well on plain math yet flopped at asking, above all small ones. Extra training by reward lifted one small model from near zero to strong scores. Asking well is a trainable skill.

Catalog record

CREATORAnte Wang, Yujie Lin, Jingyao Liu, Suhang Wu, Hao Liu, Xinyan Xiao, Jinsong Su
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23407

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration Ante Wang, Yujie Lin, Jingyao Liu, Suhang Wu, Hao Liu, Xinyan Xiao, Jinsong Su Abstract Critical thinking is essential for building robust AI systems, preventing them from blindly accepting flawed data or biased reasoning. However, prior work has primarily focused on passive critical thinking, where models simply reject problematic queries without taking constructive steps to address user requests. In this work, we introduce proactive critical thinking, a paradigm where models actively seek missing or clarifying information from users to resolve their queries better. To evaluate this capability, we present GSM-MC and GSM-MCE, two novel benchmarks based on GSM8K for assessing mathematical reasoning under incomplete or misleading conditions. GSM-MC contains 1,368 math problems with a key variable deliberately removed, requiring models to identify and request the missing information. GSM-MCE further increases the difficulty by introducing irrelevant details to test robustness against distractions. Experiments on Qwen3 and Llama series models show that, while these models excel in traditional reasoning tasks due to extensive post-training and inference-time scaling, they struggle with proactive critical thinking, especially smaller ones. However, we demonstrate that reinforcement learning (RL) can significantly improve this ability. Using our enhanced RL algorithm, we achieve substantial gains, boosting the Qwen3-1.7B’s accuracy from 0.15% to 73.98% on GSM-MC. We hope this work advances models that collaborate more effectively with users in problem-solving through proactive critical thinking. 1 Introduction Large Language Models (LLMs) have made significant strides in solving complex tasks, such as mathematical reasoning Yamauchi et al. 2023 ; IMANI et al. 2024 ; Wang et al. 2025a , code generation Feng et al. 2023 ; Dong et al. 2024 ; Shao et al. 2025 ; Wang et al. 2025b , and planning Yao et al. 2023 ; Zhou et al. 2024 ; Lin et al. 2025 . However, most existing studies focus on controlled settings where user queries are always answerable. In real-world applications, users, especially those without domain expertise, often provide insufficient or inaccurate information, making it difficult for models to solve problems effectively Zamfirescu-Pereira et al. 2023 ; Kim et al. 2024 . For instance, a patient lacking medical knowledge might omit critical symptoms, preventing an AI doctor from making a precise diagnosis Alkaabi and Elsori 2025 . Some prior work Rahman et al. 2024 ; Kirichenko et al. 2025 has acknowledged this issue, advocating for critical thinking in LLMs, which refers to the ability to reject unanswerable or flawed requests instead of attempting to process biased or incomplete inputs. Yet, we argue that this form of critical thinking remains passive , as it still relies on users to independently identify and rectify gaps in their queries, rather than actively facilitating problem-solving. Figure 1: When faced with flawed questions, Left: Vanilla reasoning models fall into repetitive speculation and eventual hallucination. Despite identifying the missing conditions during the thinking process, they still attempt to force an answer, ultimately providing an incorrect solution. Middle: Models with passive critical thinking are more cautious, refusing to answer upon identifying flaws in the question. However, this interaction remains suboptimal, as it shifts the burden to the user to recognize and fix the errors. Right: Our trained models, equipped with proactive critical thinking, actively seek clarification by offering precise feedback, greatly reducing the user effort. To address this limitation, we propose proactive critical thinking : a paradigm where the model not only detects unanswerable queries but also provides constructive feedback to guide users in supplying necessary information. As shown in Figure 1 , this approach fosters more effective human-AI collaboration, enabling iterative conversations that progressively refine the problem and lead to a solution. Previous research Ma et al. 2024 ; Fan et al. 2025 has established some benchmarks for evaluating passive critical thinking by performing operations such as removing or replacing key information in original questions using advanced LLMs. However, these datasets often lack rigorous quality control, resulting in many noisy and overly simplistic cases (see Table 1 ) that are unsuitable for evaluating proactive critical thinking. To tackle this, we develop an automated data preparation pipeline using DeepSeek-V3 Liu et al. 2024 . Following previous studies Ma et al. 2024 ; Sun et al. 2024 , we focus on the mathematical reasoning task and derive our datasets from the widely used GSM8K dataset Cobbe et al. 2021 . We first identify the key variables in the original questions. Then, we randomly remove one variable and rephrase the resulting question for fluency. This approach effectively mitigates the bias introduced in previous work, where LLMs are directly instructed to remove information from questions. Nevertheless, this process still generates many low-quality cases, similar to those in prior studies. To ensure high quality, we apply strict filtering strategies and retain only those cases that can be resolved through proactive critical thinking using DeepSeek-V3. This results in a final test set of 1,368 high-quality cases, which we named GSM-MC. To further enhance the challenge and evaluate models’ robustness against distractions, we augment these questions by injecting irrelevant information. Then the new dataset, designated GSM-MCE, undergoes the same rigorous filtering to maintain its high quality. To instill proactive critical thinking capabilities into models, we investigate both supervised fine-tuning (SFT) and reinforcement learning (RL). Using the data preparation pipeline described above, we combine revised unanswerable questions with original questions to construct the training set. Furthermore, we enhance both training paradigms by incorporating a heuristic information about question answerability. This approach effectively increases the diversity of the SFT data and accelerates the RL convergence through more dense reward signals. We evaluate popular Qwen3 Yang et al. 2025 and Llama Meta 2024 series models on our GSM-MC and GSM-MCE benchmarks. The results reveal that despite extensive post-training, these models still struggle with proactive critical thinking, particularly the smaller ones. Notably, while recent inference-time scaling approaches have significantly advanced performance on complex reasoning tasks, we find they can hinder proactive critical thinking capability. However, our training approach successfully enhances proactive critical thinking performance while maintaining accuracy on standard questions across model sizes. Most impressively, for Qwen3-1.7B, we achieve a substantial improvement in final answer accuracy from 0.15% to 73.98%. Further analysis shows that our method also generalizes effectively to out-of-distribution unanswerable questions. Our key contributions are summarized as follows: • We introduce the concept of proactive critical thinking: a paradigm where models not only identify unanswerable queries but also provide constructive feedback to help users supply missing information. • We develop an automated data preparation pipeline for generating the high-quality flawed question and conduct two novel benchmarks, GSM-MC and GSM-MCE, for evaluating proactive critical thinking. • We demonstrate that both SFT and RL can effectively cultivate proactive critical thinking abilities, and show that their efficacy is significantly enhanced by incorporating answerability heuristics. 2 Related Work While LLMs are increasingly expected to handle real-world tasks, they often struggle with the ambiguous or flawed queries that users frequently pose. To address this issue, prior work Li et al. 2024 ; Ma et al. 2024 ; Sun et al. 2024 ; Song et al. 2025 ; Fan et al. 2025 has focused on investigating the critical thinking capabilities of LLMs. Most of these studies have concentrated on

Read the complete text