“There Is No Such Thing as a Dumb Question,” But There Are Good Ones
Minjung Shin ([email protected]), Donghyun Kim ([email protected]), Jeh-Kwang Ryu ([email protected]) · 2025
Read “There Is No Such Thing as a Dumb Question,” But There Are Good Ones by Minjung Shin ([email protected]), Donghyun Kim ([email protected]), Jeh-Kwang Ryu ([email protected]) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-09923.txt.
Introduction
Minjung Shin, Donghyun Kim, and Jeh-Kwang Ryu built a framework in twenty twenty-five for judging what makes a question good. Questioning matters for people and for AI, yet question quality had no comprehensive measure. They define two dimensions: appropriateness, the social and linguistic fit of a question to its context, and effectiveness, how well it serves the asker's goal.
On those foundations they built a rubric scoring system whose criteria flex with dynamic context variables, aiming for structure without rigidity. Validation on the CAUS and SQUARE data sets showed the framework can flag both well-formed and problem questions and adapt across settings. The study is a step toward treating questioning itself as an analyzable skill for education and AI design.
Catalog record
| CREATOR | Minjung Shin ([email protected]), Donghyun Kim ([email protected]), Jeh-Kwang Ryu ([email protected]) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-09923
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
“There Is No Such Thing as a Dumb Question,” But There Are Good Ones Minjung Shin ([email protected]), Donghyun Kim ([email protected]), Jeh-Kwang Ryu ([email protected]) Abstract Questioning has become increasingly crucial for both humans and artificial intelligence, yet there remains limited research comprehensively assessing question quality. In response, this study defines good questions and presents a systematic evaluation framework. We propose two key evaluation dimensions: appropriateness (sociolinguistic competence in context) and effectiveness (strategic competence in goal achievement). Based on these foundational dimensions, a rubric-based scoring system was developed. By incorporating dynamic contextual variables, our evaluation framework achieves structure and flexibility through semi-adaptive criteria. The methodology was validated using the CAUS and SQUARE datasets, demonstrating the ability of the framework to access both well-formed and problematic questions while adapting to varied contexts. As we establish a flexible and comprehensive framework for question evaluation, this study takes a significant step toward integrating questioning behavior with structured analytical methods grounded in the intrinsic nature of questioning. Keywords: Question quality evaluation; Artificial Intelligence(AI); Large Language Model(LLM); Rubric Introduction Questioning is prevalent in all forms of discourse; however, it remains elusive in systematic approaches A.C. Graesser & Black ((1985)) . Instead, there is a pervasive tendency to encourage questioning itself rather than evaluating its quality Flammer ((1981)) , as exemplified by Carl Sagan’s famous quote, “There is no such thing as a dumb question.” Sagan ((2011)) Although questioning has been a cornerstone of human discourse throughout history, it has only recently emerged as a crucial element in artificial intelligence(AI) Shin et al. ((2023)) . Despite their remarkable generative capabilities, current AI systems struggle with real-world uncertainty and change Marcus & Davis ((2020)) . Given the growing demand for human-AI interaction, questions offer a simple yet powerful tool for addressing the inherent ambiguity of human language Bender & Koller ((2020)) . This is particularly crucial, as AI systems’ probabilistic generation can pose risks in critical domains Amodei et al. ((2016)) , suggesting a transition toward more deliberative validation steps instead of jumping to conclusions Toles et al. ((2023)) . These insights suggest a paradigm shift in AI development from fluent output generation toward question-centric approaches. However, there is a notable scarcity of studies on ‘how to ask’ compared to on ‘how to answer’ in both human-centered and AI-focused research. This research deficit translates into significant gaps in both the systematic analysis and evaluation of questioning A. Graesser et al. ((2009)) . Such scarcity can be attributed to two factors: 1) encouraging questioning instead of judging it and, 2) the unique role of questions in language use – questions are not a proposition but a speech act 1 1 1 a linguistic actions fulfilling social functions, like promising, commanding, or questioning, rather than just conveying information aimed at obtaining information Huddleston ((2002)) . Questions do not have absolute truth values Searle ((1969)) ; A.C. Graesser ((1985)) , so there is inherently no ‘right’ or ‘wrong’ way to ask a question. Extending the context to daily interactions makes ‘apt questioning’ more elusive. Questions shape conversation flows A.C. Graesser ((1985)) , build social rapport Kim et al. ((2022)) , and induce reflective thinking Petty et al. ((1981)) . As context deepens, the question appropriateness depends on various paralinguistic elements, making assessing question quality in absolute terms challenging. As a result, determining “Is it a good question?” is more difficult than “Is it a proper answer?” Table 1: Key Criteria for Question Evaluation Appropriateness Effectiveness Definition Sociolinguistic competence to use language flexibly according to given situations and contexts without violating behavioral standards or norms Strategic language competence that sucessfully fulfills individual intentions and accomplishing desired outcomes or goals. Focus Context-based evaluation Goal achievement-centered evaluation Key question Is the question aligned with the flow and context of the discourse? Does the question achieve its intended purpose? Characteristics Can score high even if not aligned with purpose Must align with purpose to score Whereas humans can intuitively navigate this complexity without much pressure or demands, machines require more explicit and structured guidance to process the same information. Specifically, machines face fundamental limitations; they rely on probabilistic generation rather than true understanding Marcus & Davis ((2020)) , lack social learning abilities Lake et al. ((2017)) , and have limited pragmatic schema Bender et al. ((2021)) . Therefore, clear criteria are necessary to systematically analyze and evaluate machine-generated questions. This study makes three main contributions to question evaluation. First, we established evaluation criteria that reflect the intrinsic nature of questioning as a speech act, considering both pragmatic functions and contextual dependencies. Second, we develop a systematic scoring framework using a rubric-based approach. Finally, we implemented an automated evaluation system utilizing Large Language Models(LLMs), demonstrating how structured human-defined criteria can be effectively applied. These contributions advance our understanding of question quality assessment and provide practical tools for improving human-AI interactions through better Question Generation(QG) and evaluation. Question Evaluation in NLP Researchers in Natural Language Processing(NLP) proposed new QG evaluation methods to highlight the shortcomings of traditional techniques Mulla & Gharpure ((2023)) . The conventional method relies on sentence similarity metrics to compare model-generated questions with human-written reference sentences Papineni et al. ((2002)) ; Lin ((2004)) ; Banerjee & Lavie ((2005)) . While this approach is effective for other NLP tasks, such as translation or summarization, it does not align with question’s context-dependency and one-to-many nature. Some researchers address these challenges by suggesting multiple reference sentences Oh et al. ((2023)) or unsupervised methods Ji et al. ((2022)) , due to the practical impossibility of obtaining diverse reference sentences given the open-ended nature of questions. Qi et al. ((2020)) . Human evaluation offers an alternative through criteria-based scoring of fluency, relevance, accuracy, and difficulty Mulla & Gharpure ((2023)) , but remains cost-inefficient and vulnerable to subjective bias. Amidst these limitations, domain-constrained evaluation criteria have arisen in various fields. Machine reading comprehension systems focus on question diversity Sultan et al. ((2020)) ; Yoon & Bak ((2023)) , visual QG systems emphasize uniqueness Jain et al. ((2017)) , and the Questioning Turing Test framework considers human-likeness, correctness, and strategicness Damassino ((2020)) . However, these approaches remain fragmented and limited in scope. The fundamental diversity of questions intensifies challenges when extended to real-world contexts. Questions vary in form and type, with evaluation needs differing by conversational purpose Flammer ((1981)) ; A.C. Graesser et al. ((1992)) . For instance, clear and concise questions suit collaborative tasks Rothe et al. ((2017)) ; Rothe et al. ((2018)) , while deep and exploratory questions are more effective in educational settings for facilitating student thinking Rickards & Di Vesta ((1974)) ; Dillon ((2006)) . Taken together, existing attempts to evaluate machinery QG show major limitations. Current methods rely on irrelevant automatic metrics, subjective assessments, and rigidly task-specific criteria. These issues hinder high-performance AI in real-world applications, especially in open-ended conversations. Thus, comprehensive evaluation criteria are needed to effectively capture questioning behavior and ensure broad applicability. Development and Validation of Question Evaluation Metric While current LLMs demonstrate human-like language, their fundamentally different mechanisms necessitate a more nuanced comparison. Following Mahowald et al., (2024) , we distinguish between formal linguistic competence ( knowledge of language rules and patterns ) and functional linguistic competence ( the ability to use language