WikiHint : A Human-Annotated Dataset for Hint Ranking and Generation
Jamshid Mozafari, Florian Gerhold, Adam Jatowt · 2024
Read WikiHint : A Human-Annotated Dataset for Hint Ranking and Generation by Jamshid Mozafari, Florian Gerhold, Adam Jatowt (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01626.txt.
Introduction
Jamshid Mozafari, Florian Gerhold, and Adam Jatowt built in 2024 a data set for teaching machines to give hints instead of answers. Direct answers from chatbots can dull the user's own reasoning. Their set, WikiHint, holds five thousand human-written hints for one thousand questions drawn from Wikipedia.
The team tuned open models to write hints with and without knowing the answer, then tested real people solving questions with and without hint help. They also built a light ranking tool to grade hints in both settings. Good hints, the results suggest, keep minds working while still moving them forward.
Catalog record
| CREATOR | Jamshid Mozafari, Florian Gerhold, Adam Jatowt |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01626
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
WikiHint : A Human-Annotated Dataset for Hint Ranking and Generation Jamshid Mozafari, Florian Gerhold, Adam Jatowt Abstract . The use of Large Language Models (LLMs) has increased significantly with users frequently asking questions to chatbots. In the time when information is readily accessible, it is crucial to stimulate and preserve human cognitive abilities and maintain strong reasoning skills. This paper addresses such challenges by promoting the use of hints as an alternative or a supplement to direct answers. We first introduce a manually constructed hint dataset, WikiHint , which is based on Wikipedia and includes 5,000 hints created for 1,000 questions. We then finetune open-source LLMs for hint generation in answer-aware and answer-agnostic contexts. We assess the effectiveness of the hints with human participants who answer questions with and without the aid of hints. Additionally, we introduce a lightweight evaluation method, HintRank , to evaluate and rank hints in both answer-aware and answer-agnostic settings. Our findings show that (a) the dataset helps generate more effective hints, (b) including answer information along with questions generally improves the quality of generated hints, and (c) encoder-based models perform better than decoder-based models in hint ranking. Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730284 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Retrieval models and ranking CCS: Information systems Evaluation of retrieval results † † cc-license: by 1. Introduction Figure 1. Hints for a question from the WikiHint dataset, with their corresponding rankings (1 being the highest and 5 the lowest) which let users find the answer. Note that the opposite arrangement of the hints would make the answer finding task easiest when hints are read from rank 1 to 5. Figure 2. Pipeline of WikiHint dataset generation. The numbers in the arrows indicate the counts of output questions. Qs and Hs denote the number of questions and hints, respectively. In recent years, question answering (QA) systems have risen in importance, offering users the opportunity to effortlessly receive answers to different types of questions ( Mavi et al., 2024 ; Karpukhin et al., 2020 ; Zhao et al., 2021 ; Abdel-Nabi et al., 2023 ) . The rapid development of Large Language Models (LLMs) ( Gemini Team et al., 2023 ; OpenAI et al., 2023 ; Dubey et al., 2024 ) has mainly contributed to this, impacting also many other Natural Language tasks (NLPs) ( Qin et al., 2024 ) . While the benefits of using LLMs for current information access are quite evident, concerns have been raised in relation to their potential effect on human cognitive development. One such worry relates to the potential weakening of important cognitive skills of users like thinking, reasoning, and remembering due to the expected widespread use of the automatic question answering technologies, in particular, ones backed by powerful AI technologies ( Heersmink, 2024 ) . Users who will rely mainly on the solutions presented by AI’s might also be discouraged to practice and improve their reasoning abilities ( Alfredo et al., 2024 ) . For example, Darvishi et al. (2024) demonstrate that students are more likely to depend on AI assistance instead of learning from it. Jošt et al. (2024) examine the impact of LLMs as an automated problem-solving technology on education and learning outcomes, demonstrating that such systems can negatively affect the development of learning skills. Furthermore, psychological studies confirm the importance of obtaining an answer independently, enhancing user self-confidence ( Bandura, 2013 ) . Letting users come up with the correct answers by themselves should then also contribute to the positive psychological effect, potentially increasing their self-confidence ( Usher and Pajares, 2006 ) . While the problem of over-reliance on chatbots is complex and still not researched well, we wish to promote approaches that involve humans in the answer finding process. Providing users with hints rather than direct answers is one such approach. Hints are designed to offer subtle clues that guide users toward the correct answer without explicitly revealing it ( Hume et al., 1996 ) . Unlike the concept of framing (aka "scattered conceptualization") ( Entman, 1993 ) which elevates particular pieces of information to promote a certain view or cause bias ( Garrido-Muñoz et al., 2021 ; Färber et al., 2020 ) , hints help humans to recall previously known information or guide them in the reasoning process. In this study, we define hints for factoid questions 1 1 1 Factoid questions are those with answers in the form of entities such as locations, persons, etc. as clues that encourage users to think critically, reason, and draw on their knowledge to deduce the answer rather than having it explicitly provided. This approach aims to support and enhance cognitive skills such as critical thinking, reasoning, and memory. To ensure that hints are meaningful and widely accessible, we generate them based on external knowledge and well-established facts. For this purpose, we select Wikipedia as the primary source, as it is one of the most widely recognized and comprehensive repositories of factual information. In this paper, we propose the first manually curated dataset for the Automatic Hint Generation (HG) task, called WikiHint 2 2 2 https://github.com/DataScienceUIBK/WikiHint , which has been constructed and designed for hint generation, ranking, and evaluation. Figure 1 shows a question from the WikiHint dataset, with its hints and the corresponding ranks reflecting the usefulness of each hint. We explore the performance of various LLMs in generating hints across different scenarios, including vanilla and finetuned models. We also examine the quality of the generated hints using both answer-aware and answer-agnostic approaches. Finally, we assess the effectiveness of a novel evaluation method for hint ranking, called HintRank , and compare it with other automatic evaluation techniques. Ranking hints allows arranging them either from easiest to hardest which can be used for selecting the most useful hints for a given question, or from hardest to easiest (as shown in Fig. 1 ). The latter arrangement could find applications in quizzes, knowledge assessment scenarios, or in certain entertainment use cases (e.g., in games). In summary, we make the following contributions in this paper: • We release the first manually created dataset called WikiHint for the HG task containing 5,000 hints and 1,000 questions. • We propose an automatic evaluation method for ranking hints called HintRank and compare with other evaluation methods. • We finetune and evaluate diverse LLMs on WikiHint to assess the dataset quality and LLMs’ capabilities in hint generation and ranking. • We present several novel observations, including the findings of a positive correlation between hint convergence and helpfulness, an inverse correlation between hint length and helpfulness, and the superiority of the answer-aware approach over the answer-agnostic approach in hint generation. 2. Related Work Automatic question answering (QA) ( Karpukhin et al., 2020 ; Zhao et al., 2021 ; Abdel-Nabi et al., 2023 ) and question generation (QG) ( Kurdi et al., 2020 ; Lu and Lu, 2021 ; Zhang et al., 2021 ) have progressed a lot in the last years. These tasks have seen numerous different datasets