Augmenting Researchy Questions with Sub-question Judgments
Jia-Huei Ju, Eugene Yang, Trevor Adriaanse, Andrew Yates · 2025
Read Augmenting Researchy Questions with Sub-question Judgments by Jia-Huei Ju, Eugene Yang, Trevor Adriaanse, Andrew Yates (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21733.txt.
Introduction
Jia-Huei Ju and three co-authors enrich a public data set of hard research questions with finer-grained relevance labels. The Researchy Questions collection holds about one hundred thousand real queries. Their answers sprawl across several aspects of a topic. Each query is already split into sub-questions by a language model. Yet nothing records which documents answer which sub-question.
The team uses a seventy-billion-parameter open model as judge. It labels every sub-question against the documents users actually clicked. That manufactures the missing associations at scale. The augmented data set is training ground for retrieval systems that serve complex needs aspect by aspect. Such supervision could teach search engines to cover all parts of a research question. They would no longer settle for its easiest facet.
Catalog record
| CREATOR | Jia-Huei Ju, Eugene Yang, Trevor Adriaanse, Andrew Yates |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 6 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21733
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Augmenting Researchy Questions with Sub-question Judgments
Jia-Huei Ju, Eugene Yang, Trevor Adriaanse, Andrew Yates
Abstract
The Researchy Questions dataset provides about 100k question queries with complex information needs that require retrieving information about several aspects of a topic. Each query in ResearchyQuestions is associated with sub-questions that were produced by prompting GPT-4. While ResearchyQuestions contains labels indicating what documents were clicked after issuing the query, there are no associations in the dataset between sub-questions and relevant documents. In this work, we augment the Researchy Questions dataset with LLM-judged labels for each sub-question using a Llama3.3 70B model. We intend these sub-question labels to serve as a resource for training retrieval models that better support complex information needs.
1
Introduction
Researchy Questions
Rosset et al. 2024
is a collection of queries taken from the Bing query log, along with the URLs that the user clicked, which are also present in the ClueWeb22 dataset
(
Overwijk et al. 2022
)
.
Such questions are complex and usually require searching with multiple queries in a single session to fully answer.
In the original dataset, the authors provide the decomposition of each question into several sub-questions by GPT4.
However, there are no relevance judgments linked to each decomposed sub-questions, which limit the usability of the collection as a sub-question training resource.
To fill the gap, we employ an automatic judgment model to generate the relevance labels for the training sub-questions.
We adapt a graded judgment prompt from CRUX
Ju et al. 2025
to capture different granularities of the relevance.
Compared to typical retrieval training collections such as MS MARCO
Bajaj et al. 2016
, sub-questions provided by Researchy Qeustions are less factual and require better semantic understanding to the questions and the documents.
Such an addition provides a concrete usage for the original Researchy Question collection for training factoid question retrieval model.
Qwen3-Reranking Instruction
System prompt
:
Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".
Prompt
:
<Instruct>: Given the list of questions as query, retrieve relevant passages that answer the questions.
<Query>: {join(subquestions)}
<Document>: {doc}"
Figure 1:
The prompt used to rerank documents with the pointwise Qwen3 0.6B reranking model.
Rubric-based Answerability Judgment
Instruction: Determine whether the question can be answered based on the provided context? Rate the context with on a scale from 0 to 5 according to the guideline below. Do not write anything except the rating.
Guideline:
5: The context is highly relevant, complete, and accurate.
4: The context is mostly relevant and complete but may have minor gaps or inaccuracies.
3: The context is partially relevant and complete, with noticeable gaps or inaccuracies.
2: The context has limited relevance and completeness, with significant gaps or inaccuracies.
1: The context is minimally relevant or complete, with substantial shortcomings.
0: The context is not relevant or complete at all.
Question: {
q
}
Context: {
c
}
Rating:
Figure 2:
The prompt used to assess the relevance of a document (context)
c
to a sub-question
q
on a scale from 0 to 5, following prior work obtaining sub-question judgments
(
Ju et al. 2025
)
.
2
Candidate Documents
To avoid indexing the entire ClueWeb22
(
Overwijk et al. 2022
)
, which contains more than 10B documents, we only consider documents in the English subset of ClueWeb22 Set B, which contains 48% of the clicked documents in Researchy Questions.
For efficiency, we retrieved documents from ClubWeb22 Set B using BM25.
To avoid overly biasing results toward lexical matches, we reranked the top 100 documents with the Qwen3 0.6B Reranker
Zhang et al. 2025
using a slightly modified instruction documented in Figure
1
.
For selecting the candidate documents for each sub-question of the Researchy Questions Train split, we collect the top 20 documents from the initial BM25 retrieval results and the top 20 documents from the reranking output to balance the lexical and semantic matching in the candidate set.
We additionally include the clicked documents (439,161 in total) of the original question supplied by the Researchy Question collection to ensure they are also processed by our downstream pipeline.
These documents are then passed to an automatic judgment model for acquiring relevance labels, which we describe in the next section.
3
Automatic Judgments
After identifying candidate documents, we used the Llama3.3 70B model to produce relevance judgments.
Each document was automatically judged for each sub-question derived from the top-level query.
Judgments were on a 0-5 scale using the prompts from from
Ju et al. 2025
, which are intended to assess the answerability
(
Dietz 2024
;
Farzi and Dietz 2024
)
of a question given a document.
The prompt is shown in Figure
2
.
4
Dataset Statistics
For all 1,288,976 sub-questions in the Researchy Question training set, we acquire 24,316,320 predictions, which on average have 18.87 judgments per sub-question.
Table
1
summarizes the distribution of the documents predicted in each relevance level.
Level
Count
Per Sub-question
5
1,907,722
1.48
4
5,444,774
4.22
3
2,909,134
2.26
2
7,920,642
6.14
1
2,651,382
2.06
0
3,482,653
2.70
Other
13
–
Table 1:
Statistics of each predicted relevance level.
5
Conclusion
In this work, we augment the Researchy Question dataset with automatic labels using CRUX.
This resource will enable modeling exploration for factoid retrieval and question answering.