Validating LLM-Generated Relevance Labels for Educational Resource Search Conference: Workshop on Large Language Models for IR Evaluation; February 2025; Virtual Event CCS: Information systems Information retrieval CCS: Information systems Test collections CCS: Information systems Relevance assessme
Ratan J. Sebastian, Anett Hoppe · 2025
Read Validating LLM-Generated Relevance Labels for Educational Resource Search Conference: Workshop on Large Language Models for IR Evaluation; February 2025; Virtual Event CCS: Information systems Information retrieval CCS: Information systems Test collections CCS: Information systems Relevance assessme by Ratan J. Sebastian, Anett Hoppe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12732.txt.
Introduction
Ratan J Sebastian and Anett Hoppe test whether large language models can judge which search results are relevant for teachers. Their 2025 paper notes that human relevance judgments are costly and need expertise, so automatic judging is attractive. Models have done well on general web search, but education search, with its own quality criteria, was untested.
The team collected 401 human judgments from teaching professionals doing lesson planning searches, and released the data set. Three prompt designs were compared: a simple two aspect scheme from earlier work, a twelve part rubric from education research, and criteria drawn from the study's own participants. Domain aware prompts agreed strongly with human judges, well above the simple baseline. The participant based scheme proved most robust.
Catalog record
| CREATOR | Ratan J. Sebastian, Anett Hoppe |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-12732
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Validating LLM-Generated Relevance Labels for Educational Resource Search Conference: Workshop on Large Language Models for IR Evaluation; February 2025; Virtual Event CCS: Information systems Information retrieval CCS: Information systems Test collections CCS: Information systems Relevance assessment Ratan J. Sebastian, Anett Hoppe Abstract . Manual relevance judgements in Information Retrieval are costly and require expertise, driving interest in using Large Language Models (LLMs) for automatic assessment. While LLMs have shown promise in general web search scenarios, their effectiveness for evaluating domain-specific search results, such as educational resources, remains unexplored. To investigate different ways of including domain-specific criteria in LLM prompts for relevance judgement, we collected and released a dataset of 401 human relevance judgements from a user study involving teaching professionals performing search tasks related to lesson planning. We compared three approaches to structuring these prompts: a simple two-aspect evaluation baseline from prior work on using LLMs as relevance judges, a comprehensive 12-dimensional rubric derived from educational literature, and criteria directly informed by the study participants. Using domain-specific frameworks, LLMs achieved strong agreement with human judgements (Cohen’s κ \kappa up to 0.650), significantly outperforming the baseline approach. The participant-derived framework proved particularly robust, with GPT-3.5 achieving κ \kappa scores of 0.639 and 0.613 for 10-dimension and 5-dimension versions respectively. System-level evaluation showed that LLM judgements reliably identified top-performing retrieval approaches (RBO scores 0.71-0.76) while maintaining reasonable discrimination between systems (RBO 0.52-0.56). These findings suggest that LLMs can effectively evaluate educational resources when prompted with domain-specific criteria, though performance varies with framework complexity and input structure. 1. Introduction Large Language Models (LLMs) are increasingly being explored as a way to address the resource-intensive challenge of generating relevance judgements for Information Retrieval (IR) evaluation ( Faggioli et al. 2023 ) . While recent research suggests LLMs can achieve human-comparable performance in general relevance judgements for web-searches ( Thomas et al. 2024 ) , their reliability for domain-specific, multi-dimensional relevance judgements remains an open question. This is particularly crucial in professional search contexts like education, where relevance assessment requires evaluating resources across multiple domain-specific criteria that reflect both enduring professional needs (grade level appropriateness, pedagogical fit, etc.) and immediate task requirements (learning resource type, current class topic, etc.). Prior research suggests that while many professional searchers rely on specialised repositories ( Russell-Rose et al. 2018 ) , teachers predominantly use general web search ( Cortinovis et al. 2019 ; De Los Arcos et al. 2016 ; Medio et al. 2019 ) . This requires them to layer their pedagogical expertise onto standard web search relevance criteria. They must evaluate resources across numerous dimensions including content match, grade-level appropriateness, student engagement, and pedagogical soundness ( Reitsma et al. 2010 ; Diekema and Olsen 2012 ; Wetzler et al. 2013 ; Bortoluzzi and Marenzi 2017 ) . This paper investigates how effectively LLMs can generate reliable relevance judgements for educational resource search while accounting for these domain-specific complexities. Previous research into using LLMs in the zero-shot setting ( Thomas et al. 2024 ; Faggioli et al. 2023 ) has found effective prompting strategies for relevance judgement of general web search documents. Particularly, Thomas et al. ( Thomas et al. 2024 ) showed that the most important component of a prompt for relevance judgements were the "aspects" of the judgement; i.e. the sub-dimensions of the overall judgement on which the LLM was asked to score the resource first before aggregating into an overall score. In this work, we examine how well this strategy works for domain-specific, professional relevance judgements in the context of educational searches by teachers. We specifically examine two key research questions: • RQ1 : Which combination of relevance dimensions leads to the best agreement with the relevance judgements of human searchers? • RQ2 : Do LLM relevance judgements lead to similar query and system rankings as human judgements? To address these questions, we used data from a user study where participants performed typical lesson planning-related tasks. During the study, teachers were asked to think aloud about their relevance judgements. These transcripts were coded on the standard TREC relevance judgement scale and used as ground truth ( Voorhees 2004 ) . We derived various breakdowns of relevance dimensions from the literature on teacher search ( Reitsma et al. 2010 ; Wetzler et al. 2013 ) as well as from the results of the user study itself. To guide LLMs in making relevance assessments, we incorporated these education-specific relevance dimensions into the best-performing prompt template from prior work on LLM-based IR relevance assessment ( Thomas et al. 2024 ) . We then compared how well the LLM performed when using these domain-specific dimensions versus the more generic aspects from this prior work: trustworthiness and content match. Through this investigation, we contribute to both the practical implementation of LLM-based IR evaluation and the theoretical understanding of how automated systems can support human judgement in professional search contexts. Our findings provide insight into which aspects of educational resource relevance are important for LLMs to make good overall relevance judgements and which require more careful human validation Our methodology offers a practical framework for implementing LLM-based relevance assessment in complex professional domains. 2. Related Work In this section, we survey important related work in the fields of relevance judgements broadly for professional search and then specifically for educational resources. We then cover recent work in using LLMs for relevance judgements and the challenges of making these judgements in domain-specific contexts. 2.1. Relevance in Professional Search Professional search encompasses both specialised database search and general web search adapted for domain-specific needs ( Russell-Rose et al. 2018 ; Cortinovis et al. 2019 ) . Across domains like healthcare, law, patents and education, professionals must evaluate resources against multiple domain-specific criteria that reflect both enduring professional contexts and immediate task requirements ( Russell-Rose et al. 2018 ; Reitsma et al. 2010 ) . For instance, healthcare professionals evaluate resources against regulatory requirements and evidence standards, while legal researchers consider jurisdictional constraints and case currency ( Russell-Rose et al. 2018 ) . Teachers represent an interesting case within professional search, as they primarily use general web search tools rather than specialised repositories ( Medio et al. 2019 ; Richter and Ehlers 2010 ) . They must layer their pedagogical content knowledge onto general search interfaces to evaluate resources across multiple dimensions including content match, situational fit, affective match ( Reitsma et al. 2010 ) and reputation judgements ( Wetzler et al. 2013 ) . 2.2. Relevance in Educational Resources The evaluation of educational resources introduces unique challenges due to the multi-dimensional nature of pedagogical relevance. Reitsma et al. ( Reitsma et al. 2010 ) identified nine specific alignment criteria for K-12 teaching, including student motivation, concept coverage, grade-level appropriateness, and the availability of supplementary materials. Their research demonstrated that breaking down relevance into specific dimensions produces more reliable judgements from human annotators compared to overall alignment assessments. Teachers’ evaluation strategies are characterised by partial rather than holistic assessment ( Diekema and Olsen 2012 ) . Rather than evaluating documents as complete units, teachers focus on how specific parts can be adapted to their needs. This aligns with Bortoluzzi et al.’s ( Bortoluzzi and Marenzi 2017 ) findings that teachers prefer finding adaptable authentic