Domain - Grounded Evaluation of LLMs in International Student Knowledge

Claudinei Daitx, Haitham Amar · 2025

Read Domain - Grounded Evaluation of LLMs in International Student Knowledge by Claudinei Daitx, Haitham Amar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20653.txt.

Introduction

Claudinei Daitx and Haitham Amar tested in 2025 how well large language models answer study-abroad questions, the kind students ask about admissions, visas, scholarships, and eligibility. Their questions came from the real advising work of ApplyBoard, a company that guides students from first search through enrolment. A wrong answer can cost a student a visa or a place, so the pair graded each reply for correctness and for invented content.

Each answer was marked correct, partial, or wrong, with partial credit tracking whether the reply covered every area the question touched. The team also scored faithfulness, meaning whether the answer stuck to the evidence, plus answer relevance and an overall hallucination score called ANAH v2. Models that reason step by step behaved measurably differently from direct answerers. GPT-5 stood out for short, on-target checklists that stayed close to the main topic instead of wandering into extra areas.

Catalog record

CREATORClaudinei Daitx, Haitham Amar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-20653

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Domain - Grounded Evaluation of LLMs in International Student Knowledge Claudinei Daitx, Haitham Amar Abstract Large language models (LLMs) are increasingly used to answer high-stakes study-abroad questions about admissions, visas, scholarships, and eligibility. Yet it remains unclear how reliably they advise students, and how often otherwise helpful answers drift into unsupported claims (“hallucinations”). This work provides a clear, domain-grounded overview of how current LLMs behave in this setting. Using realistic questions set drawn from ApplyBoard’s advising workflows—an EdTech platform that supports students from discovery to enrolment—we evaluate two essentials side by side: accuracy (is the information correct and complete?) and hallucination (does the model add content not supported by the question or domain evidence). These questions are categorized by domain scope which can be a single-domain or multi-domain—when it must integrate evidence across areas such as admissions, visas, and scholarships. To reflect real advising quality, we grade answers with a simple rubric which is correct, partial, or wrong. The rubric is domain-coverage-aware: an answer can be partial if it addresses only a subset of the required domains, and it can be over-scoped if it introduces extra, unnecessary domains; both patterns are captured in our scoring as under-coverage or reduced relevance/hallucination. We also report measures of faithfulness and answer relevance, alongside an aggregate hallucination score, to capture relevance and usefulness. All models are tested with the same questions for a fair, head-to-head comparison. Our goals are to: (1) give a clear picture of which models are most dependable for study-abroad advising, (2) surface common failure modes—where answers are incomplete, off-topic, or unsupported, and (3) offer a practical, reusable protocol for auditing LLMs before deployment in education and advising contexts. 1 Introduction Large language models (LLMs) are rapidly moving from general chat applications to high-stakes, domain-specific advising across health, finance, and education foundation_models_in_general_and_domain_specific_settings . In the study-abroad context, an assistant must combine information from multiple domains—government visa policies, program and institutional requirements, geographic and cost-of-living considerations, scholarship eligibility, timelines, and documentation—to give a student a clear, actionable answer. ApplyBoard, an EdTech platform that supports students from discovery to enrolment via a network of recruitment partners and a direct-to-student application ( https://www.applyboard.com ), exemplifies this complexity: a single query can span immigration rules, academic prerequisites, language tests, and region-specific constraints. This multi-domain composition raises two core questions for safe deployment: How accurate are LLMs when advising prospective international students, and how often do they hallucinate—i.e., introduce unsupported or misleading content that can derail a student’s planning survey_on_hallucination_in_natural_language_generation ; survey_of_hallucination_in_large_foundation_models ? Two challenges make this setting distinctive. First, cross-domain composition : correct answers often require integrating policies and requirements that vary by country, institution, program, and applicant profile. A response can be partially correct yet miss a crucial condition (e.g., a visa-specific financial threshold), making binary exact-match metrics insufficient. Second, hallucination under domain shift : when models bridge domains, they may fill gaps with plausible but unsupported details, or drift off-topic (intrinsic hallucination), or contradict established rules or curated references (extrinsic hallucination) survey_on_hallucination_in_natural_language_generation . In an advising workflow, such errors are not merely academic; they can lead to wasted time, financial loss, or missed enrolment windows, undermining user trust and institutional responsibility. To study these issues, we conduct a domain-grounded evaluation using an internal Q&A dataset drawn from ApplyBoard’s advising workflows ( https://www.applyboard.com ). The dataset follows a HealthBench-like design: student questions span multiple study-abroad domains—e.g., visas and immigration, program and school selection, admissions and eligibility, scholarships and finances, timelines and documentation—and each model response is labeled as correct, partial, or wrong to reflect graded utility rather than binary exact match healthbench . Our evaluation considers two complementary axes. Accuracy measures whether an answer is correct and sufficiently complete for the student’s goal. Hallucination measures whether an answer remains grounded in the question and avoids unsupported or off-topic content. We operationalize intrinsic hallucination with faithfulness to the prompt and answer relevance, and we report an aggregate hallucination severity index (ANAH v2) for a single, comparable score across models anah_v2 . All models are evaluated under the same system prompt and the same question set to enable a fair, head-to-head comparison. Our study is designed around practical questions that matter for deployment: RQ1: For representative study-abroad questions, how often are model answers correct, partially correct, or wrong? RQ2: When answers are not fully correct, what fraction of errors stem from incompleteness versus hallucinations? RQ3: How do faithfulness and answer relevance relate to partial-credit accuracy—i.e., can answers be helpful without drifting into unsupported claims? This paper makes two contributions: 1. A domain-grounded evaluation protocol for international-student advising that jointly assesses accuracy and hallucination. We pair a partial-credit rubric (correct/partial/wrong) with hallucination-oriented measures—faithfulness, answer relevance, and an aggregate hallucination score (ANAH v2)—to capture both usefulness and relevancy. 2. A fair, head-to-head comparison of multiple LLMs under a shared system prompt and identical questions, providing practitioners with a practical audit framework they can adapt to their own advising contexts. We report standardized metrics—factual accuracy, faithfulness, answer relevance, and ANAH v2—to facilitate like-for-like comparisons across models and prompts. Our goal is not to chase leaderboard maxima but to offer a clear, reproducible picture of how current LLMs behave when advising international students: where they are dependable, where partial answers suffice, and where hallucinations emerge as a deployment risk. The resulting protocol can guide model selection, prompt design, and guardrail policies for education-focused assistants, and provides a common language—accuracy, faithfulness, relevance, and hallucination severity—for conversations between ML teams and student-success stakeholders. 2 Overview of Accuracy Benchmark We evaluate multiple LLMs on realistic, multi-domain questions drawn from ApplyBoard’s advising workflows. A single question—“I need a visa,” “What documents do I need?,” or “Can I study in Country X?”—often spans several domains at once (immigration, finances, housing, admissions, and timelines). To reflect this reality, we operationalize accuracy with a three-level rubric—correct, partial, wrong—and compute it under two complementary settings. Factual Correctness ragas_factual_correctness measures whether the model selects the appropriate domain(s) and provides a sound short answer without external evidence. Answer Accuracy ragas_answer_accuracy measures whether the answer agrees with authoritative references when rules are country- or time-dependent. We do not treat partial as wrong: partial answers typically contain the main domain and useful content even if secondary details are missing. For domain-selection steps, we also report Hit@1 and Recall@1 to capture whether the model’s first choice aligns with the intended domain(s). 2.1 Factual Correctness We adopt the Factual Correctness metric as implemented in Ragas ( http://docs.ragas.io ; “factual_correctness”) and adapt it to our multi-domain and short answers. For each question, we specify the expected domain(s)—for example, visas/immigration, finances, housing, or program/school selection. The model must decide which domain(s) apply and provide a concise, self-contained answer without supporting documents at evaluation time. An independent judge then assigns one of three labels: • correct (the chosen domain(s) are appropriate and essential points are covered) • partial (the main domain choice is right but some required facets are missing or extraneous domains are added) • wrong (the answer centers on unrelated domain(s) or omits a critical requirement) We additionally report Hit@1 (whether the model’s first domain matches an expected domain) and Recall@1 (for multi-domain questions, whether the first choice covers at least one required domain). This

Read the complete text