Iterative NLP Query Refinement for Enhancing Domain-Specific Information Retrieval: A Case Study in Career Services

Elham Peimani, Gurpreet Singh, Nisarg Mahyavanshi, Aman Arora, Awais Shaikh · 2024

Read Iterative NLP Query Refinement for Enhancing Domain-Specific Information Retrieval: A Case Study in Career Services by Elham Peimani, Gurpreet Singh, Nisarg Mahyavanshi, Aman Arora, Awais Shaikh (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17075.txt.

Introduction

Elham Peimani and co-authors refine search queries step by step to find career service pages more reliably in this 2024 paper. Keyword search systems score niche pages poorly. Generic questions about interview prep matched college pages only weakly. The team worked with one Canadian college's career services site. Their method adds domain terms to the query in rounds. Terms came from the site's own service names and online labels.

Each round of refinement lifted the match scores of the best pages. The process is semi-automated, with human checks in the loop. Students hunting help are the end users in mind. The case study shows the fix on real pages. Other niche domains could copy the recipe. The paper is an information retrieval practice study.

Catalog record

CREATORElham Peimani, Gurpreet Singh, Nisarg Mahyavanshi, Aman Arora, Awais Shaikh
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17075

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Iterative NLP Query Refinement for Enhancing Domain-Specific Information Retrieval: A Case Study in Career Services Elham Peimani, Gurpreet Singh, Nisarg Mahyavanshi, Aman Arora, Awais Shaikh Abstract Retrieving semantically relevant documents in niche domains poses significant challenges for traditional TF-IDF-based systems, often resulting in low similarity scores and suboptimal retrieval performance. This paper addresses these challenges by introducing an iterative and semi-automated query refinement methodology tailored to Humber College’s career services webpages. Initially, generic queries related to interview preparation yield low top-document similarities (approximately 0.2–0.3). To enhance retrieval effectiveness, we implement a two-fold approach: first, domain-aware query refinement by incorporating specialized terms such as resources-online-learning , student-online-services , and career-advising ; second, the integration of structured educational descriptors like “online resume and interview improvement tools.” Additionally, we automate the extraction of domain-specific keywords from top-ranked documents to suggest relevant terms for query expansion. Through experiments conducted on five baseline queries, our semi-automated iterative refinement process elevates the average top similarity score from approximately 0.18 to 0.42, marking a substantial improvement in retrieval performance. The implementation details, including reproducible code and experimental setups, are made available in our GitHub repository https://github.com/Elipei88/HumberChatbotBackend & https://github.com/Nisarg851/HumberChatbot . We also discuss the limitations of our approach and propose future directions, including the integration of advanced neural retrieval models. 1 Introduction Effective information retrieval (IR) within specialized domains is often impeded by limited semantic overlap between user queries and document content. Traditional lexical-based methods, such as Term Frequency-Inverse Document Frequency (TF-IDF), while interpretable and resource-efficient, frequently underperform in scenarios requiring deep semantic understanding. Although neural retrieval methods [ Devlin et al., 2019 , Karpukhin et al., 2020 ] have advanced the state-of-the-art, their complexity and computational demands make them less suitable for deployment in resource-constrained environments or domains with limited data. This study focuses on enhancing IR for Humber College’s career services webpages, a specialized domain where users seek specific information related to career development, interview preparation, and educational resources. Initial evaluations using generic queries, such as “How can I prepare for an interview?”, revealed that the top-ranked documents achieved relatively low cosine similarity scores (around 0.16888), indicating weak alignment between the queries and the retrieved documents. To address this, we propose an iterative and semi-automated query refinement strategy that augments the original queries with domain-specific terminology and structured descriptors. By introducing terms like “resources-online-learning” and “student-online-services”, and phrases such as “online resume and interview improvement tools”, we aim to bridge the vocabulary gap and better capture the intent behind user queries. Furthermore, to enhance the scalability and adaptability of our approach, we automate the extraction of relevant domain keywords from the top-ranked documents, facilitating dynamic query expansion without extensive manual intervention. This methodology not only improves retrieval performance but also maintains the interpretability and efficiency advantages of TF-IDF-based systems. The main contributions of this paper are: 1. Demonstrating that iterative, NLP-driven query refinement can significantly enhance TF-IDF retrieval performance in specialized domains. 2. Providing a reproducible framework and codebase for semi-automated query refinement, accessible via our GitHub repository. 3. Offering insights into domain-specific anchoring techniques that improve IR effectiveness in niche contexts. 2 Related Work Classical IR approaches, such as those based on lexical overlap and TF-IDF weighting [ Chowdhury, 2010 ] , have long been foundational in the field. However, the advent of neural retrieval models, leveraging contextual embeddings [ Devlin et al., 2019 , Reimers & Gurevych, 2019 , Karpukhin et al., 2020 ] , has introduced more sophisticated mechanisms for capturing semantic relationships between queries and documents. Despite their advancements, neural models often require substantial computational resources and large datasets, limiting their applicability in specialized or resource-constrained settings. Query expansion and refinement techniques [ Cui et al., 2002 , Najar & Bouguila, 2023 ] remain pertinent for addressing vocabulary mismatches between queries and documents. Recent studies [ Mo et al., 2023 , Thakur et al., 2021 ] have highlighted the importance of domain adaptation and strategic query design in enhancing retrieval performance. Our approach aligns with these findings, presenting a cost-effective method to improve IR outcomes without relying on complex neural architectures. 3 Method 3.1 Data and Vectorization We utilize Humber College’s career services webpages as our primary dataset, which are preprocessed and vectorized using TF-IDF with the scikit-learn library [ Pedregosa et al., 2011 ] . The preprocessing steps include tokenization, lowercasing, and removal of stopwords to ensure uniformity and relevance in the textual data. The TF-IDF value for term t in document d from corpus D is calculated as: \text{tf-idf}(t,d,D)=\text{tf}(t,d)\times\log\frac{|D|}{|\{d^{\prime}\in D:t\in d^{\prime}\}|} where \text{tf}(t,d) represents the term frequency of t in d , and the inverse document frequency component accounts for the rarity of t across the corpus. Query and document vectors are compared using cosine similarity: \text{cosine\_sim}(q,d)=\frac{q\cdot d}{\|q\|\|d\|} This metric quantifies the similarity between the query and document vectors, facilitating the ranking of documents based on relevance. 3.2 Retrieval Process Given a query vector q , we compute cosine similarities with all document vectors in the corpus and retrieve the top-ranked matches. Baseline queries demonstrated top similarities in the range of 0.16888 to 0.20048, signifying a weak alignment and highlighting the necessity for query refinement. 3.3 Iterative and Semi-Automated Query Refinement Our approach involves a two-step refinement process: 1. Domain Term Extraction : Identify domain-specific terms from the top-ranked documents. This is achieved by analyzing URLs and extracting relevant patterns such as ‘resources-online-learning’ and ‘student-online-services’ using regular expressions. 2. Query Expansion : Enhance the original queries by incorporating the extracted domain terms and structured educational descriptors like “online resume and interview improvement tools.” This expansion aims to capture more specific aspects of the user intent and align better with the document content. To automate and streamline this process, we developed scripts that parse the top documents, extract pertinent keywords, and suggest them for inclusion in the refined queries. This semi-automated methodology reduces the reliance on manual heuristics, allowing for scalable and consistent query enhancements. Algorithm 1 Semi-Automated Query Refinement 1: Input: initial_queries, documents 2: refined_queries \leftarrow [] 3: for each query q in initial_queries do 4: scores, top_docs \leftarrow RetrieveTopDocs( q , documents) 5: domain_terms \leftarrow ExtractDomainTerms(top_docs) 6: refined_q \leftarrow RefineQuery( q , domain_terms) 7: refined_queries.append(refined_q) 8: return refined_queries 4 Implementation The implementation of our query refinement system is structured into backend and frontend components, with scripts facilitating the experimental workflow. 4.1 Backend The backend is developed in Python, utilizing modules for model management, query processing, and data preprocessing. Key components include: • Models.py : Handles the loading and management of TF-IDF vectorizers and other necessary models. • main.py : Serves as the entry point for executing query refinement and retrieval processes. • query_controllers.py : Manages the logic for extracting and integrating domain-specific terms into queries. • web_scrapper_controllers.py : Facilitates the scraping and preprocessing of career services webpages. • vectorizer.pk and nlp.pk : Serialized objects storing the TF-IDF vectorizer and NLP processing utilities, respectively. 4.2 Frontend The frontend is developed using modern web technologies, ensuring an intuitive interface for users to input queries and view retrieved documents. Components include: • index.html : The main HTML file rendering the user interface. • src/ : Contains JavaScript and TypeScript files managing frontend logic and interactions. • tailwind.config.js and postcss.config.js : Configuration files

Read the complete text