A Retrieval-Augmented Generation Framework for Academic Literature Navigation in Data Science
Ahmet Yasin Aytar, Kamer Kaya, Kemal Kilic · 2024
Read A Retrieval-Augmented Generation Framework for Academic Literature Navigation in Data Science by Ahmet Yasin Aytar, Kamer Kaya, Kemal Kilic (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15404.txt.
Introduction
Ahmet Yasin Aytar, Kamer Kaya, and Kemal Kilic present a retrieval-augmented system for data science literature. Their 2024 arXiv paper tackles a common problem for researchers. Data science papers grow fast, and finding the right one is hard. Their system aims to return precise, relevant academic sources. It pairs retrieval with language model generation.
Several methods boost the quality of retrieved material. A tool called GROBID extracts bibliographic data from papers. Fine-tuned embedding models and semantic chunking sharpen the search. An abstract-first retrieval step further focuses results. The team judged the system with a standard assessment framework. Scores improved strongly on key measures such as context relevance. The work targets better literature navigation for working data scientists.
Catalog record
| CREATOR | Ahmet Yasin Aytar, Kamer Kaya, Kemal Kilic |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 87 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15404
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Retrieval-Augmented Generation Framework for Academic Literature Navigation in Data Science Ahmet Yasin Aytar, Kamer Kaya, Kemal Kilic Abstract In the rapidly evolving field of data science, efficiently navigating the expansive body of academic literature is crucial for informed decision-making and innovation. This paper presents an enhanced Retrieval-Augmented Generation (RAG) application, an artificial intelligence (AI)-based system designed to assist data scientists in accessing precise and contextually relevant academic resources. The AI-powered application integrates advanced techniques, including the GeneRation Of BIbliographic Data (GROBID) technique for extracting bibliographic information, fine-tuned embedding models, semantic chunking, and an abstract-first retrieval method, to significantly improve the relevance and accuracy of the retrieved information. This implementation of AI specifically addresses the challenge of academic literature navigation. A comprehensive evaluation using the Retrieval-Augmented Generation Assessment System (RAGAS) framework demonstrates substantial improvements in key metrics, particularly Context Relevance, underscoring the system’s effectiveness in reducing information overload and enhancing decision-making processes. Our findings highlight the potential of this enhanced Retrieval-Augmented Generation system to transform academic exploration within data science, ultimately advancing the workflow of research and innovation in the field. 1 Introduction Like other Generative Artificial Intelligence (GenAI) technologies—such as image, video, and audio generation—Large Language Models (LLMs) that produce text also possess the potential to disrupt both business and social life by transforming how information is processed and decisions are made. However, a significant drawback of LLMs is their current tendency to produce "hallucinations" (i.e., inaccurate or misleading outputs), which can undermine trust in their applications. To address this issue, ongoing research explores Retrieval-Augmented Generation (RAG) as a promising solution to mitigate these inaccuracies. Moreover, across various disciplines, practitioners, and researchers are actively seeking to harness the capabilities of LLMs, particularly as intelligent assistants to enhance decision-making processes. RAG can potentially improve the effectiveness of intelligent assistants by integrating relevant information from external sources, allowing for more accurate and contextually appropriate responses. Data science, as a critical field in the realm of digital transformation that relies on data-driven insights, is no exception; it stands to benefit from advancements in LLM technology while addressing its inherent weaknesses. Data science is an exploratory process that necessitates the formulation and validation of hypotheses using historical datasets. The models developed by data scientists essentially represent these hypotheses, and creating effective models relies on the formulation of pertinent questions. Therefore, data scientists benefit from exposure to various alternatives, and academic literature serves as a vital resource that enhances their creative thinking abilities. However, as data science continuously evolves, it generates an overwhelming volume of research, making the efficient access to and utilization of academic literature increasingly challenging. This surge in available information complicates the ability of researchers and practitioners to identify and integrate relevant insights into their work, highlighting the need for effective tools and strategies to navigate this expanding landscape. To mitigate this issue, in this research, we have developed a RAG application specifically designed to aid data scientists in navigating academic resources and obtaining guidance tailored to their unique needs. This application encompasses a broad spectrum of data science subfields, including machine learning, deep learning, time series modeling, synthetic data generation, reinforcement learning, data preprocessing, feature engineering, and evaluation. The value of our RAG application is evident in its ability to provide precise and contextually relevant responses to specific queries posed by data scientists. This tool not only conserves time but also improves the accuracy of decisions made throughout the data science workflow by integrating advanced techniques for retrieving and embedding pertinent academic insights into a LLM. Consequently, the application enables users to make well-informed decisions regarding methodologies and techniques, thereby contributing to the development of more effective and innovative solutions for their projects. Our objective with this application is to systematically combine and assess the performance of a range of established techniques within the RAG framework, reporting their effectiveness within a particular domain to generate insights valuable to researchers. By conducting a comprehensive evaluation of these methods in a real-world setting, we aim to facilitate the broader adoption of these approaches in both academic and industrial contexts. Our RAG application responds to a significant need within the data science community for accessible and relevant academic guidance. Unlike traditional literature review methods, which can be time-consuming and overwhelming due to the vast amount of available information, our application facilitates the process by extracting and embedding only the most pertinent sections of articles into the LLM. This approach accelerates decision-making and ensures that the information provided is both current and directly relevant to the user’s query, thereby addressing the prevalent issue of information overload in data science research. The primary innovation of our RAG application lies in its five-stage enhancement process , which substantially improves the traditional RAG pipeline. Initially, we utilize GROBID (GeneRation Of BIbliographic Data) to clean and structure article data before constructing the vector database, ensuring the accuracy and organization of the information [ 22 ] . Next, we implement a specialized fine-tuning process using academic textbooks, which enhances the embedding model’s capacity to interpret and process complex queries. Furthermore, our application employs semantic chunking rather than traditional recursive chunking which ensures that text is divided into meaningful units and results in a more coherent and relevant vector database [ 14 ] . We also adopt an abstract-first method, prioritizing searches within a separate vector database of article abstracts, thereby streamlining the retrieval process. Finally, advanced prompting techniques are employed to optimize the LLM’s responses, leading to more accurate and contextually appropriate answers. Llamaindex and LangChain were instrumental in creating fine-tuning data and RAG pipelines, respectively [ 21 , 5 ] . Additionally, OpenAI’s GPT-4o model was employed as the LLM throughout the study, chosen for its cost-effectiveness and efficiency. To validate the effectiveness of our application, we employed the RAGAS (Retrieval Augmented Generation for Academic Search) framework and conducted tests with 50 sample questions covering various domains and problem types [ 7 ] . This rigorous testing process confirmed the application’s capability to deliver high-quality, relevant responses, highlighting its potential as a useful tool for data scientists. By integrating these advanced techniques, our application significantly enhances the traditional RAG pipeline, providing data scientists with a practical tool that simplifies academic exploration and supports informed decision-making. To validate the effectiveness of our application, we employed the RAGAS (Retrieval Augmented Generation for Academic Search) framework and conducted tests with 50 sample questions covering various domains and problem types [ 7 ] . This rigorous testing process confirmed the application’s capability to deliver high-quality, relevant responses, highlighting its potential as a useful tool for data scientists. Although all metrics in the RAGAS framework—Context Relevance, Faithfulness, and Answer Relevance—are important, our primary focus was on improving Context Relevance. This metric serves as the main indicator of the system’s ability to retrieve pertinent academic content, which aligns with the central objective of this study: enhancing the retrieval process for relevant information in academic literature for data scientists. The other metrics, such as Faithfulness and Answer Relevance, assess the quality of answers generated using the retrieved context, but since the language model used for answer generation remained unchanged, these metrics are less indicative of improvements brought by the fine-tuning process and other retrieval enhancements.