A RAG-Based Institutional Assistant
Gustavo Kuratomi, Paulo Pirozelli, Fabio G. Cozman, Sarajane M. Peres · 2025
Read A RAG-Based Institutional Assistant by Gustavo Kuratomi, Paulo Pirozelli, Fabio G. Cozman, Sarajane M. Peres (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13880.txt.
Introduction
Gustavo Kuratomi and co-authors build a campus assistant in a 2025 paper. Language models write well but stumble when answers live in set documents. Retrieval aided generation fixes that by fetching text chunks into the prompt. Their assistant serves the University of São Paulo, with a retriever and a generator as its two parts.
The team tried several models for each part and tuned chunk size and fetch count. The best retriever pairing anchors the final design. Students and staff ask institutions the same factual questions daily, so a grounded assistant can lift a real load. Careful tuning matters more than model size for such grounded tasks, the results suggest.
Catalog record
| CREATOR | Gustavo Kuratomi, Paulo Pirozelli, Fabio G. Cozman, Sarajane M. Peres |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13880
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A RAG-Based Institutional Assistant Gustavo Kuratomi, Paulo Pirozelli, Fabio G. Cozman, Sarajane M. Peres Abstract Resumo Although large language models (LLMs) demonstrate strong text generation capabilities, they struggle in scenarios requiring access to structured knowledge bases or specific documents, limiting their effectiveness in knowledge-intensive tasks. To address this limitation, retrieval-augmented generation (RAG) models have been developed, enabling generative models to incorporate relevant document fragments into their inputs. In this paper, we design and evaluate a RAG-based virtual assistant specifically tailored for the University of São Paulo. Our system architecture comprises two key modules: a retriever and a generative model. We experiment with different types of models for both components, adjusting hyperparameters such as chunk size and the number of retrieved documents. Our optimal retriever model achieves a Top-5 accuracy of 30%, while our most effective generative model scores 22.04% against ground truth answers. Notably, when the correct document chunks are supplied to the LLMs, accuracy significantly improves to 54.02%, an increase of over 30 percentage points. Conversely, without contextual input, performance declines to 13.68%. These findings highlight the critical role of database access in enhancing LLM performance. They also reveal the limitations of current semantic search methods in accurately identifying relevant documents and underscore the ongoing challenges LLMs face in generating precise responses. 1 Introduction Question answering (QA) has long been a central concern in natural language processing. Traditionally, models have been fine-tuned on question-answering datasets—such as SQuAD [ Rajpurkar et al. 2016 ] and MS MARCO [ Nguyen et al. 2016 ] —following an initial phase of language model pretraining. With the advent of large language models (LLMs), standalone fine-tuning has mostly become obsolete, due to to these models’ impressive in-context learning abilities [ Bubeck et al. 2023 ] . LLMs generally perform satisfactorily on question-answering tasks, especially for factual questions, since they have likely encountered relevant information during training. However, LLMs still face a significant limitation: they do not have access to external knowledge bases, making it difficult to ensure they respond according to predefined answers. This limitation is particularly pressing when answers depend on documents that are private, frequently updated, or difficult to access. Retrieval-Augmented Generation (RAG) [ Lewis et al. 2020 ] is an architecture designed to address this problem. It consists of a modular structure where—in its standard version—two components are coupled together: a retriever module, responsible for finding relevant chunks of text, and a language generation module, which takes a question and the retrieved information to create an answer. This complex architecture allows better control over the information the model uses to answer questions, enables tracing back the source of answers, and disentangles the different steps in answer generation (retrieval and generation). More importantly, new documents can be constantly added to the database without the need for additional retraining. In this paper, we present and evaluate a RAG-based application tailored for an educational context: a virtual assistant for the University of São Paulo (USP). 1 1 1 https://www5.usp.br/ . USP, one of the largest universities in Brazil, with nearly 100K students and a complex structure that includes multiple campuses, institutes, and courses, presents unique challenges in accessing information related to its institutional and regulatory aspects. This environment provides a valuable opportunity to build and assess a RAG system that can assist students, faculty, and staff in navigating USP’s extensive bureaucratic framework. Furthermore, it serves as an ideal scenario to compare the effectiveness of RAG-based systems with pure LLM approaches. Our main contributions are as follows: • Developing a RAG-based QA system specifically designed to address the normative and institutional aspects of the University of São Paulo. • Curating a comprehensive database of USP documents and constructing a QA dataset derived from these materials. • Analyzing the impact of different system components (e.g., model type, chunk size) on answer quality. All data and code are available on our GitHub page. 2 2 2 https://github.com/gustavokpc/RetrievalAugmentedGeneration . 2 Related Works Until recently, question answering was primarily addressed as a fine-tuning task, where a pretrained language model was adapted to a specific domain dataset, as exemplified by [ Rajpurkar et al. 2018 ] . However, with the rapid evolution of LLMs like GPT-4 [ OpenAI 2024 ] , Llama 2 [ Touvron and et al. 2023 ] , and Gemini [ Team 2024 ] , this method has been largely replaced by prompt engineering. However, despite their impressive ability to encode vast amounts of information, LLMs remain prone to hallucinations [ Rawte et al. 2023 ] . This vulnerability raises concerns about their reliability, particularly in contexts that require high accuracy or involve sensitive information. Retrieval-augmented generation (RAG) has emerged as an alternative to strictly parametric models [ Lewis et al. 2020 ] . In this architecture, a retrieval model first gathers relevant information, which is then fed into a generative model to produce the final answer. This approach has the advantage of utilizing a knowledge base for information, allowing the generative model to concentrate on synthesizing the retrieved information into a coherent response. In the original RAG framework, both components were neural networks, and the system was trained end-to-end on a question-answering task. While the results were impressive, the authors noted that the learning process was fragile; the generator module could easily learn to ignore the retrieved texts, especially when dealing with initially random chunks. A notable variant of RAG system, where components are built independently, has gained popularity in recent years. This modular approach facilitates the combination of various retrieval and generation modules. Typically, the embedding network employs a smaller, pre-trained model, while the generator utilizes an LLM. Over time, several enhancements to the RAG architecture have been proposed [ Gao et al. 2024 ] , including preprocessing techniques such as query rewriting [ Ma et al. 2023 ] , the incorporation of metadata [ ILIN 2023 ] , and the integration of diverse information sources like knowledge bases [ Wang et al. 2023 ] and tabular data [ Chen et al. 2020 ] . More advanced techniques include passage reranking to refine the retrieval process [ Zhuang et al. 2023 ] and iterative retrieval to address multihop problems [ Shao et al. 2023 ] . In this paper, we employ this modular variant of the RAG architecture, testing different components for both the retrieval and generative modules. 3 Dataset The University of São Paulo (USP) is one of the largest and most prestigious higher education institutions in Brazil. With an annual budget of R$8.6 billion for 2024, it comprises 42 teaching and research units across 8 campuses in 9 cities. The main campus, known as the Armando de Salles Oliveira University City in São Paulo, covers nearly 3.7 million square meters—slightly larger than Central Park. USP offers 246 undergraduate and 229 graduate programs spanning all fields of knowledge, along with community-focused extension courses. Its personnel include 5K professors, 12K staff members, and 97K students enrolled in undergraduate and graduate programs. The university’s extensive infrastructure features a radio station, research institutions like the Butantã Institute, museums such as the Museum of Ipiranga and the Museum of Modern Art, and hospitals, including the Hospital das Clínicas, the largest hospital complex in Latin America. 3 3 3 https://uspdigital.usp.br/anuario/AnuarioControle