Large Language Models in Teaching and Learning: Reflections on Implementing an AI Chatbot in Higher Education
Fiammetta Caccavale, Carina L. Gargalo, Julian Kager, Magdalena Skowyra, Steen Larsen, Krist V. Gernaey, Ulrich Krühne · 2026
Read Large Language Models in Teaching and Learning: Reflections on Implementing an AI Chatbot in Higher Education by Fiammetta Caccavale, Carina L. Gargalo, Julian Kager, Magdalena Skowyra, Steen Larsen, Krist V. Gernaey, Ulrich Krühne (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17773.txt.
Introduction
Caccavale and co-authors report on a language-model chatbot inside a university course. The assistant uses retrieval-augmented generation to stand in for a teacher-led exercise. Three iterative studies with mixed methods, one a crossover, test the swap. They probe student drive, response quality, and grades, plus how the bot feels versus a human teacher. Hallucination and shallow topic knowledge frame the risks. The results weigh where such tools fit specialized teaching.
Catalog record
| CREATOR | Fiammetta Caccavale, Carina L. Gargalo, Julian Kager, Magdalena Skowyra, Steen Larsen, Krist V. Gernaey, Ulrich Krühne |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85131 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17773
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Large Language Models in Teaching and Learning: Reflections on Implementing an AI Chatbot in Higher Education Fiammetta Caccavale, Carina L. Gargalo, Julian Kager, Magdalena Skowyra, Steen Larsen, Krist V. Gernaey, Ulrich Krühne Abstract The landscape of education is changing rapidly, shaped by emerging pedagogical approaches, technological innovations such as artificial intelligence (AI), and evolving societal expectations, all of which demand thorough evaluation of new educational tools. Although large language models (LLMs) present substantial opportunities especially in Higher Education, their propensity to generate hallucinations and their limited specialized knowledge may introduce significant risks. This study aims to address these risks by examining the practical implementation of an LLM-enhanced assistant in a university level course. We implemented a generative AI assistant grounded in a retrieval-augmented generation (RAG) model to replicate a previously teacher-led, time-intensive exercise. To assess the effectiveness of the LLM, we conducted three separate experiments through iterative mixed-methods approaches, including a crossover design. The resulting data address central research questions related to student motivation, perceived differences between engaging with the LLM versus a human teacher, the quality of AI-generated responses, and the impact of the LLM on students’ academic performance. The results offer direct insights into students’ views and the pedagogical feasibility of embedding LLMs into specialized courses. Finally, we discuss the main challenges, opportunities and future directions of LLMs in teaching and learning in Higher Education. keywords Artificial Intelligence, Large Language Models, Chatbots in Education 1 Introduction Education is constantly evolving with new pedagogies (e.g., blended learning, competency-based education), technologies (e.g., artificial intelligence, mixed and virtual reality), and societal demands (e.g., 21st-century skills, sustainability education). Among these technologies, artificial intelligence (AI) is starting to be consistently integrated in Higher Education. In recent years, researchers have been particularly intrigued large language models (LLMs) and the impact that this technology could have on education. Possible benefits that LLMs could provide include personalized learning, increased teaching efficiency, tailored student support, which could lead to enhanced learning experience and overall higher engagement of students ( 18 ) . However, there are also arguably as many challenges introduced by this technology, such as over-reliance on LLMs ( 1 ) , which could hinder critical thinking and reasoning skills ( 39 ) . These challenges also include the tendency of LLMs to hallucinate, leading to students receiving incorrect information, and to be too generalized to provide domain-specific knowledge that the Higher Education curriculum requires. Other issues include possible ethical concerns, ensuring transparency, privacy and accountability ( 40 ) . Therefore, there is a need for educators to scrupulously consider applications and alignmentof LLMs with learning objectives, as well as thoroughly validate these models before they are used in education. If no precautions are taken, these issues could cause the introduction of LLMs in education to inhibit rather than facilitate learning. The purpose of this study is to contribute to the ongoing research and discussion on the introduction of LLMs in Higher Education. We introduce a chatbot previously developed by the authors ( 5 ) , which is based on a pre-trained LLM using retrieval-augmented generations (RAG), to a Master’s Degree course. The chatbot is created to simulate an interview-like exercise between the students (asking the questions) and a fictional company (originally represented by the teachers of the course) An iterative mixed-methods approach is used to validate the developed chatbot. We provide extensive first-hand opinions from students who tested this technology in three different experiments, including a crossover study. With the collected data and analysis in this contribution we show: (i) reasons and motivations of students to interact with a chatbot instead of a teacher, (ii) the main differences between interacting with a teacher compared to the chatbot, (iii) the effect on the grading, and (iv) the perceptions of adding speech-to-text and text-to-speech functions on the user experience. The learning from this experiments Finally, we summarize the opportunities and challenges that LLMs can introduce to education, and in Section 5 , we highlight the pedagogical challenges associated with AI and present our considerations on how this technology can be leveraged in collaborative learning. 2 Background This section introduces the current background for the introduction of AI models in Higher Education. First, the opportunities that LLMs could bring to this field will be discussed, followed by an introduction to their benefits to collaborative learning. Lastly, current challenges and possible drawbacks and limitations will be presented. 2.1 Opportunities introduced by LLMs in education Current advancements in AI, specifically with regards to large language models (LLMs) offer substantial benefits in Higher Education, including personalized learning ( 11 ) , tailored student support ( 37 ) , teaching efficiency ( 38 ) , greater student engagement ( 35 ) , support for research, and increased accessibility and inclusivity ( 11 ; 38 ) . These benefits can lead to an enhanced learning experience and overall higher engagement of students ( 18 ) , as well as improvements in student comprehension and academic outcomes ( 37 ) . Research indicates that AI can greatly improve learning efficiency by automatically regulating cognitive load, offering personalized teaching, and dynamically adjusting learning pathways ( 12 ) . This research highlights that AI-based strategies are particularly advantageous in high cognitive load domains such as STEM, where they can substantially enhance understanding, memory retention, and problem-solving abilities ( 12 ) . This evidence on the efficacy of LLMs as learning aids suggests that the future of engineering education will include LLMs ( 17 ) . 2.2 LLM support in collaborative learning Research has observed that LLMs can support collaborative learning by acting as intelligent conversational agents, providing real-time answers or feedback, and helping students by generating discussions tailored to group activities ( 27 ; 28 ) . For example, LLMs can support group reflection and offer alternative solutions, helping students engage more deeply, both with content and with each other. They can also enable inclusive collaboration by detecting and modeling group dynamics and offer feedback to improve teamwork and communication skills ( 6 ; 22 ) . 2.3 Pedagogical challenges introduced by AI Educational research often struggles to keep up with the pace of the changes introduced by new pedagogies, technologies and societal demands, making it difficult to conduct timely and relevant studies and capture the nuances of new learning environments ( 21 ) . Moreover, traditional educational research may not be fully equipped to study AI in education ( 7 ) . Despite the hype surrounding AI, there is a shortage of evidence for its successful incorporation and validation in real-world educational environments ( 40 ) . Studies indicate that most LLM advancements lack a focus on human-centered design and the involvement of educational stakeholders. Consequently, these tools might impede learning rather than improve it ( 40 ) . The integration of innovative technology into education presents a complex challenge ( 33 ) , as learning objectives and teaching activities need meticulous adjustment ( 2 ) . Additionally, the lack of transparency in data collection could result in the misuse of student information and an erosion of trust ( 31 ) . Academic integrity may likewise suffer, as students could employ AI to circumvent learning objectives, thereby diminishing the significance of assessments and complicating the