Autonomous LLM-generated Feedback for Student Exercises in Introductory Software Engineering Courses

Andreas Metzger · 2026

Read Autonomous LLM-generated Feedback for Student Exercises in Introductory Software Engineering Courses by Andreas Metzger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20803.txt.

Introduction

Metzger presents NAILA, a tool that gives 24/7 self-run feedback for student tasks in introductory software engineering courses. Using modern large language models, NAILA processes student solutions in open document formats against teacher-set model answers through specialized prompt forms. He ran an empirical study with over 900 active students at the University of Duisburg-Essen. It checked reasons to adopt or reject NAILA, user take-up, and how often and steadily students engage with NAILA. It also checked how AI feedback impacts academic performance versus human feedback.

Catalog record

CREATORAndreas Metzger
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45438
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20803

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Autonomous LLM-generated Feedback for Student Exercises in Introductory Software Engineering Courses Andreas Metzger Abstract Introductory Software Engineering (SE) courses face rapidly increasing student enrollment numbers, participants with diverse backgrounds and the influence of Generative AI (GenAI) solutions. High teacher-to-student ratios often challenge providing timely, high-quality, and personalized feedback a significant challenge for educators. To address these challenges, we introduce NAILA, a tool that provides 24/7 autonomous feedback for student exercises. Utilizing GenAI in the form of modern LLMs, NAILA processes student solutions provided in open document formats, evaluating them against teacher-defined model solutions through specialized prompt templates. We conducted an empirical study involving 900+ active students at the University of Duisburg-Essen to assess four main research questions investigating (1) the underlying motivations that drive students to either adopt or reject NAILA, (2) user acceptance by measuring perceived usefulness and ease of use alongside subjective learning progress, (3) how often and how consistently students engage with NAILA, and (4) how using NAILA to receive AI feedback impacts on academic performance compared to human feedback. 1 Problem Statement Teaching Software Engineering (SE) programmes – particularly undergraduate introductory courses on SE – faces important challenges due to recent trends and developments. We first discuss and illustrate these challenges using actual data from our first-year undergraduate course on ”introduction to SE” at the University of Duisburg-Essen (Germany). We then then present the key contributions of this paper. 1.1 Challenges (a) High number of students: As expressed in [ 10 , 18 ] , the growing number of students enrolling in computer science programs is pushing educators to their limits. This poses significant challenges to computer science education, and consequently also to SE education. Fig 1 depicts the students’ participation in our undergraduate SE course measured by the number of students that have been admitted to the final exams. As can be seen, enrollment numbers drastically increased (with a CAGR of 32.7%) due to the fact that our SE course became mandatory in five different degree programs (cf. Fig 2 ). Fig. 1 : Total number of participants in UDE’s introductory SE course (CAGR = 32.7%) Specifically, the resulting high teacher-to-student ratio makes it challenging to provide timely and high-quality feedback [ 23 ] . Manual assessment can be prone to subjectivity and inconsistencies across a large number of submissions, and often teachers do not have sufficient time and resources to provide detailed feedback. (b) Diverse types of students: Introductory courses often include interdisciplinary students who are not SE majors. In our introductory SE course, we teach students from a broad range of degree programs. Fig. 2 shows the distribution of students that participated in the final exam of the winter semester 25/26 course. Fig. 2 : Degree programs in UDE’s introductory SE course ( N=670 ) (c) Widespread use of Generative AI (GenAI): Many of the classical student assignments and exercises to reinforce and train the material presented during lectures can easily be solved by modern GenAI solutions, such as Large Langage Models (LLMs). As students even in their first year of colleague show an increasing proficiency in using GenAI [ 19 ] , these two factors mean that educators face rapidly changing classroom environments and disrupted teaching principles, as solutions for assignments and exercises may be fully generated via GenAI [ 25 ] . In our SE course, the students’ self-reported GenAI experience (see Figure 3 ) indicates that the majority of students (89%) that responded to an online survey during one of our lectures has GenAI knowledge of intermediate or better. Fig. 3 : Students AI experience levels in UDE’s introductory SE course ( N=116 ; single choice) This means that assignments and exercises – in their current form – loose their effectiveness, as it becomes difficult to determine whether the student has learned the given topic [ 5 ] . (d) Preference for E-Learning: Survey- and focus-group-based research shows a clear preference for e-learning, as this provides flexibility for scheduling learning sessions [ 6 ] . We observe this also in our SE course as visible from Table I , where videos, AI-based learning and lecture slides are forming the top 3 learning resources. Learning Resource Percentage Explanatory videos 81% AI-based interactive learning 72% Lecture slides 71% E-books / PDF / Kindle 37% Classic books in paper form 25% Other 11% TABLE I : The different uses of learning resources as reported by SE students ( N=359 ; multiple choice) 1.2 Contributions To address the aforementioned trends and problems, we introduce naila , the ( N ovel AI -based L earning A pp). naila provides 24/7 autonomous feedback for introductory SE exercises. naila covers a range of different types of SE questions, ranging from ones, where the exact solution is required (i.e., has to match the model solution) to ones where students may answer in a free-text form not prescribed by a model solution. Thereby naila goes beyond the typical form of assessment for programming assignments (see Section 4 ). By autonomously providing personalized feedback to students, naila aids educators in efficiently and effectively delivering high-quality feedback. To assess the efficiency and effectiveness of naila , we answer the following research questions: • RQ1: Why do students use or don’t use NAILA? This question helps us to understand the motivation for using naila , but also provides the reasons for why students may have decided against using naila . • RQ2: How do students accept and perceive naila ? This question assesses self-reported learning gains of students by determining the perceived usefulness, ease of use, and learning progress when using naila . • RQ3: How is naila used by students? This question quantitatively assesses the usage patterns of students, including how many students use naila and how often and repeatedly for the same exercise they use naila . • RQ4: How does naila affect learning? This is the most fundamental research question we aim to answer. It assesses in how far naila ’s feedback improves the students’ academic performance compared to human feedback provided during face-2-face exercise meetings. The remainder of paper is structured as follows. Section 2 describes the architecture and realization of naila , particularly focusing on how the underlying LLM was prompted. Section 3 introduces the setup and the results of our empirical study to answer RQ1–RQ4. Section 4 discusses related work. Section 5 provides a critical discussion how the above challenges and trends are addressed, sketches future directions and raises regulatory considerations for the use of AI in education. 2 Realization of naila Figure 4 depicts the main components of naila . naila takes a student’s solution to a given exercise i and breaks it down into its individual answers a_{i,k} (one per each question q_{i,k} ). Exercises are provided as open document (ODT format) files, which use hashtags ’#’ to specify the areas where answers are to be inserted and what the maximum points that can be achieved are. An example is shown in Figure 5 below. Fig. 4 : Conceptual Architecture and Data Flow of naila Question 8.1a) The following two quality assurance techniques are available in a software project: code review , for which experience shows it finds 10 defects in 5 person hours;

Read the complete text