Prompt-Based Cost-Effective Evaluation and Operation of ChatGPT as a Computer Programming Teaching Assistant
Marc Ballestero-Ribó, Daniel Ortiz Martínez · 2025
Read Prompt-Based Cost-Effective Evaluation and Operation of ChatGPT as a Computer Programming Teaching Assistant by Marc Ballestero-Ribó, Daniel Ortiz Martínez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17176.txt.
Introduction
Marc Ballestero-Ribo and Daniel Ortiz Martinez test chatbots as programming teaching assistants in a 2025 paper. Large language models promise near one to one tutoring, with help on demand for students stuck on basic coding tasks. Two well known models were judged on the feedback they give beginners, and their cost in real use was weighed.
The stronger model far outperformed the weaker one, yet still produced wrong information that novices may not catch, so it is not classroom ready alone. A carefully built prompt with worked examples automated much of the grading of feedback and set a floor estimate for the share of wrong comments, saving staff time and money.
Catalog record
| CREATOR | Marc Ballestero-Ribó, Daniel Ortiz Martínez |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17176
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt-Based Cost-Effective Evaluation and Operation of ChatGPT as a Computer Programming Teaching Assistant Marc Ballestero-Ribó, Daniel Ortiz Martínez Abstract The dream of achieving a student-teacher ratio of 1:1 is closer than ever thanks to the emergence of large language models (LLMs). One potential application of these models in the educational field would be to provide feedback to students in university introductory programming courses, so that a student struggling to solve a basic implementation problem could seek help from an LLM available 24/7. This article focuses on studying three aspects related to such an application. First, the performance of two well-known models, GPT-3.5T and GPT-4T, in providing feedback to students is evaluated. The empirical results showed that GPT-4T performs much better than GPT-3.5T, however, it is not yet ready for use in a real-world scenario. This is due to the possibility of generating incorrect information that potential users may not always be able to detect. Second, the article proposes a carefully designed prompt using in-context learning techniques that allows automating important parts of the evaluation process, as well as providing a lower bound for the fraction of feedbacks containing incorrect information, saving time and effort. This was possible because the resulting feedback has a programmatically analyzable structure that incorporates diagnostic information about the LLM’s performance in solving the requested task. Third, the article also suggests a possible strategy for implementing a practical learning tool based on LLMs, which is rooted on the proposed prompting techniques. This strategy opens up a whole range of interesting possibilities from a pedagogical perspective. 1 Introduction Achieving a 1:1 student-to-teacher ratio holds the promise of deeply alter educational environments. With this personalized approach, teaching methods could be tailored to each student, accommodating the diverse learning styles, abilities, and preferences that exist within a classroom. It would also facilitate timely feedback on assignments, enabling students to learn from their mistakes and make continuous improvements. Unfortunately, having a 1:1 student-to-teacher ratio has been an elusive goal due to the significant resource challenges that would be inherently involved, from the necessity of finding qualified teachers or additional classroom spaces to the tremendous financial impact that implementing this model would have on the limited budgets of educational institutions. One good example of this problematic are the programming courses during the first year of Computer Engineering degrees. Such courses have a crucial role in the basic training of students as future professionals, since for many of them, it is within those courses where they are in contact with the task of programming for the first time. However, the learning process for students within these initial courses is not straightforward. In the first year of a regular Computer Engineering degree, the number of enrolled students can be expected to be very high (e.g., more than 100 students enrolled in the Computer Engineering degree of the Universitat de Barcelona during the 2023-2024 academic year). Furthermore, in this context characterized by a high concentration of students, it would be ideal for each one to be able to solve a large number of basic code implementation exercises (e.g., define a Python function to calculate the factorial of a number given as input), always with the assistance and feedback provided by the course’s faculty. This assistance should be available, not only during teaching sessions or office hours, but also when the students work independently. During that time, if the students encounter difficulties in solving coding problems, they should contact the teaching staff via email. This process may be time-consuming and make the learning less effective. The ideal situation where each student receives constant and personalized assistance is far from being achievable, given that the number of teachers who are available to attend the students’ needs is limited. However, the emergence of artificial intelligence (AI) technologies based on large language models (LLMs) makes it possible to aspire to a new scenario in which learning tools could be offered to students, always within a restricted and controlled environment, acting as virtual teaching assistants for the course and thereby bringing closer the goal of a 1:1 student-to-teacher ratio. These teaching assistants would be available 24/7 to address programming queries from students. The assistance provided by LLMs would aim to guide the student to the solution of programming problems, rather than simply providing the solution itself. There already exist prominent examples of LLMs being used as coding assistants. The most popular of them being GitHub Copilot [ GitHub(2021) ] . GitHub Copilot is an innovative AI-powered coding assistant developed by GitHub in collaboration with OpenAI. It serves as an extension to integrated development environments (IDEs) and assists developers in writing code by providing context-aware code suggestions and completions. However, it is important to stress out that GitHub Copilot would not be aligned with the specific necessities that a virtual teaching assistant of initial undergraduate computer programming courses would need to cover. As it was explained above, the ideal LLM-based assistant that would be required should be focused on guiding the student to the solution instead of providing it. This contrasts with the general purpose of tools like GitHub Copilot, which are more targeted to individuals who are already able to write code and want to increase their productivity. The most prominent LLM technology available so far that has a strong potential to be applied as a teaching assistant is ChatGPT [ Brown et al.(2020) ] . ChatGPT is a chatbot built on top of an LLM technology called generative pre-training transformer [ Radford et al.(2018) ] (GPT). ChatGPT is able to understand and generate human-like text based on the input it receives. ChatGPT has demonstrated a remarkable ability in different tasks, such as answering questions, providing information or engaging in dynamic conversations. However, using ChatGPT as a teaching assistant presents several challenges. One of them would be that ChatGPT is a general purpose model. In spite of the fact that the model’s training data encompasses a vast array of internet text, it cannot be expected to have the course-specific depth and accuracy required for a dedicated teaching assistant role. Its general purpose nature may also hinder its ability to adapt its responses to diverse learning styles or to provide pedagogically sound explanations, where it is of utmost importance that the model stimulates the academic growth of the students instead of merely handing requested information to them. Another challenge to apply ChatGPT as a teaching assistant is the necessity to know what this technology can and cannot do in its present form when applied for that purpose. Although there are countless examples where ChatGPT as well as other LLMs have already demonstrated a remarkable ability to solve complex tasks, their outputs are not immune to errors, biases, or misunderstandings. Evaluating the performance of ChatGPT’s or other LLMs is a challenging task. Ideally, evaluation should be carried out by human experts, generating one or multiple manual evaluation metrics that capture aspects related to the performance or quality of the model’s responses. However, manual evaluation metrics present significant drawbacks, with the main one being their high cost in resources. This problem is even more serious in the extremely dynamic ecosystem of LLMs today, where new models or variants of them requiring evaluation are