Evaluating the AI-Lab Intervention: Impact on Student Perception and Use of Generative AI in Early Undergraduate Computer Science Courses CCS: Social and professional topics Computing education
Ethan Dickey, Andres Bejarano, Rhianna Kuperus, Bárbara Fagundes · 2025
Read Evaluating the AI-Lab Intervention: Impact on Student Perception and Use of Generative AI in Early Undergraduate Computer Science Courses CCS: Social and professional topics Computing education by Ethan Dickey, Andres Bejarano, Rhianna Kuperus, Bárbara Fagundes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00100.txt.
Introduction
Four authors evaluate a short teaching unit that shows new computer science students how to use generative AI tools in a careful, learning-first way. The unit, called AI-Lab, combines a pre-lab orientation, in-class critique of AI outputs, and a homework reflection. It ran across two semesters in three computing courses and one first-year engineering course at a United States university, with paired before-and-after surveys from more than 800 students plus focus groups.
Openness grew for asking the tools conceptual questions and for homework help, and comfort rose for study, debugging, and homework uses. Reported use on graded work did not rise overall, though debugging use did. In focus groups, students described checking answers more skeptically and setting clearer limits for themselves. The authors argue that brief, structured teaching can shift habits without driving heavier use.
Catalog record
| CREATOR | Ethan Dickey, Andres Bejarano, Rhianna Kuperus, Bárbara Fagundes |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00100
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating the AI-Lab Intervention: Impact on Student Perception and Use of Generative AI in Early Undergraduate Computer Science Courses CCS: Social and professional topics Computing education Ethan Dickey, Andres Bejarano, Rhianna Kuperus, Bárbara Fagundes Abstract . Background and Context. Generative AI (GenAI) is becoming increasingly common in computer science education, yet its effects on student learning, skill development, and perceptions remain relatively underexplored. Despite widespread concerns about overreliance on GenAI, few studies have systematically examined how structured scaffolding can shift students’ engagement with GenAI tools in formal educational settings. Objectives. We investigate whether and how a dedicated “AI-Lab” intervention, which emphasizes scaffolding and mindful usage, shapes students’ understanding, perceptions, and applications of GenAI tools in undergraduate computer science and first-year engineering courses. Our main goal is to determine whether students develop a more deliberate approach to using GenAI that preserves and enhances essential skill building. Method. Over three regular semesters at Purdue University, we implemented the AI-Lab in two mandatory Data Structures and Algorithms courses, one elective Competitive Programming course, and one first-year engineering course, yielding 831 valid student responses. Using a mixed-methods approach, we collected and analyzed quantitative survey data (pre- and post-intervention) as well as qualitative focus group transcripts to capture both broad trends and in-depth perspectives. Findings. While the overall frequency of GenAI usage for homework or programming projects remained largely stable, we observed large effect sizes in comfort and openness across conceptual, debugging, and homework problems. Notably, usage patterns for debugging also shifted statistically significantly, reflecting students’ more mindful and deliberate approach. Focus group discussions corroborated these results, suggesting that the intervention “bridged the gap” between naive GenAI usage and more nuanced, reflective integration of AI tools into coursework, ultimately heightening students’ awareness of their own skill development. Implications. Our findings underscore that structured scaffolding – such as the AI-Lab – can help students harness GenAI’s benefits without undermining core problem-solving abilities. By demonstrating how student perceptions and practices evolve through targeted intervention, we provide evidence-based guidance for educators seeking to integrate GenAI effectively and responsibly into computing curricula, and we highlight directions for further research on GenAI-supported pedagogy. 1. Introduction The integration of AI into computer science education has been well documented, evolving alongside advancements in both technical and theoretical aspects of the field ( Denny et al. 2024 ; Tlili 2024 ; Sentance and Waite 2022 ; Donaire et al. 2025 ) . However, the widespread availability of generative AI (GenAI) tools has significantly disrupted academic settings. In particular, large language model-based tools enable students to generate human-like text based on prompted instructions, raising new pedagogical and ethical challenges ( Lau and Guo 2023 ; Becker et al. 2023 ; Finnie-Ansley et al. 2022 ; Moradi Dakhel et al. 2023 ) . The academic community responded quickly by examining the capabilities of these tools, assessing their impact on established educational practices, and exploring both the potential benefits and challenges associated with their use in learning environments ( Kerslake et al. 2025 ; Hassan et al. 2025 ; Raihan et al. 2025 ) . Existing guidance on integrating GenAI in education primarily adopts a broad academic perspective, focusing on classroom applications. Common strategies emphasize equipping students with the skills to critically evaluate the accuracy and efficacy of AI-generated content while fostering discussions on ethical considerations. Bull et al. propose a pedagogical approach to computing education that introduces students to how professional software developers leverage GenAI in industry ( Bull and Kharrufa 2024 ) . Their framework underscores the importance of cultivating problem-solving skills through scaffolding and fading techniques, iterative assignment modifications, and a structured, gradual immersion process. To support AI literacy in computing education, the SEAME framework has been introduced, structuring lessons around five key dimensions: social impacts, ethical considerations, application of AI, mechanisms of AI functionality, and the evaluation of AI systems ( Harrison 2025 ) . Similarly, researchers at Aalto University in Finland have explored the use of GenAI to provide personalized learning experiences and automated feedback in computing education. Their approach emphasizes training students to write effective prompts and leveraging AI tools to improve programming skills ( Logacheva et al. 2024 ) . Several studies highlight the necessity of educating students on the ethical implications and limitations of GenAI ( Ozkaya and Schmidt 2024 ; Ai 2025 ) . Key concerns include safeguarding sensitive information, recognizing the potential inaccuracies of AI-generated content, understanding AI’s capacity to produce harmful material, and adhering to platform-specific terms of use. In the spirit of protecting the integrity of CS education, we use the AI-Lab Intervention, first introduced by ( Dickey et al. 2024 ) in 2023 on the preprint server arXiv.com and published the following year, in second-year, collegiate computer science and first-year engineering courses and provide the first empirical evaluation of its effectiveness over the course of two semesters. We study it using a mixed methods model with pre- and post-surveys as well as focus groups, and present our findings in Section 4 . Detailed breakdown of student population and courses can be found in Section 3.1 . 1.1. AI-Lab The AI-Lab framework ( Dickey et al. 2024 ) offers a structured approach for instructors to introduce GenAI to students in a pedagogically effective manner. Grounded in Vygotsky’s More Knowledgeable Other theory, this framework positions the instructor as an expert in the course’s technical content while acknowledging the evolving nature of GenAI tools. Rather than assuming absolute authority over AI applications, instructors are encouraged to engage with these tools in the classroom at the same level as students, providing guidance and scaffolding to facilitate meaningful learning experiences. Figure 1 . AI-Lab Framework for integrating GenAI in programming courses ( Bejarano et al. 2025 ) . AI-Lab Framework aimed at faculty for integrating GenAI in programming courses. The AI-Lab framework is structured into multiple stages (see Figure 1 from ( Bejarano et al. 2025 ) ) to facilitate a guided exploration of GenAI in education. The first stage (Instructor Preparation) involves the instructor selecting an AI tool and a relevant topic, allowing them to assess the nature and accuracy of the tool’s outputs. Ideally, the chosen topic should expose AI limitations, producing incorrect or ambiguous responses to prompt critical analysis. In the second stage (Prelab), students engage with offline preparatory materials designed to familiarize them with fundamental AI concepts, prompting strategies, and the generation of scholarly content. At this stage, students also complete an anonymous survey to report their prior usage and perceptions of AI tools in their learning process. The third stage (In-class Lab) consists of an in-class intervention in which the instructor integrates AI into the lesson to generate responses to basic definitions, examples, and analytical scenarios. This demonstration is followed by a structured discussion on the appropriate and ethical use of AI in education. Students are then encouraged to experiment independently with the AI tool by posing more complex queries related to the course topics. Peer discussion is central to this process, as students share their findings, critique incorrect AI-generated outputs, and collaboratively propose corrections. In the fourth and final stage (Post-Lab), students complete a homework assignment that requires them to apply AI tools to solve a set of problems. They must provide