Analyzing Undergraduate Problem-Solving in Physics Through Interaction With an AI Chatbot
Syed Furqan Abbas Hashmi, N. Sanjay Rebello · 2025
Read Analyzing Undergraduate Problem-Solving in Physics Through Interaction With an AI Chatbot by Syed Furqan Abbas Hashmi, N. Sanjay Rebello (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14778.txt.
Introduction
Syed Furqan Abbas Hashmi and Sanjay Rebello study how undergraduates solve physics problems with an AI chatbot in a 2025 paper. Giving each student tailored hints at scale is a standing teaching challenge. Their chatbot asks Socratic questions rather than giving answers, and it was used in a large first year mechanics course at a Midwestern public university.
Full chat logs came from 150 first year science majors. Post chat surveys gave the bot solid ratings for knowledge skills and fair ratings overall. Student questions grew far more specific over each chat, from a small share at first to all by the last turn. Specific questions also tracked with higher expected course grades.
Catalog record
| CREATOR | Syed Furqan Abbas Hashmi, N. Sanjay Rebello |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-14778
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Analyzing Undergraduate Problem-Solving in Physics Through Interaction With an AI Chatbot Syed Furqan Abbas Hashmi, N. Sanjay Rebello Abstract Providing individualized scaffolding for physics problem solving at scale remains an instructional challenge. We investigate (1) students’ perceptions of a Socratic Artificial Intellegence (AI) chatbot’s impact on problem-solving skills and confidence and (2) how the specificity of students’ questions during tutoring relates to performance. We deployed a custom Socratic AI chatbot in a large-enrollment introductory mechanics course at a Midwestern public university, logging full dialogue transcripts from 150 first-year STEM majors. Post-interaction surveys revealed median ratings of 4.0/5 for knowledge-based skills and 3.4/5 for overall effectiveness. Transcript analysis showed question specificity rose from approximately 10–15% in the first turn to 100 % by the final turn, and specificity correlated positively with self-reported expected course grade (Pearson r ≈ 0.43 r\approx 0.43 ). These findings demonstrate that AI-driven Socratic dialogue not only fosters expert-like reasoning but also generates fine-grained analytics for physics education research, establishing a scalable dual-purpose tool for instruction and learning analytics. I Introduction Problem solving is widely recognized as the fundamental means by which students demonstrate conceptual mastery and quantitative reasoning in physics Reif1982 ; PhysRevSTPER.10.020119 ; PhysRevPhysEducRes.13.020101 . Yet, despite decades of discipline-based research revealing expert–novice differences in strategic planning, representation use, and metacognitive monitoring Chi1981 ; PhysRevSTPER.4.010111 ; PhysRevSTPER.3.020101 , traditional assessments continue to capture only final answers and sparse work artifacts, rendering the rich cognitive processes invisible PhysRevPhysEducRes.12.010130 ; PhysRevPhysEducRes.18.010109 . This invisibility presents a critical gap: without real-time data on how students engage in problem solving, instructors cannot scaffold expert-like strategies nor provide targeted feedback. Recent advances in Artificial Intellegence (AI) and Natural Language Processing (NLP) offer a powerful solution. Intelligent tutoring systems and AI chatbots have demonstrated near human-level gains in one-on-one settings, often approaching the “two-sigma” benchmark of human tutoring effectiveness VanLehn_2011 ; Graesser2005 ; PhysRevPhysEducRes.21.010153 . Generative-AI tools have been applied to grade explanations and generate feedback with human-comparable accuracy PhysRevPhysEducRes.21.010126 ; PhysRevPhysEducRes.21.010128 , and data-augmentation studies have shown how synthetic responses can train reliable analytic models PhysRevPhysEducRes.19.020150 . However, these innovations primarily focus on aggregate performance metrics or educator training applications PhysRevPhysEducRes.21.010154 ; PhysRevPhysEducRes.20.010152 ; PhysRevPhysEducRes.20.020144 ; PhysRevPhysEducRes.20.010109 ; PhysRevPhysEducRes.19.020163 , leaving a second gap: few studies integrate Socratic dialogue systems that both scaffold metacognitive reflection and capture complete interaction transcripts for fine-grained analytics. To address these gaps, we developed and deployed a custom Socratic AI physics chatbot in an introductory mechanics course at a large U.S. Midwestern university. Our system prompts students with targeted, sequential questions, logs full dialogue transcripts, and applies automated analytics to quantify question specificity and problem-solving progress in real time SIRNOORKAR2024100318 . We investigate two primary research questions: • How do students perceive the chatbot’s impact on their problem-solving skills and confidence? • How does the specificity of students’ questions during the tutoring dialogue relate to their problem-solving progress and performance? By combining AI-driven scaffolding with research-grade data streams, this work advances both classroom pedagogy and PER methodology. II Theoretical Bases Our design builds on established frameworks for expert problem solving and Socratic inquiry. Adams and Wieman’s multi-skill model identifies conceptual understanding, strategic planning, and self-monitoring as core sub-skills in physics problem solving Adams_Wieman_2015 ; Chi_Bassok_Lewis_Reimann_Glaser_1989 . Epistemic-game theory further maps the cognitive moves experts use to structure solutions PhysRevSTPER.3.020101 , while multiple-representation studies highlight the importance of fluid transitions among algebraic, graphical, and pictorial modes PhysRevSTPER.4.010111 . Rubric-based assessments provide reliable snapshots of written work PhysRevPhysEducRes.12.010130 ; PhysRevPhysEducRes.13.020101 , and metacognitive post-reflection exercises yield measurable gains in strategic reasoning PhysRevPhysEducRes.18.010109 ; PhysRevPhysEducRes.13.020101 . Socratic tutoring, which uses targeted questions to elicit self-explanations, has long been shown to enhance learning by revealing and correcting misconceptions Graesser2005 ; Chi_Bassok_Lewis_Reimann_Glaser_1989 . Intelligent tutoring systems that implement Socratic dialogue achieve substantial learning gains in physics and other domains VanLehn_2011 . More recently, PER has begun to explore AI chatbots: Dunlap et al. demonstrated Large Language Model (LLM) performance on kinematics graphs PhysRevPhysEducRes.20.010109 , and Polverini et al. evaluated ChatGPT on the BEMA inventory and concluded that ChatGPT exhibits significant difficulties in engaging with physics tasks involving visual representations. PhysRevPhysEducRes.21.010154 . Fussell et al. compared LLMs for lab-note analysis PhysRevPhysEducRes.21.010128 , and Chen & Wan showed human-level feedback generation on problem-solving explanations PhysRevPhysEducRes.21.010126 . Kieser et al. applied data augmentation with ChatGPT to train analytic models PhysRevPhysEducRes.19.020150 , and Kortemeyer et al. explored AI grading feasibility on handwritten exams PhysRevPhysEducRes.20.020144 ; PhysRevPhysEducRes.19.020163 . Despite these advances, transcript-based dialogue analytics remain scarce. Existing studies tend to focus on performance metrics or teacher training, rather than authentic student–bot interactions parsed for question specificity and linked directly to learning outcomes PhysRevPhysEducRes.21.010153 ; SIRNOORKAR2024100318 . By integrating Socratic scaffolding with automated analytics on complete dialogue transcripts, our work fills this critical methodological and pedagogical gap in PER. III Methods The study took place in an first-semester, calculus-based physics course at a large U.S. Midwestern public university. A total of N=150 first-year STEM majors participated in the study which was conducted in Week 9 of the Spring 2025 semester. At the start of each session, students received a brief orientation to the Socratic chatbot and were instructed: “Interact with this tool just as you would with a human teaching assistant.” Students then individually engaged with the chatbot via a web interface (a Django web application deployed on Heroku) using their own laptops or phones. The chatbot presented a multi-step physics problem scenario and guided the student through it in dialogue. The problem shown in Figure (adapted from a Context Rich Problem Heller_Keith_Anderson_1992 from the University of Minnesota PER group website) involved a human cannonball being launched by a spring – requiring energy conservation, projectile motion, and kinematics concepts to determine if the performer would clear a 15-foot wall. This rich context required students to integrate Hooke’s law, gravitational potential energy, and projectile motion principles. The chatbot was pre-loaded with the problem statement (which students could review at any time) and began by asking the student to consider the given data and “What is your approach?” or similar broad prompts. Figure 1: Problem statement of the context rich problem the students were asked to solve using the chatbot. The AI tutor’s system architecture was designed to emulate an interactive Socratic dialogue while ensuring correctness of the solution path. The backend (Django) maintained session state and managed Application Programming Interface (API) calls to OpenAI’s GPT-4o model. Two GPT-4o instances were used: one as the Instructor which interacted with the stedent and responded to them, and one as the Verifier, which would verify the correctness of the student response. The Instructor is responsible for generating the next question or hint for the student, whereas the Verifier evaluates the student’s latest answer by using ChatGPT’s default knowledge-base. The problem-solving dialogue is structured as a sequence of steps (e.g., conceptual understanding, setting up equations, performing calculations). At each step, the Instructor poses a question. When the student submits an answer, the Verifier agent is invoked to check the answer’s correctness and quality. All student–chatbot conversations were then submitted by the students. After the 50-minute session (or when a student finished earlier), students were directed to complete an online survey about