From Intuition to Understanding: Using AI Peers to Overcome Physics Misconceptions
Ruben Weijers, Denton Wu, Hannah Betts, Tamara Jacod, Luke Yuxiang Guan, Vidya Sujaya, Kushal Dev, Toshali Goel, William Delooze, Reihaneh Rabbany, Ying Wu, Jean-François Godbout, Kellin Pelrine · 2025
Read From Intuition to Understanding: Using AI Peers to Overcome Physics Misconceptions by Ruben Weijers, Denton Wu, Hannah Betts, Tamara Jacod, Luke Yuxiang Guan, Vidya Sujaya, Kushal Dev, Toshali Goel, William Delooze, Reihaneh Rabbany, Ying Wu, Jean-François Godbout, Kellin Pelrine (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00408.txt.
Introduction
Ruben Weijers and a large team of co-authors tested an 'AI Peer' for physics teaching in twenty twenty-five. Students often hold stubborn misconceptions about Newtonian mechanics. Instead of an authoritative tutor, their AI peer is openly fallible: students are told it may answer up to forty percent of questions wrong. The idea is that arguing with a peer-like partner builds independent thinking.
In a randomized trial with one hundred sixty-five students, those who debated the AI peer on target concepts gained over ten points more on post-tests than a control group discussing physics history. Ninety-one percent of the peer chats were rated helpful. Gains did not hinge on the AI being right, suggesting the dialogue itself drives learning. The study sketches a new role for AI in classrooms.
Catalog record
| CREATOR | Ruben Weijers, Denton Wu, Hannah Betts, Tamara Jacod, Luke Yuxiang Guan, Vidya Sujaya, Kushal Dev, Toshali Goel, William Delooze, Reihaneh Rabbany, Ying Wu, Jean-François Godbout, Kellin Pelrine |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-00408
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Intuition to Understanding: Using AI Peers to Overcome Physics Misconceptions Ruben Weijers, Denton Wu, Hannah Betts, Tamara Jacod, Luke Yuxiang Guan, Vidya Sujaya, Kushal Dev, Toshali Goel, William Delooze, Reihaneh Rabbany, Ying Wu, Jean-François Godbout, Kellin Pelrine Abstract Generative AI has the potential to transform personalization and accessibility of education. However, it raises serious concerns about accuracy and helping students become independent critical thinkers. In this study, we designed a helpful AI “Peer” to help students correct fundamental physics misconceptions related to Newtonian mechanic concepts. In contrast to approaches that seek near-perfect accuracy to create an authoritative AI tutor or teacher, we directly inform students that this AI can answer up to 40% of questions incorrectly. In a randomized controlled trial with 165 students, those who engaged in targeted dialogue with the AI Peer achieved post-test scores that were, on average, 10.5 percentage points higher—with over 20 percentage points higher normalized gain—than a control group that discussed physics history. Qualitative feedback indicated that 91% of the treatment group’s AI interactions were rated as helpful. Furthermore, by comparing student performance on pre- and post-test questions about the same concept, along with experts’ annotations of the AI interactions, we find initial evidence suggesting the improvement in performance does not depend on the correctness of the AI. With further research, the AI Peer paradigm described here could open new possibilities for how we learn, adapt to, and grow with AI. 1 Introduction Students have recently been exposed to the remarkable capabilities of Generative AI (AI) in education (AIED). For example, OpenAI’s ChatGPT has been reported to successfully support teaching preparation, assessment design and grading, and student learning ( Lo, 2023 ) . Systems like ChatGPT show potential to save time and enhance teaching and learning, including critical and higher-order thinking tasks ( Lo, 2023 ) . However, there is limited concrete evidence whether these tools are effective at improving student learning outcomes ( Samson R, 2025 ) . In fact, LLMs are well-reported to hallucinate information and provide sycophantic answers ( Wei et al., 2023 ; Perez et al., 2023 ) , suggesting that AI could actually be detrimental to student learning if it introduces inaccuracies and biases into classroom materials. Given that an increasingly large number of students are now using AI to help them with their school assignments ( Foundation, 2024 ) , there is a growing concern that students must be taught ‘AI literacy’ to recognize and evaluate potential errors generated by LLMs ( Wineburg & Ziv, 2024 ) . Although recent techniques such as Retrieval-Augmented Generation ( Piktus et al., 2020 ) have improved the veracity of generated content, experts disagree on whether hallucinations will be consistently preventable even in the future ( Samson R, 2025 ) —as they are clearly not today. Thus, there is an urgent need to determine whether AIED can be an effective tool for education despite these inherent limitations. This research evaluates the potential of LLMs to support student learning through text-based conversations in an introductory university physics class. The study explores how LLMs can be used to facilitate learning of physics concepts, positioning the AIED as a non-expert peer rather than an expert teacher. Students complete a modified Force Concept Inventory (FCI) as a pre-test. Our AI Peer is prompted with the mistakes made by each student, resulting in a personalized discussion focused on the identified misconceptions. Students then completed the standard FCI as a post-test that asesses the same concepts via a different questionnaire. Our results demonstrate that the treatment group, after a focused discussion with the AI, experienced significantly higher learning gain between tests compared to the control group, who discussed irrelevant content (a quiz on historical figures in physics) with the AI Peer. Importantly, our AI, which leverages the full abilities of GPT-4o without any artificial reduction, was not a reliable source of truth; when tested, it answered up to 40% of the FCI questions incorrectly. Thus, this study marks a first step towards achieving a comprehensive individualized approach to AI-assisted education by aligning human expectations to current limitations of generative AI. Our key contributions include: • An experimental framework for measuring AI’s educational capabilities. In our experiment, we create a new FCI 1 1 1 https://github.com/ComplexData-MILA/AI4Ed and focus on physics. The framework can be extended to other domains and other AI systems. • Showing that AI can help correct common, deep-seated physics misconceptions that persisted over the first half of a semester of traditional instruction. • A “peer” rather than “instructor” approach that we find preserves learning, while lowering barriers in how much AI accuracy is needed. With further research, this could potentially expand the areas in which we can benefit from AI, while empowering the development of human critical thinking skills. 2 Literature Review Educators, policymakers, EdTech companies, and students are generally optimistic about the potential for AI to deliver personalized education, despite the limited evidence that these tools can be effective at improving student learning outcomes ( Samson R, 2025 ) . In a systematic review of 113 papers relating to AI Education tools, Chiu et al. (2023) found that, while AIED can generally improve student motivation and output, methods used to evaluate AIED were often ineffective at measuring students’ learning. The authors suggested that further research is needed to “devise new methods for evaluating the success of AI systems.” They also noted that, despite the potential of AIED to provide equitable education through personalized feedback, AI could potentialy worsen educational inequity if the AIED design process lacked consideration of pedagogy and learning sciences. Pedagogical research has identified numerous teaching strategies with the potential to significantly accelerate student learning, as described by Hattie (2009) in a synthesis of meta-analyses on student achievement. Among them are discussion with another student (argumentation), and prompt oriented or directional feedback. Baidoo-Anu & Ansah (2023) confirm that AI tools can enhance teaching and learning experiences by supporting personalized and interactive learning. For instance, students could leverage the capabilities of advanced generative AI to systematically explain complex concepts. Mollick & Mollick (2023) suggest several teaching strategies that could potentially enhance student learning in the presence of AI while mitigating the associated risks of these technologies. Their research emphasizes the importance of maintaining human involvement in the educational process and positioning AI as a supportive tool rather than a substitute for human instructors. As de Jong et al. (2023) argue, an AIED that combines direct instruction with inquiry should be more effective at explaining new concepts than having the AI directly answer student questions. This is precisely the approach taken by Jurenka et al. (2024) , who tested an LLM-based support tool with a class of 113 Arizona State University students. The tool, HallMate, could discuss course videos, direct learners to relevant content, provide scaffolded homework help, and assist with time management and broad learning strategies. The study tracked whether students used HallMate, how they used it, and whether students felt it was useful. It also evaluated the pedagogical quality of the conversations. While promising, this study focused on usability and engagement but did not attempt to measure the impact of AI tools on students’ learning or performance. To summarize, existing studies on