AI Reasoning Models for Problem Solving in Physics

Amir Bralin, N. Sanjay Rebello · 2025

Read AI Reasoning Models for Problem Solving in Physics by Amir Bralin, N. Sanjay Rebello (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20941.txt.

Introduction

Amir Bralin and N Sanjay Rebello grade a reasoning AI model on textbook physics problems, in a 2025 study. Reasoning models are a new class of language models built for hard, multi-step work. The team fed OpenAI's o3-mini five tries at each of 408 problems from twenty chapters of a standard undergraduate text. In all, 2,040 solutions were checked by hand.

The model solved ninety-four percent of the problems. It excelled in early mechanics topics. Later chapters on waves and heat proved harder. The authors present the sweep as a yardstick for how far such models have come in physics, and where they still slip.

Catalog record

CREATORAmir Bralin, N. Sanjay Rebello
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20941

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AI Reasoning Models for Problem Solving in Physics Amir Bralin, N. Sanjay Rebello Abstract Reasoning models are the new generation of Large Language Models (LLMs) capable of complex problem solving. Their reliability in solving introductory physics problems was tested by evaluating a sample of n = 5 n=5 solutions generated by one such model—OpenAI’s o3-mini —per each problem from 20 chapters of a standard undergraduate textbook. In total, N = 408 N=408 problems were given to the model and N × n = 2,040 N\times n=2,040 generated solutions examined. The model successfully solved 94% of the problems posed, excelling at the beginning topics in mechanics but struggling with the later ones such as waves and thermodynamics. I Introduction Traditional education in physics emphasizes solving well-structured, mathematical word problems also called story problems Jonassen 2003 , since they are commonly embedded within a shallow story context. At the undergraduate level, story problems in physics start as some standard textbook, end-of-chapter problems and exercises. At the graduate level, they continue playing an important role in student learning, typically as a part of problem sets assigned by the instructors. There are well-known issues with this kind of problem solving: “When solving these problems, students are not motivated to search for underlying concepts, but rather are encouraged to look locally for formulas and worked-out examples and then do plug-and-chug to get a correct answer Ding et al. 2011 .” These issues get exacerbated as technology advances and offers new ways to look up a given problem’s solution. In this paper, we assume that using powerful technology for the sake of learning how to solve story problems in physics is unavoidable, and we explore how it can be done safely and reliably. Artificial Intelligence (AI) systems are becoming increasingly sophisticated, yet they still exhibit limitations in their reasoning, numerical accuracy, and evaluative capacity—the issues so crucial for problem solving in physics. One study explored the problem-solving capability of popular AI-based software ChatGPT by OpenAI Wang et al. 2024 . In total, 40 real-world physics problems were given to its user interface and the resulting output evaluated. Of all problems, 24 were under-specified in terms of numerical values given in the problem statement. ChatGPT failed to make reasonable assumptions about the physical situation described in those problems and estimate the appropriate quantities. That is, only 2 out of 24 under-specified problems were solved. For the rest of the problems which were specified by the authors, in contrast, ChatGPT solved 10 (out of 16). In addition, in both categories of the problems used, ChatGPT committed many calculation mistakes. This shows the key limitation of the underlying language model which ChatGPT at the time was based on: it is designed for handling words and sentences, not numbers and formulae. Another study also investigated the problem-solving capability of ChatGPT but on a smaller scale Kieser and Wulff 2024 . A single mechanics problem about a point mass sliding down a circular track was given to the user interface as input. Ten runs were conducted to obtain output along with its statistical variation. As a result, ChatGPT solved the problem correctly only 5 out of 10 times. To study its output more systematically, the authors identified four phases of problem solving: (1) problem representation, (2) strategy selection, (3) execution, and (4) evaluation. They found that, by default, ChatGPT did not engage in the last phase—evaluating the generated solution to meet the problem’s original goal—albeit successfully engaging in the other three. This could be improved, they noted, by adding a verbal instruction called user prompt to the input: “In particular, describe whether your solution is plausible and and for what reasons you chose your solution.” I.1 Background The language models powering such software as ChatGPT are called Large Language Models (LLMs). They synthesize vast Internet data to produce intelligible linguistic output. The more data is used in this process and the more computational effort is exerted, the more “intelligent” this technology becomes. Moreover, crossing some threshold in processing so much information may suddenly unlock certain model capabilities such as translating languages or computer coding, akin to phase transitions Huberman and Hogg 1987 . As a result, the AI industry appears to make progress unexpectedly for the general public. LLMs, as artificial neural networks Bengio et al. 2021 , simply generate the most likely strings of characters in a text, following a given input. Stable sequences of characters such as prefixes and suffixes, and even some short words, alongside standard characters such as the alphabet letters, numerals, and punctuation symbols, which occur most often in English, are treated as the units of LLM input/output called tokens . All unique tokens stored in a model’s memory comprise its “vocabulary.” For the state-of-the-art models, this vocabulary size may be quite large: OpenAI’s GPT-3.5, which ChatGPT was based on when it was first released on Nov. 30, 2022 OpenAI 2022 , had reportedly around 50k tokens in total. Later versions have increased this number to 100k and beyond Yang et al. 2024 . Advanced LLMs are tested against certain benchmarks commonly accepted in the AI research community. For example, FrontierMath Glazer et al. 2024 sets the standard for mathematical capability and GPQA (Google-Proof Q&A) Rein et al. 2023 evaluates the models on physics, chemistry, and biology. LLMs loosely referred to as reasoning models OpenAI nda achieve impressive scores on these benchmarks, resulting in a computer-generated output indistinguishable from human text. I.2 Motivation Knowing what the advanced AI models are capable of will inform educators about their potential harm as well as benefit in using them for teaching and learning physics. In problem solving, with every progressive step that AI takes, it must be reliable to a high degree in order to be used in the classroom. The goal of this study was to evaluate the reasoning models in the context of introductory physics. Our working definition of reliability was the ability of a model to solve correctly and repeatedly story problems on a given physics topic. The model performance by individual topic provides a more precise view of the relevant model properties when compared with typical single-number benchmarks used in the broader AI community. Our guiding questions were: 1. How reliable are AI reasoning models when solving story problems in physics? 2. What is the distribution of the story problem-solving ability of AI reasoning models across standard topics in physics? As presented in the following sections, the state-of-the-art AI reasoning models may be reliable problem-solvers in the beginning topics of a typical introductory physics course, but still struggle with solving problems in later topics such as waves and thermodynamics. II Methods Among various available Large Language Models (LLMs), the family of reasoning models called the “o-series” by OpenAI was selected for this study due to the popularity of ChatGPT. This family started with a preview of its first reasoning model o1 OpenAI 2024 . Soon, it included an updated version o3 working at full capacity, as well as a “lightweight” version o3-mini . Specifically, the model o3-mini OpenAI 2025a was used in this study due to its affordability and its availability at the time of data collection and analysis. The access was provided by OpenAI’s

Read the complete text