Using large language models for solving thermodynamic problems

Rébecca Loubet, Pascal Zittlau, Luisa Vollmer, Marco Hoffmann, Sophie Fellenz, Fabian Jirasek, Heike Leitte, Hans Hasse · 2025

Read Using large language models for solving thermodynamic problems by Rébecca Loubet, Pascal Zittlau, Luisa Vollmer, Marco Hoffmann, Sophie Fellenz, Fabian Jirasek, Heike Leitte, Hans Hasse (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05195.txt.

Introduction

Rébecca Loubet and co-authors grade large language models on thermodynamics in a 2025 paper. The team built a test of 22 problems, from plain drills to hard cases, and ran five known models through it. Trained experts marked the answers much as they would mark a student exam.

Models did well on the plain problems but wobbled on the hard ones. Answers swung between runs, context was often misread, and wrong starting guesses crept in. The authors say pure text statistics strain at this domain logic. They call for a tighter merge of stored field knowledge with language models before such tools can be trusted in class.

Catalog record

CREATORRébecca Loubet, Pascal Zittlau, Luisa Vollmer, Marco Hoffmann, Sophie Fellenz, Fabian Jirasek, Heike Leitte, Hans Hasse
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-05195

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Using large language models for solving thermodynamic problems Rébecca Loubet, Pascal Zittlau, Luisa Vollmer, Marco Hoffmann, Sophie Fellenz, Fabian Jirasek, Heike Leitte, Hans Hasse Abstract Large Language Models (LLMs) have made significant progress in reasoning, demonstrating their capability to generate human-like responses. This study analyzes the problem-solving capabilities of LLMs in the domain of thermodynamics. A benchmark of 22 thermodynamic problems to evaluate LLMs is presented that contains both simple and advanced problems. Five different LLMs are assessed: GPT-3.5, GPT-4, and GPT-4o from OpenAI, Llama 3.1 from Meta, and le Chat from MistralAI. The answers of these LLMs were evaluated by trained human experts, following a methodology akin to the grading of academic exam responses. The scores and the consistency of the answers are discussed, together with the analytical skills of the LLMs. Both strengths and weaknesses of the LLMs become evident. They generally yield good results for the simple problems, but also limitations become clear: The LLMs do not provide consistent results, they often fail to fully comprehend the context and make wrong assumptions. Given the complexity and domain-specific nature of the problems, the statistical language modeling approach of the LLMs struggles with the accurate interpretation and the required reasoning. The present results highlight the need for more systematic integration of thermodynamic knowledge with LLMs, for example, by using knowledge-based methods. Journal: Computers and Chemical Engineering 1 Introduction Thermodynamics is an essential part of both science and engineering, and it is key to understanding energy systems and mitigating climate change. It involves some of the most fundamental principles of physics, such as the first and second laws of thermodynamics, and its applications span almost every field of scientific inquiry. However, while the theoretical foundation of thermodynamics is well-established, applying this knowledge in practice presents a significant challenge. Solving practical thermodynamic problems often involves not only interpreting complex information but also understanding how to adapt and apply these theoretical physical principles to specific scenarios. Currently, most of this work has to be done by human experts, eventually assisted by computational tools, which are, for the most part, designed to solve domain-specific problems. For example, all process simulation tools used in the chemical industry contain hard-coded thermodynamic knowledge. Given the importance of thermodynamics, it is not surprising that its knowledge has also been included in ontologies that were developed for a variety of purposes [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] . We have recently developed a system for solving thermodynamic problems, called KnowTD, that is based on such an ontology [ 9 ] . This knowledge-based system selects the equations that describe a given problem from the ontology and then automatically and accurately solves them. KnowTD currently encompasses only a small part of the thermodynamic knowledge, but it is extendable and demonstrates the feasibility of knowledge-based approaches for flexibly solving thermodynamic problems. Another knowledge-based system is WolframAlpha, a computational engine based on the language Mathematica [ 10 ] that uses symbolic computation and real-time data processing for problem solving and can handle natural language input. It includes some thermodynamic knowledge but can handle only thermodynamic problems from narrow, well-defined classes. Its strength lies in solving the pertinent equations rather than in finding them. A different approach to solving thermodynamic problems has become practically feasible only very recently: Thermodynamic problems can be given as prompts to large language models (LLMs) in the hope of obtaining a correct answer. This approach is explored in the present work. LLMs are sophisticated computational models designed to process and generate human-like text [ 11 ] . They operate by converting text into smaller units called tokens, which are numerical representations of words, subwords, or characters. The models are based on a transformer architecture, which uses self-attention mechanisms to capture relationships and dependencies between tokens across an input sequence [ 12 ] . Through training on large datasets, these models learn to predict the likelihood of the next token given the context, enabling them to perform various tasks such as text generation, summarization, and question answering. Their performance is driven by two primary factors: the size of the training corpus, which influences the diversity of linguistic patterns they can learn, and the number of model parameters, which determine their capacity to encode and process complex relationships. LLMs have advanced significantly in recent years, with several new models launched in 2024, including OpenAI’s GPT-4o [ 13 ] , Meta’s Llama 3.1 [ 14 ] , and Mistral’s le Chat [ 15 ] . Their applications, once centered on language tasks, have rapidly broadened. This includes the domain of problem solving in science and engineering. In the recent past, a number of studies have been published in which LLMs were used in this domain. These studies often have an educational background [ 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 ] , and many of them come from the field of mathematics [ 17 , 25 , 26 , 27 , 28 , 29 ] . A mixed picture emerges from these studies: While in certain applications impressive strengths of the LLMs in problem solving were found, major drawbacks have also become clear: LLMs are by nature statistical models and so are their answers, i.e., prompting the same question generally leads to different answers that are often of significantly differing quality. Furthermore, prompt design greatly affects the quality of the LLMs’ results, which often sound convincing but may be hallucinations [ 25 ] . Thus, verifying LLM outputs is essential [ 17 ] . This need for scrutiny ties into a broader and ongoing debate about the analytical and reasoning skills of LLMs: It is often argued that they lack such skills, e.g., when it comes to graduate-level problems [ 26 ] . However, there are also more differentiated views, such as that of Venkatasubramanian [ 30 ] , who argues that LLMs have what he describes as “geometrical” reasoning skills but lack analytical reasoning capabilities. Recently, Yao et al. [ 31 ] have shown that step-by-step solutions, known as a "tree-of-thought", can be used to improve the problem-solving capacity of LLMs. A concern with LLMs is their opacity. Some of the most powerful LLMs are essentially black-box models, i.e., the publicly available information on their training and mechanisms is limited [ 32 ] . Hence, empirical testing is the only practical approach to exploring the abilities of LLMs for problem solving. This is what we have done in this study for thermodynamics. Previous studies have used LLMs for problem solving in physics, sometimes also including problems from thermodynamics: Pursani et al. [ 33 ] evaluated GPT-3.5-Legacy, GPT-3.5-Turbo, and GPT-4 on the 110 problems from the US fundamentals of engineering environmental exam. K. Wang et al. [ 20 ] used GPT-4 for 40 problems from a college-level engineering physics course, and X. Wang et al. [ 34 ] introduce SciBench, a benchmark of 869 problems in mathematics, chemistry, and physics collected from college-level textbooks, and apply different LLMs for their solution. Other work in the field loosely related to thermodynamics is described in [ 16 , 35 , 36

Read the complete text