FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs

Saeed Mohammadzadeh Department of Mechanical EngineeringBoston UniversityBoston, MA [email protected], Erfan Hamdi Department of Mechanical EngineeringBoston UniversityBoston, MA [email protected] Joel Shor 1. Move37 Labs2. Department of Mechanical Engineering, Boston Univeristy [email protected] Emm · 2025

Read FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs by Saeed Mohammadzadeh Department of Mechanical EngineeringBoston UniversityBoston, MA [email protected], Erfan Hamdi Department of Mechanical EngineeringBoston UniversityBoston, MA [email protected] Joel Shor 1. Move37 Labs2. Department of Mechanical Engineering, Boston Univeristy [email protected] Emm (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20732.txt.

Introduction

Saeed Mohammadzadeh, Erfan Hamdi, Joel Shor, and Emma Lejeune built a test suite called FEM-Bench. It checks whether large language models can write correct code for computational mechanics, the field that models how physical systems respond to forces and deformation. The tasks follow the finite element method taught in a first graduate course.

Thirty-three tasks cover basic but real modeling challenges with clear right answers. Even so, top models did not solve them reliably across five tries each. The strongest function writer finished thirty tasks at least once, while unit test writing lagged well behind. FEM-Bench gives researchers a firm yardstick for scientific coding skill.

Catalog record

CREATORSaeed Mohammadzadeh Department of Mechanical EngineeringBoston UniversityBoston, MA [email protected], Erfan Hamdi Department of Mechanical EngineeringBoston UniversityBoston, MA [email protected] Joel Shor 1. Move37 Labs2. Department of Mechanical Engineering, Boston Univeristy [email protected] Emm
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE123 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20732

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs Saeed Mohammadzadeh Department of Mechanical EngineeringBoston UniversityBoston, MA [email protected], Erfan Hamdi Department of Mechanical EngineeringBoston UniversityBoston, MA [email protected] Joel Shor 1. Move37 Labs2. Department of Mechanical Engineering, Boston Univeristy [email protected] Emma Lejeune Department of Mechanical EngineeringBoston UniversityBoston, MA [email protected] Abstract As LLMs advance their reasoning capabilities about the physical world, the absence of rigorous benchmarks for evaluating their ability to generate scientifically valid physical models has become a critical gap. Computational mechanics, the discipline that develops and applies mathematical models and numerical methods to predict the behavior of physical systems under forces, deformation, and constraints, provides an ideal foundation for structured scientific reasoning based evaluation. Problems follow clear mathematical structure, enforce strict physical and numerical constraints, and support objective verification. The discipline also requires constructing explicit models of physical systems and reasoning about geometry, spatial relationships, and material behavior, which connects directly to emerging goals in AI related to physical reasoning and world modeling. We introduce FEM-Bench, a computational mechanics benchmark designed to evaluate the ability of LLMs to generate correct finite element method (FEM) and related code. FEM-Bench 2025 contains a suite of introductory but nontrivial tasks aligned with material from a first graduate course on computational mechanics. These tasks capture essential numerical and physical modeling challenges while representing only a small fraction of the complexity present in the discipline. Despite their simplicity, state-of-the-art LLMs do not reliably solve all of them. In a five attempt run, the best performing model at function writing, Gemini 3 Pro, completed 30/33 tasks at least one out of five times, and 26/33 tasks five out of five times. The best performing model at unit test writing, GPT-5, had an Average Joint Success Rate of 73.8%. Other popular models showed a broad range of performance on the benchmark. FEM-Bench establishes a structured foundation for evaluating AI-generated scientific code, and future iterations will incorporate increasingly sophisticated tasks to track progress as models evolve. Keywords Large Language Models (LLMs) \cdot Finite Element Method (FEM) \cdot LLM Benchmark \cdot Scientific Machine Learning \cdot Computational Mechanics \cdot Scientific Computing \cdot Code Generation 1 Introduction Modern AI systems are increasingly evaluated on their ability to build internal models of the physical world, including reasoning about forces, geometry, spatial relationships, and the mechanical behavior of physical systems ( 8 ; 69 ) . Simultaneously, large language models (LLMs) show growing potential to assist with scientific and engineering workflows, including reasoning about physical systems and automated generation of simulation and analysis code ( 18 ) . As these models advance, a central question emerges: can LLMs produce implementations of physics-based numerical methods that are correct, reliable, and scientifically meaningful? Existing code-generation benchmarks test general programming logic ( 6 ; 13 ) , software engineering skills ( 36 ) , and mathematical reasoning ( 23 ) , but no widely used benchmarks evaluate an LLM’s ability to carry out the physical modeling, numerical discretization, and structured computational reasoning required for advanced scientific computing ( 66 ) . Computational mechanics, the discipline that formulates and solves mathematical models of physical systems using numerical methods, provides a natural and rigorous domain in which to investigate these capabilities. Computational-mechanics–based tasks represent a particularly important frontier for LLM capabilities ( 34 ) . Physics-based simulations grounded in this discipline underpin real-world applications in domains as diverse as robotics ( 31 ) , digital twins of aircraft and human systems ( 49 ) , climate modeling ( 19 ) , and engineering design and optimization ( 65 ) . Across these areas, several themes are consistent: successful simulation requires the seamless integration of mathematics, physics, geometry, numerical methods, and programming, and correctness is non-negotiable. A program that runs but produces a non-convergent, unstable, or physically impossible result is of no scientific value ( 53 ) . For this reason, computational mechanics has a long and mature tradition of verification, validation, and uncertainty quantification ( 50 ) , providing structured, quantitative tools for assessing whether a model is numerically consistent, physically plausible, and predictive. These practices make the domain especially well-suited for evaluating LLM-generated code, because failures are interpretable, quantitatively measurable, and can be traced not only to programming or structural errors but also to deeper mistakes in implementing governing equations, enforcing numerical consistency, handling geometric transformations, or managing floating-point–sensitive operations such as numerical integration and matrix assembly ( 5 ; 2 ) . Thus, this branch of scientific computing provides a well-structured and scientifically grounded testbed for critically evaluating the emerging capabilities of LLMs. Problems in computational mechanics follow a precise mathematical pipeline: e.g., governing equations are formulated, converted into weak or variational forms, discretized into finite-dimensional approximations, and assembled into global algebraic systems ( 17 ; 68 ) . Solutions are then validated through objective numerical checks such as symmetry, consistency, and mesh convergence ( 71 ) . From the perspective of LLM reasoning, these tasks span multiple forms of structured cognition: hierarchical reasoning in which local element routines combine into global models; geometric transformations across coordinate systems; the construction of explicit models of physical systems; and careful handling of numerical integration, stability, and floating-point–sensitive operations. Moreover, the computational mechanics literature is extensive and mature. The computational mechanics literature spans classic textbooks ( 32 ) , widely taught graduate curricula in engineering ( 22 ; 60 ) , and ongoing research at the frontier of numerical methods development ( 37 ) , offering a rich, well-understood space of problems with clear expectations for correctness and implementation quality ( 64 ) . In this work, we introduce FEM-Bench, a benchmark designed to evaluate the ability of LLMs to generate correct implementations of finite element method (FEM) and related computational mechanics tasks. FEM-Bench 2025 focuses on introductory but nontrivial tasks aligned with material from a first graduate course on FEM. These tasks isolate essential numerical and physical modeling challenges while remaining amenable to automated evaluation. Despite their relative simplicity, we find that state-of-the-art LLMs show significant room for improvement, revealing significant gaps between current model capabilities and the requirements of physics-based scientific computing. The contributions of this paper are threefold. First, we present the FEM-Bench framework, including its design principles and scope. Second, we provide a suite of tasks and corresponding evaluation tools grounded in canonical computational mechanics concepts. Third, we conduct a baseline evaluation across several leading LLMs to characterize current performance. Looking forward, FEM-Bench establishes a foundation for increasingly challenging computational mechanics tasks in future iterations, enabling systematic tracking of progress in scientific computing specific code generation. 2 Background This Section provides background on computational mechanics as a domain for evaluating LLMs, emphasizing physical grounding, algorithmic structure, and rigorous verification culture. We then motivate the need for FEM-Bench as a benchmark designed to probe these capabilities in a scientifically meaningful setting. 2.1 Computational Mechanics as a Test of Physical World Modeling Computational mechanics provides the mathematical and numerical foundation for simulating the behavior of physical systems under forces, deformation, and constraints ( 10 ; 26 ) . At its core, the field translates physical laws into solvable mathematical models, which are then discretized and implemented as algorithms suitable for computation ( 41 ) . In

Read the complete text