On the Evaluation of Engineering Artificial General Intelligence
Sandeep Neema, Susmit Jha, Adam Nagel, Ethan Lew, Chandrasekar Sureshkumar, Aleksa Gordic, Chase Shimmin, Hieu Nguygen, Paul Eremenko · 2025
Read On the Evaluation of Engineering Artificial General Intelligence by Sandeep Neema, Susmit Jha, Adam Nagel, Ethan Lew, Chandrasekar Sureshkumar, Aleksa Gordic, Chase Shimmin, Hieu Nguygen, Paul Eremenko (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10653.txt.
Introduction
Sandeep Neema and co-authors propose how to grade artificial intelligence that designs physical things, in a 2025 paper. Their target is engineering AGI: agents meant to handle the broad work of designing systems and their controllers, software aside. Such an agent needs recalled facts, tool fluency, knowledge of standard parts and design families, and creative transfer between problems. Grading that bundle is itself a hard problem.
The framework adapts Bloom's taxonomy of learning to engineering design. It builds a rich ladder of test questions, from method knowledge to real design problems. Crucially, grading can inspect structured artifacts like CAD and system models, not just text answers. The authors offer the design as an open, pluggable benchmark path for the field.
Catalog record
| CREATOR | Sandeep Neema, Susmit Jha, Adam Nagel, Ethan Lew, Chandrasekar Sureshkumar, Aleksa Gordic, Chase Shimmin, Hieu Nguygen, Paul Eremenko |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-10653
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Evaluation of Engineering Artificial General Intelligence Sandeep Neema, Susmit Jha, Adam Nagel, Ethan Lew, Chandrasekar Sureshkumar, Aleksa Gordic, Chase Shimmin, Hieu Nguygen, Paul Eremenko Abstract We discuss the challenges and propose a framework for evaluating engineering artificial general intelligence ( eAGI ) agents. We consider eAGI as a specialization of artificial general intelligence (AGI), deemed capable of addressing a broad range of problems in the engineering of physical systems and associated controllers. We exclude software engineering for a tractable scoping of eAGI and expect dedicated software engineering AI agents to address the software implementation challenges. Similar to human engineers, eAGI agents should possess a unique blend of background knowledge (recall and retrieve) of facts and methods, demonstrate familiarity with tools and processes, exhibit deep understanding of industrial components and well-known design families, and be able to engage in creative problem solving (analyze and synthesize), transferring ideas acquired in one context to another . Given this broad mandate, evaluating and qualifying the performance of eAGI agents is a challenge in itself and, arguably, a critical enabler to developing eAGI agents. In this paper, we address this challenge by proposing an extensible evaluation framework that specializes and grounds Bloom’s taxonomy - a framework for evaluating human learning that has also been recently used for evaluating LLMs - in an engineering design context. Our proposed framework advances the state of the art in benchmarking and evaluation of AI agents in terms of the following: (a) developing a rich taxonomy of evaluation questions spanning from methodological knowledge to real-world design problems; (b) motivating a pluggable evaluation framework that can evaluate not only textual responses but also evaluate structured design artifacts such as CAD models and SysML models; and (c) outlining an automatable procedure to customize the evaluation benchmark to different engineering contexts. 1 Introduction Engineering the physical world remains among the most cognitively and technically demanding human endeavors. Designing aircraft, electric vehicles, energy systems, robots, or climate control infrastructure requires not only a deep understanding of physical laws but also the ability to model, analyze, and optimize systems that are tightly coupled across disciplines. Unlike software engineering, where modularity and abstraction have enabled powerful automation, physical systems engineering remains largely manual, iterative, and expertise-driven. Engineers must reason across thermal, structural, electrical, and control domains; often with incomplete data, uncertain constraints, and evolving design goals. To mitigate this complexity, the industry has embraced tools such as Computer-Aided Engineering (CAE) and Model-Based Systems Engineering (MBSE) frameworks. These tools simulate physical behavior, guide architecture decisions, and improve verification workflows. Yet their adoption is limited by steep learning curves and reliance on domain-specific heuristics. Even with these tools, design iteration remains time-consuming and error-prone, especially when novel architectures or trade-offs must be explored. Recent advances in AI and deep learning, particularly in Large Language Models (LLMs) and agentic AI frameworks, offer promising avenues to reduce this burden. Early applications of LLMs include code generation for simulation scripts, language interfaces for CAE tools, and documentation assistants. However, these tools essentially serve as procedural copilots. They improve interaction efficiency but do not exhibit the kind of domain reasoning, multi-physics modeling, or design iteration that is necessary for true engineering collaboration. To realize transformative gains, role of AI in engineering must evolve beyond interface-level assistance and become an active, reasoning partner in the design process. This evolution requires a new class of AI agents – eAGI agents – that can engage in creative problem-solving, simulate physical systems, evaluate trade-offs, and produce artifacts such as structured designs, models, or control strategies. eAGI agents are not just retrieval systems; they must be capable of reasoning with constraints, synthesizing cross-domain knowledge, and adapting learned principles to novel problems. A central challenge in developing such systems is the absence of robust, structured, and context-aware methods for evaluating engineering intelligence. Unlike general NLP benchmarks or vision tasks with fixed outputs, engineering problems are context dependent and often have no single “correct” answer. Performance must be assessed across different levels of reasoning and creativity, taking into account how well an eAGI agent understands the design context, satisfies system constraints, and explores the solution space across multiple domains of physics and engineering. 2 Engineering Artificial General Intelligence ( eAGI ) eAGI aims at cognitive automation of what a human engineer does, that is, eAGI is a specialization of AGI designed to perform expert-level reasoning, analysis, and creative synthesis for physical systems engineering. Similar to human engineers, eAGI agents must combine factual and methodological knowledge with tool fluency, domain-specific modeling intuition, and adaptive, iterative, reasoning-guided design thinking. An eAGI agent should exhibit the following characteristics: • Background knowledge and recall: It should retrieve relevant engineering laws, formulas, heuristics, and design rules when prompted. • Process and tool familiarity: It should understand workflows in CAE and MBSE environments and be able to interface with simulation pipelines or design models. It should be aware of available tools, understand their limitations, configuration requirements, and constraints, be able to generate valid inputs to these tools, and ingest their output. • Contextual understanding: It should recognize components, systems, and recurring patterns from industry-standard designs in a domain, such as heat exchangers or propeller assemblies. • Creative and adaptive reasoning: It should explore novel design solutions, adapt knowledge to unfamiliar systems, and optimize performance across competing objectives. • Collaborate and communicate: It should participate and collaborate with human engineers in design activities through communication that is natural, domain, and context-aware. The goal of eAGI is cognitive automation, so we explicitly exclude surrogate modeling and approximating detailed simulation or analysis tools and instead expect eAGI to be able to call such tools or their machine learning accelerated surrogates. We expect eAGI to be able to distill key design drivers from requirements, postulate one or more solutions, and conduct first-order design sizing using tools when needed. This multifaceted skillset differentiates eAGI from both narrow AI applications in engineering (e.g., automated meshing, parametric sweeps) and general-purpose LLMs (use in generative and predictive auto-completion of artifacts). While current LLMs excel at natural language understanding and code generation, they fall short in areas where structured physical modeling, precise constraints, qualitative and quantitative cross-domain causal reasoning is required. Given the breadth of tasks expected of an eAGI agent, evaluating its capabilities becomes a foundational challenge . Existing benchmarks either test shallow factual knowledge (e.g., textbook questions) or rely on rigid, domain-specific problems with fixed outputs. Neither of these approaches captures the range of reasoning, modeling, and synthesis skills needed for effective engineering intelligence. We propose a comprehensive evaluation framework for eAGI agents that: 1. Specializes Bloom’s taxonomy of human learning outcomes for engineering contexts, providing a structured hierarchy of cognitive abilities from factual recall to reflective, cross-domain design. 2. Introduces a rich taxonomy of evaluation questions—spanning from low-level methods to high-level system synthesis requiring end-to-end problem solving, that can be instantiated across multiple engineering problem domains. 3. Supports pluggable and adaptive benchmark customization, where evaluations can be dynamically generated and tailored to different systems (e.g., eVTOL propulsion, HVAC design) and physics domains (e.g., thermal, electrical, structural). Our framework is designed to evolve alongside the growing capabilities of eAGI offering a scalable and extensible methodology to quantify engineering competence. It