Continual Learning Strategies for 3D Engineering Regression Problems: A Benchmarking Study

Kaira M. Samuel, Faez Ahmed · 2025

Read Continual Learning Strategies for 3D Engineering Regression Problems: A Benchmarking Study by Kaira M. Samuel, Faez Ahmed (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12503.txt.

Introduction

Kaira Samuel and Faez Ahmed benchmark ways for engineering models to keep learning as data change in a 2025 paper. Design work produces costly data in small amounts, and new designs keep arriving. Retraining a model from scratch each time is too expensive, yet naive updates make models forget what they learned before.

The team applied continual learning methods to five engineering datasets and built nine new test setups for three-dimensional regression tasks. Plain continual methods already beat naive baselines. A replay method, which revisits stored old examples, matched full retraining on several tests while cutting training time nearly in half. Code and data will be released. The study opens a path for design tools that improve over time.

Catalog record

CREATORKaira M. Samuel, Faez Ahmed
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12503

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Continual Learning Strategies for 3D Engineering Regression Problems: A Benchmarking Study Kaira M. Samuel, Faez Ahmed Abstract Engineering problems that apply machine learning often involve computationally intensive methods but rely on limited datasets. As engineering data evolves with new designs and constraints, models must incorporate new knowledge over time. However, high computational costs make retraining models from scratch infeasible. Continual learning (CL) offers a promising solution by enabling models to learn from sequential data while mitigating catastrophic forgetting, where a model forgets previously learned mappings. This work introduces CL to engineering design by benchmarking several CL methods on representative regression tasks. We apply these strategies to five engineering datasets and construct nine new engineering CL benchmarks to evaluate their ability to address forgetting and improve generalization. Preliminary results show that applying existing CL methods to these tasks improves performance over naive baselines. In particular, the Replay strategy achieved performance comparable to retraining in several benchmarks while reducing training time by nearly half, demonstrating its potential for real-world engineering workflows. The code and datasets used in this work will be available at: https://github.com/kmsamuel/cl-for-engineering-release . 1 Introduction Machine learning (ML) can be leveraged in engineering applications to advance research in areas such as aerodynamic prediction, rapid prototyping, and design optimization. Traditionally, these tasks have relied on computationally expensive and time-intensive simulations, including computational fluid dynamics (CFD) simulations and finite element analysis (FEA). To address these limitations, ML methods have been developed to obtain field variable predictions more quickly by circumventing these time-consuming processes [ 1 ] . In particular, trained ML models can replace costly high-fidelity engineering simulations in a process called surrogate modeling . In engineering applications, obtaining high-fidelity predictions of continuous values (i.e., drag coefficients, stresses, thermal loads) is essential, making regression a key focus. Such engineering regression tasks can span from predicting aerodynamic quantities, such as drag coefficients from car [ 2 ] [ 3 ] or airplane geometries [ 4 ] , to computing material properties of meta-material designs [ 5 ] . In all of these applications, successful surrogate models define precise mappings between input features and continuous target values. However, this success is highly dependent on access to large, high-quality training datasets that cover the full design space, which are often expensive to obtain in engineering contexts. Furthermore, data availability is rarely static in real-world design pipelines. Engineering design processes are inherently dynamic: as new data, constraints, or performance targets emerge, shifting data distributions, models must adapt accordingly. Numerous examples illustrate the importance of continual updates. A digital twin, for example, aims to mirror a physical system’s state in real time and must ingest streaming data as the system changes [ 6 , 7 ] . Likewise, a surrogate model for rapid aerodynamics or structural prediction may need to expand its valid range whenever new design candidates or simulation regimes are introduced. This poses a challenge for traditional surrogate modeling approaches, which typically assume a fixed training set. Standard practice would require merging old and new data, and then retraining from scratch, which can be computationally expensive and prone to data management challenges [ 8 ] . Here, continual learning offers a promising alternative by allowing models to update incrementally and learn from new data more efficiently than through retraining, mitigating catastrophic forgetting (where the model’s performance on old data degrades as it encounters new data). This stands in stark contrast to the conventional train-once-and-deploy paradigm, which is increasingly mismatched to the evolving nature of modern engineering workflows. However, most continual learning research focuses on classification in domains like computer vision or NLP, leaving regression tasks and engineering problems under-explored. This paper explores how classical CL strategies— such as rehearsal-based methods (Experience Replay, Gradient Episodic Memory) and parameter regularization (Elastic Weight Consolidation)— perform in these challenging engineering regression scenarios. We motivate three types of task sequences that commonly arise and assemble a suite of engineering datasets to benchmark two of them. By systematically comparing CL approaches, we reveal best practices, limitations, and prospects for seamlessly integrating CL into real engineering workflows. The main contributions of this paper are summarized as follows: 1. Introduce continual learning to engineering regression tasks, specifically for surrogate modeling. 2. Propose continual learning scenarios tailored explicitly for regression tasks: bin incremental , input incremental , and multi-target incremental , aligning with realistic engineering data generation workflows. 3. Benchmark several established continual learning strategies, including Experience Replay (ER), Elastic Weight Consolidation (EWC), and Gradient Episodic Memory (GEM), across five representative 3D engineering datasets and two regression continual learning scenarios, creating nine distinct benchmarks to facilitate systematic evaluation. 4. Demonstrate empirically that adapting existing continual learning methods can significantly mitigate catastrophic forgetting and improve generalizability in engineering surrogate modeling tasks compared to naive incremental training. 5. Analyze the effectiveness of each strategy, highlighting the impact of data representation, dataset size, and dimensionality on performance, thereby providing practical guidelines for the integration of continual learning in engineering design processes. 2 Preliminaries In this section, we discuss the empirical motivations and background knowledge relevant to the continual learning setting. We first provide an example of the problem on one of our engineering benchmarks and subsequently define important terminology in continual learning. (a) (b) Figure 1 : Catastrophic forgetting is demonstrated using the car drag prediction problem from the DrivAerNet dataset. A surrogate model is trained on each data batch as the batches incrementally become available. Fig. 1(a) shows a KDE plot of the final model’s predictions on the entire dataset, which consists of all batches. As seen, the model exclusively predicts values within the third and fourth batches, demonstrating a lack of retention of earlier batch samples. Fig 1(b) emphasizes this discrepancy by reporting the percent error in predictions for each batch. Problem Demonstration The DrivAerNet dataset is used to demonstrate the problems associated with naively training a model on dynamic data (see Figure 1 ). The dataset consists of geometric car geometries as inputs and their CFD-obtained coefficient of drag values as target values. In this scenario, the data becomes available in batches, which is an expected aspect of data generation cycles. In the extreme case, we can assume that the new batches of data do not overlap in their drag value distributions; thus, the new batches of data contain increasing drag values within their distribution range, depicted in Figure 1(a) . A model is trained to predict the drag value from the input car geometry. It is trained on each batch incrementally to simulate new data becoming available at different times, and the final model is evaluated, with the kernel density estimate (KDE) distributions for the truth values (solid curve) and predicted values (perforated curve) plotted in Figure 1(a) . We observe that the earlier batches of data have a much higher percent error than the final batch (batch 4), as seen in Figure 1(b) . These results demonstrate the phenomenon of catastrophic forgetting, the key problem targeted by the domain of continual learning. Thus, we demonstrate a need for better suited strategies of retaining old distributions in the engineering context without requiring expensive retraining on all previous data. What is Continual Learning? Continual learning—the ability to learn from a stream of tasks or

Read the complete text