Transfer Learning of Surrogate Models via Domain Affine Transformation Across Synthetic and Real-World Benchmarks

Shuaiqun Pan, Diederick Vermetten, Manuel López-Ibáñez, Thomas Bäck, Hao Wang · 2025

Read Transfer Learning of Surrogate Models via Domain Affine Transformation Across Synthetic and Real-World Benchmarks by Shuaiqun Pan, Diederick Vermetten, Manuel López-Ibáñez, Thomas Bäck, Hao Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14012.txt.

Introduction

Shuaiqun Pan and four co-authors reuse cheap stand-in models across related tasks in a 2025 paper. A stand-in model mimics a costly real process, but training a good one takes lots of data.

The team asks whether a stand-in trained on one task can serve on another when the two input ranges differ by a simple stretch and shift. Earlier work did this for smooth model types. The authors extend it to random forests, which do not offer the same math handles. They test the method on a standard set of test problems and on four real cases. Reuse cut both the new data needed and the computing cost, which matters when each real run is expensive.

Catalog record

CREATORShuaiqun Pan, Diederick Vermetten, Manuel López-Ibáñez, Thomas Bäck, Hao Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14012

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transfer Learning of Surrogate Models via Domain Affine Transformation Across Synthetic and Real-World Benchmarks Shuaiqun Pan, Diederick Vermetten, Manuel López-Ibáñez, Thomas Bäck, Hao Wang Abstract Surrogate models are frequently employed as efficient substitutes for the costly execution of real-world processes. However, constructing a high-quality surrogate model often demands extensive data acquisition. A solution to this issue is to transfer pre-trained surrogate models for new tasks, provided that certain invariances exist between tasks. This study focuses on transferring non-differentiable surrogate models (e.g., random forests) from a source function to a target function, where we assume their domains are related by an unknown affine transformation, using only a limited amount of transfer data points evaluated on the target. Previous research attempts to tackle this challenge for differentiable models, e.g., Gaussian process regression, which minimizes the empirical loss on the transfer data by tuning the affine transformations. In this paper, we extend the previous work to the random forest and assess its effectiveness on a widely-used artificial problem set - Black-Box Optimization Benchmark (BBOB) testbed, and on four real-world transfer learning problems. The results highlight the significant practical advantages of the proposed method, particularly in reducing both the data requirements and computational costs of training surrogate models for complex real-world scenarios. PubID: pubid: 979-8-3315-3431-8/25/$31.00 ©2025 IEEE I Introduction Surrogate modeling [ 1 , 2 , 3 , 4 ] is commonly utilized across various fields of computer science and engineering, particularly for complex simulations or physical experiments where data acquisition is expensive. Common learning algorithms for building surrogate models include artificial neural networks [ 5 ] , support vector machine [ 6 , 7 ] , random forests [ 8 , 9 ] and Gaussian process regression [ 10 , 11 , 12 ] . These learning algorithms often have a high sample complexity, especially when the dimension of the independent variable becomes large. As such, when surrogate-modeling a new regression task, one might wish to exploit surrogate models trained on previous tasks, given similarities and symmetries between tasks. Domain shift is a prominent type of task similarity, where the probability distribution of the input variable shifts from one task to the other while the predictive distribution remains the same. Under this assumption, one can transfer a model from the source task to the target task by learning the domain shift. In this work, we focus on a specific type of domain shift - affine shift - the input distributions between tasks are related by an unknown affine transformation (rotation and translation). A prior study [ 13 ] proposed a transfer learning method for differential models, e.g., Gaussian process regression. In this work, we propose to extend this transfer learning method to non-differentiable models, for instance, random forests. Source function Target function Original RFR Trained with 50 samples Transferred with 50 samples Trained with 100 samples Transferred with 100 samples Fig. 1 : Synthetic transfer learning tasks are constructed by transferring from different instances of the same BBOB functions (F3 Rastrigin is taken in this example). From left to right, we show the source function f^{\text{S}} , the target function f^{\text{T}} , the original RFR \hat{f}^{\text{S}} trained to approximate f^{\text{S}} , the RFR trained directly on 50 samples of f^{\text{T}} , and the original RFR transferred with the same 50 samples. We also show the RFR models with 100 sample points. We illustrate our approach in Fig. 1 with a random forest regression (RFR) model on an artificial function taken from BBOB. The first three columns show the contour lines of the source function, the target function, and the model \hat{f}^{\text{S}} , respectively. The next two columns show the contour lines of the RFR model trained from scratch with 50 sample points evaluated on f^{\text{T}} and the transferred model. The last two columns display the situation with 100 sample points. As the target function is essentially a “rotated” variant of the source function, the transferred model demonstrates a significant advantage over the one trained from scratch, achieving markedly lower symmetric mean absolute percentage error (SMAPE). Our contributions are: • We extend the transfer learning method in [ 13 ] to handle non-differentiable surrogates, e.g., random forest regression. • We verify the effectiveness of our extension on the Black-Box Optimization Benchmark suite and four real-world transfer learning problems. • We conduct an empirical analysis to investigate when the transferred model outperforms the model trained from scratch on the transfer dataset. II Related Works II-A Transfer learning in RFR surrogate modeling Segev et al. [ 14 ] introduced two distinct algorithms for transferring RFR. The first method employs a greedy search strategy to iteratively refine the structure of each decision tree. This approach enables localized modifications, such as expanding or pruning individual nodes, to improve the model’s adaptability. The second method, in contrast, preserves the original tree structure but focuses on recalibrating the parameters, specifically by adjusting the decision thresholds at the internal nodes. II-B Transfer learning with affine model Saralajew et al. [ 15 ] proposed a manifold-based adaptation technique for classification models, such as generalized learning vector quantization classifiers (GLVQ) [ 16 , 17 ] , to address distribution shifts between datasets. The affine transformation was also considered to minimize discrepancies between source and target domains [ 18 ] . Mandl et al. [ 19 ] applied affine transformations within the framework of Physics-Informed Neural Networks (PINNs). III Affine Transfer Learning for Non-differentiable Models Consider a source regression task to approximate function f^{\text{S}}\colon\mathbb{R}^{d}\to\mathbb{R} and a target task f^{\text{T}}\colon\mathbb{R}^{d}\to\mathbb{R} . We assume an affine symmetry between them: there exist \mathbf{W}\in\operatorname{SO}\left(d\right) and \mathbf{v}\in\mathbb{R}^{d} such that f^{\text{T}}(\mathbf{x})=f^{\text{S}}(\mathbf{W}\mathbf{x}+\mathbf{v}) , where \operatorname{SO}\left(d\right) denotes the d -dimensional rotation group. In practice, the affine parameters \mathbf{W} and \mathbf{v} are unknown. Given a high-quality model \hat{f}^{\text{S}} trained on f^{\text{S}} , we wish to transfer it to f^{\text{T}} by affine reparameterization \hat{f}^{\mkern 1.0mu\text{T}}(\mathbf{x})\coloneqq\hat{f}^{\text{S}}(\mathbf{W}\mathbf{x}+\mathbf{v}) , where the unknown parameters are determined by minimizing the empirical loss of \hat{f}^{\mkern 1.0mu\text{T}} on a small transfer data set \mathcal{T} evaluated on f^{\text{T}} , namely \mathcal{L}(\mathbf{v},\mathbf{W})=n_{\text{T}}^{-1}\sum_{\mathbf{x}\in\mathcal{T}}(\hat{f}^{\text{S}}\left(\mathbf{W}\mathbf{x}+\mathbf{v}\right)-f^{\text{T}}(\mathbf{x}))^{2} . If the surrogate model \hat{f}^{\text{S}} is differentiable, e.g., GPR, we can employ the Riemannian gradient algorithm to minimize the loss [ 13 ] . For non-differential models, e.g., RFR, we propose the following optimization procedure. We propose to use Covariance matrix adaptation evolution strategy (CMA-ES) [ 20 , 21 , 22 ] to tune the affine parameters. CMA-ES can be applied directly to the search space of the translation parameter \mathbf{v} , which is Euclidean. However, special treatment is needed for \mathbf{W} , which lives in a smooth manifold \operatorname{SO}\left(d\right) . To solve this issue, we consider the Lie group representation \mathfrak{so}(d)=\{\mathbf{A}\in\mathbb{R}^{d\times d}\colon\mathbf{A}^{\top}=-\mathbf{A}\} , which is a flat space (with dimension d(d-1)/2 ), and optimize this representation with CMA-ES. A rotation matrix \mathbf{W} can be recovered its representation \mathbf{A} with the exponential map, i.e., \mathbf{W}=\operatorname{Exp}(\mathbf{A}) . For each search point \mathbf{z}\in\mathbb{R}^{d(d-1)/2} , we have to transform it into a d\times d antisymmetric matrix to preserve the structure of \mathfrak{so}(d) : the components in \mathbf{z} are sequentially assigned to the upper triangular of \mathbf{A} (diagonal entries are zero) row by row. The lower triangular entries are then filled by the negative value of the transposition of the upper triangular. We summarize

Read the complete text