Optimization of randomized neural networks for transfer operator approximation

Mohammad Tabish, Stefan Klus · 2026

Read Optimization of randomized neural networks for transfer operator approximation by Mohammad Tabish, Stefan Klus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23689.txt.

Introduction

Tabish and Klus present RaNNDy, a randomized neural network for transfer operators. Hidden weights and biases are randomly set and fixed, with only the output trained, yielding a closed-form solution at far lower cost than optimized networks. But RaNNDy is restricted to its weight and bias selection, which sets the basis functions, so the hidden-layer activation matters. The authors propose optimizing the activation itself while keeping weights fixed, giving a better dictionary. They test it on stochastic differential equations and random walks on graphons.

Catalog record

CREATORMohammad Tabish, Stefan Klus
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32871
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.23689

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimization of randomized neural networks for transfer operator approximation Mohammad Tabish, Stefan Klus Abstract RaNNDy is a randomized neural network architecture for the data-driven approximation of transfer operators associated with complex dynamical systems. The weights and biases of the hidden layers of the network are randomly initialized and kept fixed, only the output layer is trained. This has several advantages over fully optimized neural networks, notably a closed-form solution for the output layer and significantly lower training costs. Despite these advantages, RaNNDy is restricted to the initial selection of weights and biases that parametrize the basis functions required for the operator approximation. Since the basis functions are determined by the activation function, choosing an appropriate activation function for the hidden layers is crucial. In this work, we propose an algorithm that optimizes the activation function itself, while keeping the weights and biases in the randomized neural network fixed, providing a more suitable dictionary. We illustrate the efficacy of the approach using various benchmark problems, including stochastic differential equations and random walks on graphons. 1 Introduction Data-driven methods for the approximation of transfer operators have become an area of intense research, as they enable the analysis of the global behavior of complex dynamical systems from data. Spectral decompositions of these operators can be used to analyze the long-term behavior of dynamical systems, e.g., metastable states of molecules or coherent structures in fluid flows, see, e.g., [ 7 ] for an overview of different applications. Existing data-driven methods such as extended dynamic mode decomposition (EDMD) [ 22 , 8 ] and the variational approach of conformation dynamics (VAC) [ 15 , 16 ] learn a finite-dimensional representation of transfer operators using a fixed set of basis functions, also known as a dictionary. The dictionary could, for instance, comprise monomials, trigonometric functions, Gaussian functions, or indicator functions. The optimal choice of basis functions, however, is highly system-dependent and in general an open problem. Neural networks (NNs) have been used to represent the dictionary required to approximate the operators, and several architectures have been proposed in the literature, see, e.g., [ 10 , 24 , 13 ] . The training of these networks is performed using backpropagation, which iteratively minimizes a loss function to update the networks’ parameters. This approach is computationally expensive for complex networks and can also lead to problems such as slow convergence, numerous local minima, exploding/vanishing gradients, and sensitivity to hyperparameters such as the learning rate [ 4 , 21 ] . Randomized neural networks avoid using backpropagation by fixing the weights and biases of the hidden layers and training only the output layer for a particular task [ 25 , 12 ] , making the training computationally cheaper. Based on this idea, we recently proposed RaNNDy [ 20 ] , a randomized neural network for the data-driven approximation of transfer operators. RaNNDy computes eigenfunctions of the operator directly via a closed-form solution for the output layer. Fixing the hidden layers’ weights and biases, however, leaves no scope for the optimization of the randomized basis. That is, if the randomly generated basis is not suitable, i.e., it is not able to represent the eigenfunctions of the operator accurately, it could lead to poor approximations. Hence, we propose an algorithm to optimize hyperparameters of parametrized activation functions while keeping the parameters of the model fixed. In other words, we optimize the network while maintaining the computational advantages of fixed weights and biases. We use the variational principle to optimize the activation parameters. The main contributions of this work are: 1. We propose a novel algorithm to optimize the randomized basis of RaNNDy by tuning the hyperparameters of the activation function, while keeping the network weights and biases fixed. 2. We present various applications, including the Bickley jet, high-dimensional protein folding processes, and random walks on graphons. The proposed algorithm can be regarded as a compromise between fully randomized and fully trained neural networks for transfer operator approximation. The paper is structured as follows: In Section 2 , we will introduce transfer operators and randomized neural networks. In Section 3 , we will discuss the proposed algorithm to optimize the randomized basis used by RaNNDy. Numerical results will be outlined in Section 4 and a conclusion will be presented in Section 5 . 2 Background We start by briefly introducing transfer operators for reversible and non-reversible dynamical systems and randomized neural networks. 2.1 Transfer operators Transfer operators offer an alternative viewpoint for analyzing stochastic processes in terms of observables and probability densities instead of states of the system. The Koopman operator, for example, describes the evolution of observables and the Perron–Frobenius operator the evolution of the probability densities of the system. These operators are infinite-dimensional but linear representations of the underlying finite-dimensional but nonlinear dynamics. Since we cannot work with infinite-dimensional operators in practice, the goal is to find suitable finite-dimensional approximations of these operators. Let \{X_{t}\} be a stochastic process defined on the state space \mathbb{X}\subset\mathbb{R}^{d} that is governed by a stochastic differential equation (SDE) of the form \mathrm{d}X_{t}=b(X_{t})\hskip 1.00006pt\mathrm{d}t+\sigma(X_{t})\hskip 1.00006pt\mathrm{d}W_{t}, (2.1) where b\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} is the drift term, \sigma\colon\mathbb{R}^{d}\rightarrow\mathbb{R}^{d\times d} is the diffusion term, and W_{t} is a d -dimensional Wiener process. In what follows, let p_{\tau}\colon\mathbb{X}\times\mathbb{X}\rightarrow\mathbb{R} be the conditional probability of X_{t+\tau}=y given X_{t}=x , where \tau is a fixed lag time. Additionally, let L^{r} denote the space of (equivalence classes of) r -Lebesgue integrable functions and L^{r}_{\mu} the corresponding \mu -weighted space, with 1\leq r\leq\infty . Definition 2.1 (Transfer operators) . Let \tau>0 be a fixed lag time. 1. The Perron–Frobenius operator \mathcal{P}^{\tau} is defined by \displaystyle\mathcal{P}^{\tau}\rho(x)=\int_{\mathbb{X}}p_{\tau}(y,x)\hskip 1.00006pt\rho(y)\hskip 1.00006pt\mathrm{d}y. 2. The Koopman operator \mathcal{K}^{\tau} is defined by \displaystyle\mathcal{K}^{\tau}f(x)=\int_{\mathbb{X}}p_{\tau}(x,y)f(y)\hskip 1.00006pt\mathrm{d}y. 3. The reweighted Perron–Frobenius operator \mathcal{T}^{\tau} that propagates densities w.r.t. a reference density \mu is defined by \displaystyle\mathcal{T}^{\hskip 0.81949pt\tau}\hskip 1.00006ptu(x)=\frac{1}{\nu(x)}\int_{\mathbb{X}}p_{\tau}(y,x)u(y)\mu(y)\hskip 1.00006pt\mathrm{d}y, where \nu=\mathcal{P}^{\tau}\mu . It is well known that the eigenvalues \lambda_{i} and the corresponding eigenfunctions \varphi_{i} of these transfer operators—or compositions thereof—can help us identify, for example, metastable states in molecular dynamics or coherent sets in fluid flows [ 14 , 19 , 3 , 8 , 1 ] . Definition 2.2 (Invariant density) . The eigenfunction of \mathcal{P}^{\tau} corresponding to the eigenvalue \lambda=1 is called stationary density or invariant density , usually denoted by \pi , i.e., it satisfies \mathcal{P}^{\tau}\pi=\pi . The operators introduced above are well-defined as \mathcal{P}^{\tau}\colon L^{2}_{1/\pi}\rightarrow L^{2}_{1/\pi} , \mathcal{T}^{\tau}\colon L^{2}_{\pi}\rightarrow L^{2}_{\pi} and \mathcal{K}^{\tau}\colon L^{2}_{\pi}\rightarrow L^{2}_{\pi} , if \pi exists. We refer the reader to [ 9 ] for more details on the domains. Example 2.3 . For the overdamped Langevin equation given by \displaystyle\mathrm{d}X_{t}=-\nabla V(x)\hskip 1.00006pt\mathrm{d}t+\sqrt{2\beta^{-1}}\hskip 1.00006pt\mathrm{d}W_{t}, where V(x) is a confining potential and \beta>0 is the inverse temperature, there exists a unique invariant density given by \pi(x)=e^{-\beta V(x)} . Definition 2.4 (Reversibility) . The process X_{t} is called reversible if the so-called detailed balance condition is satisfied, i.e., \pi(x)\hskip 1.00006ptp_{\tau}(x,y)=\pi(y)\hskip 1.00006ptp_{\tau}(y,x) for all x,y\in\mathbb{X} . For reversible processes, the Koopman operator \mathcal{K}^{\tau} and the Perron–Frobenius operator \mathcal{P}^{\tau} are self-adjoint with respect to suitably reweighted inner products [ 16 ] . However, for non-reversible dynamical systems, they are in general not self-adjoint. That is, their eigenvalues and eigenfunctions may be complex-valued. In this case, we approximate the singular values and singular functions of these

Read the complete text