Understanding the role of autoencoders for stiff dynamical systems using information theory
Vijayamanikandan Vijayarangan, Harshavardhana A. Uranakara, Francisco E. Hernández-Pérez, Hong G. Im · 2025
Read Understanding the role of autoencoders for stiff dynamical systems using information theory by Vijayamanikandan Vijayarangan, Harshavardhana A. Uranakara, Francisco E. Hernández-Pérez, Hong G. Im (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06325.txt.
Introduction
Vijayamanikandan Vijayarangan, Harshavardhana Uranakara, Francisco Hernandez-Perez, and Hong Im use information theory to explain how autoencoders tame stiff dynamical systems. A past study paired an autoencoder with a neural ODE and saw stiffness drop sharply. Their work came out in twenty twenty-five.
The team traces mutual information as the network trains, and finds two clear phases in how coding and decoding learn. Density plots show the trick: rare events in the raw space turn into likely events in the hidden space. That reshaping, done by the nonlinear map, is why the learned surface is smooth and low in dimension.
Catalog record
| CREATOR | Vijayamanikandan Vijayarangan, Harshavardhana A. Uranakara, Francisco E. Hernández-Pérez, Hong G. Im |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06325
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding the role of autoencoders for stiff dynamical systems using information theory Vijayamanikandan Vijayarangan, Harshavardhana A. Uranakara, Francisco E. Hernández-Pérez, Hong G. Im Abstract Using the information theory, this study provides insights into how the construction of latent space of autoencoder (AE) using deep neural network (DNN) training finds a smooth low-dimensional manifold in the stiff dynamical system. Our recent study [ 1 ] reported that an autoencoder (AE) combined with neural ODE (NODE) as a surrogate reduced order model (ROM) for the integration of stiff chemically reacting systems led to a significant reduction in the temporal stiffness, and the behavior was attributed to the identification of a slow invariant manifold by the nonlinear projection of the AE. The present work offers fundamental understanding of the mechanism by employing concepts from information theory and better mixing. The learning mechanism of both the encoder and decoder are explained by plotting the evolution of mutual information and identifying two different phases. Subsequently, the density distribution is plotted for the physical and latent variables, which shows the transformation of the rare event in the physical space to a highly likely (more probable) event in the latent space provided by the nonlinear autoencoder. Finally, the nonlinear transformation leading to density redistribution is explained using concepts from information theory and probability. Journal: arXiv 1 Introduction Many complex physical phenomena such as turbulent chemically-reacting flows are dynamical systems, which are mathematically described by time-dependent nonlinear partial differential equations (PDE). The equations are computationally solved in a discretized form by constructing a system of ordinary differential equations (ODE), and are integrated by the method of lines. The dimensionality of the ODE system is large, proportional to the number of solution variables and the number of computational cells. The large dimensionality is also accompanied by a wide range of time scales, posing a numerical stiffness that requires many small time steps and often forms the bottleneck. As such, the computational demand to simulate a modest physical size of reacting flows can easily reach tens of millions of core hours in modern high performance computing (HPC) hardware. As the physical problems under study become more sophisticated, effective acceleration of simulations by various means of reduced dimensionality and temporal stiffness without loss of fidelity is of paramount importance. The key principle for developing high-fidelity reduced-order models (ROM) is to identify the correct minimal coordinates, referred to as intrinsic low-dimensional manifolds, in the hyper-dimensional phase space [ 2 , 3 ] . In general, the physical state variables are mapped onto the low-dimensional manifolds using a linear or nonlinear transformation, thereby reducing the number of physical state variables and the dynamic range of time scales. The successful development of ROMs allows significantly reduced computational costs and facilitates the physical interpretation of complex phenomena [ 4 ] . The development of ROMs can be broadly classified into two groups: physics-based and data-driven. In the physics-based ROM approach, the governing equations are simplified using assumptions deduced from the observed solution behavior. An earlier example is the modeling of atmospheric flows, where the Lorenz equations [ 5 ] described two-dimensional convective flows using three ODEs. In this approach, the state-space (Poincaré) model linearizes the system at fixed points and constructs models using averaging and homogenization methods [ 6 , 7 ] . For reactive systems, the computational singular perturbation (CSP) [ 8 , 9 ] has been used to identify low-dimensional manifolds to reduce the number of state variables and eliminate unnecessary fast time scales. This approach has been successfully implemented in ODE solvers for the accelerated integration of complex chemically reacting systems [ 10 , 11 ] . With advances in data science and machine-learning algorithms, data-driven ROM development has recently been gaining strong research interests. In this approach, a large amount of generated data from experiments or simulations is further processed to determine a proper transformation map. The method can be broadly classified into (i) Koopman analysis and (ii) neural networks (NN). In the former, a nonlinear coordinate transformation maps a dynamical system to a linear system of observables [ 12 , 13 ] . This has been applied to computational fluid dynamics (CFD) [ 14 , 15 , 16 ] , combined with matrix decomposition tools such as principal component analysis (PCA) and dynamic mode decomposition (DMD) [ 17 ] . PCA has also been used successfully in chemically reacting flow simulations to reduce state variables [ 18 , 19 ] . In the latter approach based on NN, the input data is processed using a series of affine transformations along with the nonlinear activation functions to construct a ROM. This approach has widely been used in CFD for model discovery [ 20 , 21 ] , flow optimization and control [ 22 , 23 ] , and reconstruction of high-resolution flow fields from sparse measurements [ 24 , 25 , 26 , 27 , 28 ] . A variation of the standard deep neural network (DNN) approach for dimensionality reduction is the autoencoder (AE), which maps the physical state variables into the low-dimensional manifolds, referred to as the latent space. Unlike Koopman analysis, AEs offer the flexibility of using either linear or nonlinear transformations to obtain the latent space variables. The reduced variables in the latent space are approximated using time-series forecasting methods (e.g. transformers, neural ODE, recurrent neural networks, etc.), which are recovered back in physical space using a decoder. This three-stage approach is widely adopted in the study of dynamical systems, with notable improvements in ROM fidelity along with computational speedup [ 29 , 30 , 31 , 1 ] . In our recent study [ 1 ] , the AE combined with neural ODE (NODE) was used in a homogeneous reactive system with a large number of reactive scalars, and a significant level of stiffness reduction was achieved through an optimal training strategy to dynamics-informed construction of the latent space. Although AE-based methods have been shown to be an effective tool for developing ROMs with compressed dimensionality and dynamics, the optimal design of AE is based on many fine-tunings of the hyper-parameters, such as the activation function, number of units per layer, the total number of hidden layers, dimension of the latent space, often in a trial-and-error manner. Moreover, there is little understanding of the underlying mathematical or physical explanation of how training leads to finding an optimal latent space with a removal of temporal stiffness. To provide insights into this issue, several mathematical frameworks have been used to understand the mechanism and behavior of the “black box" of neural networks in representing the complex input data. For example, the universal approximation theorem (UAT) was used to determine the validity of approximating any continuous function using NN [ 32 ] . Differential geometry concepts have also been used to understand the non-Euclidean nature of the data learning framework [ 33 ] . Alternatively, linear algebra-based projection theorems were used to explain the process of projection on linear sub-manifolds [ 34 ] , although it was argued that the linear theory is limited to shallow NNs only [ 35 ] . A combination of different mathematical tools was also used to explain the NN-based data learning process. For