REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs

Sourav Mishra 1 3 , Shreya Hallikeri 4 5 and Suresh Sundaram 2 3 · 2025

Read REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs by Sourav Mishra 1 3 , Shreya Hallikeri 4 5 and Suresh Sundaram 2 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02267.txt.

Introduction

Sourav Mishra, Shreya Hallikeri, and Suresh Sundaram design a new activation curve for physics-informed networks. Their paper, from twenty twenty-five, presents REAct. Such networks solve physics problems by learning, but training often stalls. The activation curve inside each unit shapes how well learning travels across problems.

Their curve bends a classic shape with four learnable dials. That lets one curve fit many physical systems. On heat problems, error fell to a thousandth of the classic curve's level. It also held up on finer grids and points beyond the training zone. Inverse tasks with noisy data gave cleaner parameter guesses as well.

Catalog record

CREATORSourav Mishra 1 3 , Shreya Hallikeri 4 5 and Suresh Sundaram 2 3
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-02267

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

REAct: Rational Exponential Activation for Better Learning and Generalization in PINNs Sourav Mishra 1 3 , Shreya Hallikeri 4 5 and Suresh Sundaram 2 3 Abstract Physics-Informed Neural Networks (PINNs) offer a promising approach to simulating physical systems. Still, their application is limited by optimization challenges, mainly due to the lack of activation functions that generalize well across several physical systems. Existing activation functions often lack such flexibility and generalization power. To address this issue, we introduce Rational Exponential Activation (REAct), a generalized form of tanh consisting of four learnable shape parameters. Experiments show that REAct outperforms many standard and benchmark activations, achieving an MSE three orders of magnitude lower than tanh on heat problems and generalizing well to finer grids and points beyond the training domain. It also excels at function approximation tasks and improves noise rejection in inverse problems, leading to more accurate parameter estimates across varying noise levels. I Introduction and Background Physics-Informed Neural Networks (PINNs) have recently made significant progress in modeling physical systems by incorporating physical laws, expressed as ordinary and partial differential equations (ODEs and PDEs), into the training process as soft constraints. Traditional methods for simulating physical systems are either data-driven, which struggles with sparse or noisy data, or numerical, which are computationally expensive. PINNs offer a compromise between the two, being able to learn from sparse data unlike data-driven methods and more computationally efficient than numerical approaches [ 1 , 2 ] . As a result, PINNs have been successfully applied to model fluid flow [ 3 ] , heat flow [ 4 ] , control system design [ 5 , 6 ] , electromagnetics [ 7 ] , nano-optics and metamaterials [ 8 ] and several other areas. However, despite their wide applicability, PINNs face optimization challenges due to the use of PDE-based loss functions, which can lead to ill-conditioned training [ 9 , 10 , 11 ] . Activation functions are a key factor affecting PINN optimization as they determine how well the network captures the underlying dynamics of a physical system [ 12 ] . Activation functions play a key role in neural networks by introducing non-linearity, enabling models to capture complex patterns, making them an important research area [ 13 ] . In continuous settings like those in PINNs, selecting activation functions becomes even more critical, as it impacts the network’s ability to represent complex physical signals [ 14 , 15 , 16 ] . The optimal activation function is often problem-specific; for instance, recent studies show that hyperbolic tangent can lead to instability in simulating certain dynamics [ 14 ] , while sinusoidal functions offer smoother optimization [ 15 ] . Therefore, the dynamics of the system being modeled plays a key role in choosing a suitable activation function for PINNs [ 14 , 17 ] , unlike supervised learning problems where Relu is universally used regardless of the modality of the data [ 18 , 19 ] . Vibrating systems might exhibit resonance, leading to unusually high output magnitudes, making normalization techniques of little use in modeling the dynamics. For PINNs to accurately model physical systems, their activation functions must meet crucial requirements: they need to be smooth and continuously differentiable to handle physics-informed loss functions, which involve higher-order derivatives. Additionally, the function should allow unbounded outputs, unlike tanh and sin, which are restricted between -1 and 1. Furthermore, activation functions must avoid saturation to prevent vanishing gradients, which hinder learning [ 10 ] . As conventional activation functions used in classification tasks don’t satisfy the above criteria, recent PINN literature is taking a data-driven approach to design more generalizable activation functions [ 20 , 21 , 22 ] . These methods include learnable parameters in activation functions, improving convergence rate and solution quality [ 20 , 22 ] by dynamically altering the loss landscape during training. The Adaptive Blending Unit (ABU) PINNs [ 17 ] optimizes the search for the best activation function at each layer by learning a convex combination of a pre-selected set of standard activation functions, allowing the network to capture system-specific features. Despite their benefits, the above activations offer limited control over key shape characteristics like zero crossings, frequency, saturation regions, and convexity, restricting their generalization capabilities. The performance of ABU-PINNs is further constrained by the learnable convex combination of a fixed set of activations chosen a-priori. Better performance comes at a higher computational cost by including more diverse activation functions in the set. Additionally, it may be desirable to have controlled saturation beyond some range to reject the influence of noise in estimating system parameters from noisy sensor data in inverse problems. To overcome these issues, we introduce Rational Exponential Activation (REAct), a more generalized version of the hyperbolic tangent with four learnable parameters. REAct improves PINN performance on forward problems, reducing MSE by 3 orders of magnitude on the heat problem and generalizing well to finer grids and to points outside the training domain. It also captures the variability in complicated functions, leading to better function approximation accuracy, and demonstrates accurate parameter estimation in inverse problems across a range of noise levels. II Method II-A Physics Informed Neural Networks (PINNs) Consider an Initial Boundary Value Problem (IBVP) described by a Partial Differential Equation (PDE) \mathcal{N}(u,x,t)=0 over the domain (x,t)\in[0,L]\times[0,T] , with known initial conditions u(x_{i},t=0) at points \{x_{i}\}_{i=1}^{N_{I}} , and boundary conditions at x=0 and x=L for time instants \{t_{b}\}_{b=1}^{N_{B}} . The PINN approximates the solution of the IBVP u(x,t) as \hat{u}(x,t) . u(x,t) may be known at the points \{(x_{d},t_{d})\}_{d=1}^{N_{D}} . To enforce the dynamics, the physics loss is computed as the PDE residual at collocation points \{(x_{c},t_{c})\}_{c=1}^{N_{C}} . The model outputs must also satisfy initial and boundary conditions while matching the given data. Each component has its respective loss given below: \displaystyle\mathcal{L}_{phy}=\dfrac{1}{N_{C}}\sum_{c=1}^{N_{C}}(\mathcal{N}(\hat{u}_{c},x_{c},t_{c}))^{2} (1) \displaystyle\mathcal{L}_{IC}=\dfrac{1}{N_{I}}\sum_{i=1}^{N_{I}}(\hat{u}(x_{i},t=0)-u(x_{i},t=0))^{2} (2) \displaystyle\mathcal{L}_{BC}=\dfrac{1}{N_{B}}\sum_{b=1}^{N_{B}}(\hat{u}(x=0,t_{b})-u(x=0,t_{b}))^{2} \displaystyle+(\hat{u}(x=L,t_{b})-u(x=L,t_{b}))^{2} (3) \displaystyle\mathcal{L}_{data}=\dfrac{1}{N_{D}}\sum_{d=1}^{N_{D}}(\hat{u}(x_{d},t_{d})-u(x_{d},t_{d}))^{2} (4) The total loss is a weighted sum of these components: \mathcal{L}=\lambda_{p}\mathcal{L}_{phy}+\lambda_{I}\mathcal{L}_{IC}+\lambda_{B}\mathcal{L}_{BC}+\lambda_{d}\mathcal{L}_{data} (5) where \lambda_{p},\lambda_{I},\lambda_{B},\lambda_{d} are hyperparameters controlling the weight of each loss term. II-B Rational Exponential Activation (REAct) Our motivation for designing more generalizable activation functions for PINNs is based on the following points: • They must be smooth and differentiable for calculating the physics loss. • Outputs should be unbounded, as the magnitude of PINN outputs is unknown in advance. • Control over key shape properties (e.g., zero crossings, frequency, convexity) to adapt to several signals and mitigate noisy measurements in inverse problems. • More flexibility and fewer learnable parameters than ABU-PINN. Common activations lack these features, particularly in controlling shape properties. Since tanh and sin are widely used in PINNs [ 20 , 15 ] , arise in the solution of many PDEs and ODEs, and are related to the exponential family, we introduce a generalized activation called Rational Exponential Activation (REAct) with four learnable shape parameters: REAct(x)=\dfrac{1-\exp(ax+b)}{1+\exp(cx+d)} (6) REAct meets these criteria and is in the same form as the activations in [ 20 , 22 ] . Therefore, PINNs using REAct avoid suboptimal critical points, as proven in [ 20 , 22 ] . Figure 1 shows REAct for various values of its shape parameters, showing it is more flexible than STan [ 22 ] . Fig. 1: REAct for different values of

Read the complete text