Efficient PINNs via Multi-Head Unimodular Regularization of the Solutions Space

Pedro Tarancón-Álvarez, Pablo Tejerina-Pérez, Raul Jimenez, Pavlos Protopapas · 2025

Read Efficient PINNs via Multi-Head Unimodular Regularization of the Solutions Space by Pedro Tarancón-Álvarez, Pablo Tejerina-Pérez, Raul Jimenez, Pavlos Protopapas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12116.txt.

Introduction

Pedro Tarancón-Álvarez, Pablo Tejerina-Pérez, Raul Jimenez, and Pavlos Protopapas present a 2025 framework to make Physics-Informed Neural Networks work better. Such networks, called PINNs, are used to solve non-linear differential equations, but stiff and multiscale cases stay hard to solve.

Their plan trains a network with many heads, so it learns a whole space of solutions at once, not just one case. A second step, called Unimodular Regularization, shapes the latent space of those solutions. The pair speeds transfer to new cases and helps with inverse problems, where the goal is to work back from data to causes.

Catalog record

CREATORPedro Tarancón-Álvarez, Pablo Tejerina-Pérez, Raul Jimenez, Pavlos Protopapas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12116

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient PINNs via Multi-Head Unimodular Regularization of the Solutions Space Pedro Tarancón-Álvarez, Pablo Tejerina-Pérez, Raul Jimenez, Pavlos Protopapas Abstract Non-linear differential equations are a fundamental tool to describe different phenomena in nature. However, we still lack a well-established method to tackle stiff differential equations. Here we present a machine learning framework to facilitate the solution of nonlinear multiscale differential equations and, especially, inverse problems using Physics-Informed Neural Networks (PINNs). This framework is based on what is called multi-head (MH) training, which involves training the network to learn a general space of all solutions for a given set of equations with certain variability, rather than learning a specific solution of the system. This setup is used with a second novel technique that we call Unimodular Regularization (UR) of the latent space of solutions. We show that the multi-head approach, combined with Unimodular Regularization, significantly improves the efficiency of PINNs by facilitating the transfer learning process thereby enabling the finding of solutions for nonlinear, coupled, and multiscale differential equations. keywords AI, PINNs, Transfer Learning, Multi-head, Differential Geometry 1 Introduction Non-linear differential equations (DEs) describe complex systems in nature, such as fluid dynamics, climate modeling, and general relativity, to mention a few. While linear equations are widely applicable, non-linear equations often arise in systems where interactions or feedback loops create more intricate and unpredictable behaviors. Yet, in most cases, analytic solutions are either impossible or highly complex, necessitating the use of numerical methods to provide solutions Pinney (1955) . With the advent of deep neural networks, particularly Physics-Informed Neural Networks (PINNs), it has become increasingly feasible to solve complex DEs more efficiently. Extensive literature now exists on how to use PINNs to solve a wide variety of nonlinear, multiscale DEs including those in complex, real-world systems where traditional numerical methods may struggle. An added layer of complexity arises when the goal is to solve inverse problems, where the objective is to infer unknown parameters or functions within a system based on available data, such as initial or boundary conditions Khalique et al. (2014) . Using PINNs to solve DEs offers several advantages over traditional methods. For instance, a PINN can be trained on a specific range of boundary conditions (BCs) or initial conditions (ICs). The specific solution(s) to a DE can only be determined if the corresponding ICs or BCs are specified as part of the problem. The existence and uniqueness theorem Coddington and Levinson (1984) ensures that there is a unique solution for a set of DEs once specific ICs or BCs are chosen. In traditional numerical methods, numerical integration is performed for each set of ICs-BCs. If multiple IC-BVs are given, the numerical integration must be run separately for each one. However, for PINNs, we can train the neural network (NN) on a specific set of DEs in a given range of IC-BVs referred to as a bundle , which must be sampled Flamant et al. (2020) . The training process may take more computational resources. However, once trained, the PINN can be evaluated almost instantaneously Chen et al. (2020) ; Lu et al. (2021) for any choice of IC-BV within the bundle, even if it was not trained specifically on it, and will still yield a solution. This may be extremely useful in cases where a very large number of solutions is required, each corresponding to different ICs-BCs or parameters Karniadakis et al. (2021) . Previous work has been done on using NNs for learning the operator mapping free functions u(x) , BCs or ICs given as inputs that appear explicitly in the DEs, to the corresponding solutions s(x) . The operators learned in DeepONets Lu et al. (2021) and Fourier Neural Operators (FNO) Li et al. (2021) can be used to predict new solutions \tilde{s}(x) for stiffer input functions \tilde{u}(x) that model has not trained on. In this sense, the end goal is similar to the one in this work: learning a higher-dimensional “response” of the NN (in our case, the latent space) that allows to compute solutions to different DEs by varying u(x) , BCs/ICs or parameters. However, the key difference stems in the fact that these other methods need to train on known solutions s(x) for each of the (random-)sampled u(x) functions, computed from traditional numerical methods. In our approach, we directly solve the DEs, so no previous knowledge from any solutions is needed. In this way, the method discussed in the paper is a different and complementary approach to DeepONets or FNOs. Stiff DEs, e.g. DEs where there is a large separation of scales between different features of the solution, can be challenging to solve Hairer and Wanner (1996) ; Seiler et al. (2025) . Stiffness can manifest in various forms, including eigenvalue-based stiffness (where the system’s Jacobian has widely varying eigenvalues), multi-scale stiffness (involving both fast and slow time scales such as the flame equation (strictly speaking, the flame equation exhibits multiple types of stiffness) ), damping-induced stiffness (common in physical systems with strong damping such as the Van der Pol oscillator), nonlinear stiffness, interval-dependent stiffness, and transient stiffness Lambert (1991) . In the cases we present, traditional numerical methods are capable of solving the equations. However, in the context of solving DE systems with PINNs, we present differential geometry techniques that significantly improve, or even unlock, our ability to solve them Raissi et al. (2019) . Moreover, these techniques are particularly relevant when there is no established numerical setup to solve the problem. This is the case of inverse problems. PINNs are a well-suited tool for solving such problems Lu et al. (2021) . There are previous works where concepts of differential geometry have been used in the context of NNs. In Kaul and Lall (2020) the authors explore and characterize the behavior of NNs during training through the theory of Riemannian geometry and curvature - they relate the output of the NN with the input transformations through the curvature equations. An analysis and comparison between the convergence properties of the metric of NNs (used for classification) and the metric under the Ricci-DeTurck flow is done in Chen et al. (2022) . In Glass et al. (2020) ; Sun et al. (2023) , the authors remove edges of low importance (prune) the computational graph using the definition of Ricci curvature. More concretely, some recent works have proven that differential geometry techniques can increase the performance and accuracy of PINNs Arora et al. (2023) ; Sahli Costabal et al. (2024) ; Cardoso-Bihlo and Bihlo (2025) . Here, we present two efficient algorithms to use in PINNs that are aimed at significantly improving the solution of inverse problems for non-linear, coupled and multi-scale DEs. The first technique is called Multi-Head (MH) training, where we train the network to learn the general space of all solutions (or latent space) to different variations of the same DEs, from which we can perform transfer learning to different regimes Zou and Karniadakis (2023) ; Desai et al. (2022) ; Zhou and Xu (2024) ; Berardi et al. (2025) ; Lei et al. (2023) ; Pellegrin et al. (2022) . The second one is a technique that we call Unimodular regularization (UR) of the latent space of solutions. It is based

Read the complete text