Neural Chains and Discrete Dynamical Systems

Sauro Succi, Abhisek Ganguly, Santosh Ansumali · 2026

Read Neural Chains and Discrete Dynamical Systems by Sauro Succi, Abhisek Ganguly, Santosh Ansumali (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00473.txt.

Introduction

Abhisek Ganguly, Santosh Ansumali, and Sauro Succi revisit the idea that deep neural networks behave like discrete dynamical systems, where each layer advances a state much like a step in a simulation. They compare exact and numerical solutions of the Burgers and Eikonal equations with solutions learned by physics-informed neural networks, which are trained to obey the governing equations.

Their comparison shows the networks reach the same underlying dynamics by a different computational route. Layer-by-layer evolution heads toward attractor states, and many different parameter settings can give similar answers. Unlike the neat operators of finite-difference methods, the networks spread information across many parameters. That usually means more parameters and less direct interpretability, a trade-off the authors argue matters for scientific computing.

Catalog record

CREATORSauro Succi, Abhisek Ganguly, Santosh Ansumali
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00473

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Neural Chains and Discrete Dynamical Systems Sauro Succi, Abhisek Ganguly, Santosh Ansumali Abstract 1 Abstract We inspect the analogy between machine-learning (ML) applications based on the transformer architecture without self-attention, neural chains hereafter, and discrete dynamical systems associated with discretised versions of neural integral and partial differential equations (NIE, PDE). A comparative analysis of the numerical solution of the (viscid and inviscid) Burgers and Eikonal equations via standard numerical discretization (also cast in terms of neural chains) and via PINN’s learning is presented and commented on. It is found that standard numerical discretization and PINN learning provide two different paths to acquire essentially the same knowledge about the dynamics of the system. PINN learning proceeds through random matrices which bear no direct relation to the highly structured matrices associated with finite-difference (FD) procedures. Random matrices leading to acceptable solutions are far more numerous than the unique tridiagonal form in matrix space, which explains why the PINN search typically lands on the random ensemble. The price is a much larger number of parameters, causing lack of physical transparency (explainability) as well as large training costs with no counterpart in the FD procedure. However, our results refer to one-dimensional dynamic problems, hence they don’t rule out the possibility that PINNs and ML in general, may offer better strategies for high-dimensional problems. 2 Introduction The transformer architecture, augmented with the mechanism of self-attention, stands out as a milestone of modern machine-learning research, with a major impact especially on the predictive capabilities of Large Language Models (LLM) [ 18 ] . In broad strokes, the power of self-attention stems mainly from the inclusion of non-local correlations in feature space and time (memory). In mathematical terms, this is associated to non-Markov chains. In this note, we take a step back and revisit the analogy between transformers without self-attention (which simplifies into a basic DNN), hereafter simply “neural chains”, and discrete dynamical systems in relaxation form, originating from the discretization of neural integral equations [ 19 ] . The main aim of this work is different from the study of neural ordinary differential equations [ 2 ] , where one focuses on the development of differential equations and discrete time representation to mimic layers. Here we treat the features as a continuous variable, which immediately leads to integral equations. This approach represents a natural generalization of neural ODE idea, which has recently emerged as a powerful paradigm in deep learning, by combining continuous-time dynamical systems with neural networks [ 8 ] . This perspective has since inspired a broad family of models including Neural Controlled DEs, Neural SDEs (for stochastic processes), Hamiltonian Neural Networks, and continuous normalizing flows, physics-informed learning, generative modeling, and beyond. The focal point of this work is to investigate the potential connection between the mechanisms that drive the learning process of a PINN network with the numerical solution of neural chains as discrete dynamical systems, keeping in mind the standard, well interpretable numerical methods like the Finite Difference [ 13 ] . This study may also be seen as a step toward advancing the emerging field of Mechanistic Interpretability [ 15 ] for systems that learn physics. Initial directions in this area have already begun to appear in the literature [ 7 , 14 ] . This connection is interesting for several reasons. First, in the case where PINN is designed to learn converged solutions of neural PDE’s, it permits to assess the size of the neural chain required to achieve the desired target [ 6 ] . Second, and more generally, it helps shedding light on the explainability/interpretability of the learning procedure. In passing, it is worth reminding that a universal approximation theorem guarantees that a neural network with just a single hidden layer can represent arbitrary continuous functions, provided suitable restrictions are imposed on the activation function [ 4 , 9 , 10 ] . For all its beauty and relevance, this hides the power of spreading the learning process across multiple layers, which, based on the aforementioned connection between neural chains and discrete dynamical systems, is tantamount to reaching the target as a result of the unfolding of a multi-step trajectory over time. Bringing such connection to full light should help in elucidating the theoretical foundations of machine learning and facilitate their extension to more general and possibly more reliable and efficient architectures for future AI scientific applications. 3 Neural chains: transformer-like architecture without self-attention The basic idea of deep neural networks [ 12 ] is to represent a given d -dimensional output y (target) through the recursive application of a nonlinear and nonlocal map to the input data x . For a deep neural network (DNN) consisting of an input layer x , L hidden layers z_{1}\dots z_{L} , each containing N neurons, and an output layer y , the update chain x\to z_{1}\dots\to z_{L}\to y reads symbolically as follows: \displaystyle z_{0} \displaystyle=x, (1) \displaystyle z_{1} \displaystyle=f(W_{1}x-b_{1}), \displaystyle\vdots \displaystyle z_{L} \displaystyle=f(W_{L}z_{L-1}-b_{L}), \displaystyle z_{L+1} \displaystyle=f(W_{L+1}z_{L}-b_{L+1})=y. where W_{l} are N\times N matrices of weights, b_{l} are N-dimensional arrays of biases and f is a nonlinear activation function, to be chosen out of a large palette of options. Note that each layer contains the same number of neurons, N , which is the distinctive property of transformers, possibly with additional attention layers sandwiched in between. For the sake of simplicity, the input and output are taken in the form of scalar functions of a single variable q , a coordinate in feature space. The output y(q) is then compared with a given target y_{T}(q) (Truth) for a large set of input data x(q) , and the weights are recursively updated in such a way as to minimize the discrepancy between y(q) and y_{T}(q) , known as the Loss function ( E(y,y_{T}) in Fig. 1 ) , up to the desired tolerance. This error back-propagation step is usually performed via a steepest descent minimization (or some stochastic equivalent, e.g, Stochastic Gradient Descent or SGD for large training data) W^{\prime}=W-\alpha\frac{\partial E}{\partial W}, (2) where E[W]=||y_{T}-y|| is the error (Loss function) in some suitable metric and \alpha is a relaxation parameter controlling the convergence (learning rate) of the procedure. The two basic step above are complemented with a normalization constraint, ||z(t)||=||z(0)||, (3) to ensure the finiteness of the solution at all times. Convergence of the learning procedure is associated with the attainment of local minima of the Loss function. Given the large number of dimensions of weight space \mathcal{W} and the corrugated shape of the Loss function landscape, the number of local minima is generally extremely large and it is hard to tell apart a-priori those that deliver a genuine learning function from those that don’t. This is the hard-core problem of machine learning, but in this paper we are not concerned with this issue because, as detailed in the sequel. Figure 1: The basic blocks of the DNN architecture. The procedure is repeated over a very large set of input data. The basic idea of the entire ML procedure is to derive a general input-output relation y(q)=F_{f}[x(q);W,b], (4) built out of the training dataset consisting of all input data X\equiv\{x^{(r)}(q)\} where r=1,N_{R} runs over a set of N_{R} realizations. The hope is that

Read the complete text