Fractional-order Jacobian Matrix Differentiation and Its Application in Artificial Neural Networks

Xiaojun Zhou, Chunna Zhao, Yaqun Huang, Chengli Zhou, Junjie Ye, Kemeng Xiang · 2025

Read Fractional-order Jacobian Matrix Differentiation and Its Application in Artificial Neural Networks by Xiaojun Zhou, Chunna Zhao, Yaqun Huang, Chengli Zhou, Junjie Ye, Kemeng Xiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07408.txt.

Introduction

Xiaojun Zhou and co-authors extend fractional calculus into neural network training in a 2025 paper. Fractional derivatives behave unlike ordinary ones and may improve how networks learn. But no matrix form of them fit the automatic differentiation tools that modern training relies on.

The authors define a fractional-order Jacobian matrix, which supports a chain rule for matrix calculations. On that base they build a fractional training method that can run in hidden layers of a network. Swapping standard layers for fractional ones in a test network improved loss and test scores in their trials. They also tracked training time and memory use. The work aims to make fractional methods practical in deep learning.

Catalog record

CREATORXiaojun Zhou, Chunna Zhao, Yaqun Huang, Chengli Zhou, Junjie Ye, Kemeng Xiang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07408

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fractional-order Jacobian Matrix Differentiation and Its Application in Artificial Neural Networks Thanks: This research was funded by the Yunnan University Graduate Research Innovation Fund Project. Xiaojun Zhou, Chunna Zhao, Yaqun Huang, Chengli Zhou, Junjie Ye, Kemeng Xiang Abstract Fractional-order differentiation has many characteristics different from integer-order differentiation. These characteristics can be applied to the optimization algorithms of artificial neural networks to obtain better results. However, due to insufficient theoretical research, at present, there is no fractional-order matrix differentiation method that is perfectly compatible with automatic differentiation (Autograd) technology. Therefore, we propose a fractional-order matrix differentiation calculation method. This method is introduced by the definition of the integer-order Jacobian matrix. We denote it as fractional-order Jacobian matrix differentiation ( 𝐉 α {{\bf{J}}^{\alpha}} ). Through 𝐉 α {{\bf{J}}^{\alpha}} , we can carry out the matrix-based fractional-order chain rule. Based on the Linear module and the fractional-order differentiation, we design the fractional-order Autograd technology to enable the use of fractional-order differentiation in hidden layers, thereby enhancing the practicality of fractional-order differentiation in deep learning. In the experiment, according to the PyTorch framework, we design fractional-order Linear (FLinear) and replace nn.Linear in the multilayer perceptron with FLinear. Through the qualitative analysis of the training set and validation set L ​ o ​ s ​ s Loss , the quantitative analysis of the test set indicators, and the analysis of time consumption and GPU memory usage during model training, we verify the superior performance of 𝐉 α {{\bf{J}}^{\alpha}} and prove that it is an excellent fractional-order gradient descent method in the field of deep learning. K eywords Fractional-order optimization methods, Fractional-order matrix differentiation, Fractional-order Jacobian matrix, Fractional-order gradient descent method, Artificial neural networks 1 Introduction Integer-order matrix differentiation is an important theoretical basis for gradient calculation in artificial neural networks (ANN). Based on it, scholars have proposed many high-performance deep learning optimizers, which have promoted the development of ANN technology. Deep learning optimization algorithms have always been improved from the perspectives of first-order and second-order differentiations. However, with the development of fractional-order differentiation, some scholars began to study deep learning optimization algorithms from the perspective of fractional-order differentiation. These methods can be collectively referred to as fractional-order gradient descent (FGD) methods. FGD methods have different characteristics from integer-order methods in theory. These characteristics include having built-in advanced learning rate scheduling strategies and being less likely to get stuck in saddle points (See Appendix A ), as well as having built-in regularization capabilities (Proof in Appendix B ). Due to these advantages, scholars have attempted to introduce fractional calculus into deep learning and have achieved promising results Yang et al. 2021 ; Roy et al. 2016 ; Yang et al. 2025 . However, in ANNs, data are represented in tensor form. Consequently, when fractional differentiation is extended from scalars to tensors, the fractional matrix differentiation within the hidden layers of ANNs cannot be computed in the same manner as integer-order differentiation, making the matrix-form chain rule inapplicable. As a result, fractional-order differentiation exhibits poor compatibility with integer-order automatic differentiation (Autograd) frameworks. Currently, there are three approaches to applying FGD methods in ANNs: (1) Start from the perspective of symbolic differentiation to study FGD methods. To avoid calculating fractional-order matrix differentiation and the subsequent chain calculations in ANNs, they only use fractional-order matrix differentiation on the Mean Squared Error Loss function (MSELoss). That is, they use the fractional-order power function differentiation formula to solve the differentiation of child nodes in the root node, then perform back-propagation based on the chain rule, and participate in the calculation of gradients of intermediate and leaf nodes. In fact, the differentiations of intermediate and leaf nodes are obtained using integer-order Autograd technology. No matter how the MSELoss is transformed, essentially, these are methods that can only use fractional-order differentiation at the output layer Wang et al. 2022 ; Khan et al. 2019 ; Chen et al. 2017 ; Zhou et al. 2025a ; Lou et al. 2022 . (2) Transform the Grünwald-Letnikov (G-L) formula and use Autograd to make the gradient solution become the nonlinear accumulation of gradients at different time nodes. Essentially, they are integer-order optimization methods with special momentum and special learning rates Zhou et al. 2023 ; Yu et al. 2022 ; Kan et al. 2021 . (3) Abandon the chain rule and use a specific symbolic differentiation method to solve gradients in specific ANNs based on Leibniz’s rule Xie et al. 2023 . The above methods conclude that fractional-order differentiation is superior to integer-order differentiation through theoretical analysis and experimental comparison. However, in practical application scenarios, the defects of these methods are also obvious. On the one hand, scholars have ignored the combination of relevant methods with mainstream deep learning frameworks and have not designed fractional-order deep learning optimizers from the perspective of Autograd technology. This may affect their further promotion in the field of deep learning optimization methods. On the other hand, many fractional-order parameter matrix gradients are not derived from Jacobian matrix differentiation like integer-order matrices. Many of them are computationally complex during the solution process, and when the formulas are transformed, the basis of fractional-order matrix differentiation is weakened to varying degrees Elnady et al. 2025 . In short, the fractional-order deep learning optimizers proposed based on existing FGD methods are not sufficient to comprehensively surpass the numerous integer-order deep learning optimizers. In addition, scholars have conducted convergence analysis on the proposed FGD methods, and most of them emphasize faster convergence rate on the training set than the corresponding integer-order methods. However, the existing FGD methods and their variants are improvements based on existing first-order optimization algorithms, and their convergence performance is the same as that of the corresponding integer-order optimization algorithms. Since there is a linear mapping on the gradient matrix between some existing FGD methods and integer-order optimization methods or their variants, adjusting their fractional-order orders can ultimately be regarded as adjusting the learning rate. This further affects the engineering application of FGD methods. Therefore, in order for FGD methods to have long-term development and application in deep learning, we need to re-examine FGD methods and redefine them according to the research ideas of Autograd technology and integer-order matrix differentiation. In this process, the biggest problem faced by FGD methods is that fractional-order differentiation cannot be applied to the hidden layers of ANNs Zhou et al. 2025b ; Chen et al. 2020 ; Zhu et al. 2018 ; Chen et al. 2022 ; Wahab et al. 2022 ; Viera-Martin et al. 2022 ; Joshi et al. 2023 . Thus, This paper proposes fractional-order Jacobian matrix differentiation {{\bf{J}}^{\alpha}} based on the definition of integer-order Jacobian matrices and fractional-order symbolic differentiation. In the experiment, combined with the mainstream deep learning framework PyTorch, the Linear module is reconstructed into a fractional-order Linear module. In fact, this paper preliminarily proposes fractional-order Autograd technology. By similarly reconstructing other modules in torch.nn, we can also establish a complete ecosystem of fractional-order Autograd technology. The biggest difference between the work in this paper and previous FGD methods is that through calculating the differentiations of each node on the computation graph, the fractional-order gradient is finally obtained. Unlike existing FGD methods that make modifications in Optimizer.step(). In fact, this paper

Read the complete text