Layer-wise Derivative Controlled Networks

Rowan Martnishn, Sean Anderson, [4pt] Sentivity AI · 2026

Read Layer-wise Derivative Controlled Networks by Rowan Martnishn, Sean Anderson, [4pt] Sentivity AI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15463.txt.

Introduction

Rowan Martnishn and Sean Anderson introduce ChainzRule, a net design that swaps jagged linear pieces for a polynomial engine. The engine is steered by rules on its inner derivatives. The goal is steadier conduct: small input shifts should not cause large output swings. Tests pit it head to head with standard nets. It wins with far fewer weights. On a digit task it cuts peak gradient swings by about 23.1 percent. On a review-rating task it stays in step with rivals.

Catalog record

CREATORRowan Martnishn, Sean Anderson, [4pt] Sentivity AI
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73987
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15463

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Layer-wise Derivative Controlled Networks Rowan Martnishn, Sean Anderson, [4pt] Sentivity AI Abstract As machine learning models grow in complexity, they increasingly struggle with three conflicting demands: the need for high accuracy, the requirement for hardware efficiency, and the necessity of functional stability. Traditional architectures often achieve performance at the expense of spiky or unpredictable behavior, where small changes in input lead to massive swings in output - a critical flaw for real-world deployment in sensitive environments. This paper introduces ChainzRule (CR) , a novel neural architecture designed to harmonize these competing goals. ChainzRule replaces standard piecewise-linear activations with a Polynomial Engine governed by Differential Regularization (DREG) . Unlike traditional methods that impose global, coarse-grained constraints on a model’s Lipschitz constant, DREG acts as a targeted regularization on intermediate derivatives. This approach suppresses extreme sensitivity (the ’spiky’ behavior) without attenuating the representational power inherent in the Polynomial Engine. In head-to-head ”Fair Fight” benchmarks, ChainzRule outperformed standard models while using 15.5 × \times fewer parameters. On the MNIST dataset, it reduced peak gradient volatility by an average of 23.1%, ensuring a smoother and more predictable manifold. On Yelp Full ordinal regression under explicit DREG regularization, ChainzRule achieves 70.17% accuracy, validating that derivative-aware regularization is compatible with competitive performance on realistic tasks. By embedding gradient awareness into the architecture via DREG, ChainzRule demonstrates that stability and accuracy need not be competing objectives. 1 Introduction Modern methods of approximating functions have grown with near-exponential pace over the past few years 20 . With the creation of such models (e.g. Transformers), the goal is typically to push theoretical bounds of accuracy and capability 3 . However, this pursuit has led to bloated, unstable, and sensitive frameworks 14 ; 35 . As competing organizations boast developments that are largely equal in comparison, a new frontier is emerging: deployment in real-world environments (e.g. defense), heightened affordability in training, and practical reliability 16 . In high-stakes surrogate modeling pipelines, real-time control systems, and scientific computing - reliability depends not only on predictive accuracy but on predictable sensitivity . Existing methods struggle because global constraints (Spectral Normalization) or terminal penalties (Input Gradient Penalty) cannot control internal amplification events created by chain-rule composition across depth 25 ; 10 . Recent work on Physics-Informed Neural Networks further underscores that stable derivatives are critical for convergence and physical consistency 29 ; 9 , yet no general-purpose architecture has integrated layer-wise derivative control directly into the forward pass. Given this gap, we introduce ChainzRule , a novel neural architecture that harmonizes accuracy, efficiency, and functional stability. ChainzRule replaces piecewise-linear activations with a Polynomial Engine governed by Differential Regularization (DREG). This approach dampens extreme sensitivity without curtailing the representational power inherent in the Polynomial Engine. 1.1 Our Contribution: The ChainzRule Engine and Differential Regularization Inspired by modern regularization techniques such as Dropout, Weight Decay, and L2 33 ; 22 ; 2 , we enforce stability via layer-wise restrictions rather than global restraints. We created a custom PolyLayer in which each neuron learns its own polynomial coefficients under the explicit constraint of DREG 21 . 1.2 Summary of Results: SEA (Sensitivity, Efficiency, Accuracy) Performance We structure our validation as follows: • MNIST: Rigorous stress-testing of DREG’s sensitivity control using tail-ratio analysis and paired statistical tests across multiple model capacities. • Yelp: Validation on realistic NLP data (Yelp Full) showing that the stability mechanism does not sacrifice accuracy (70.17% competitive accuracy). • CIFAR: Assessment of robustness benefits on vision tasks with natural and adversarial corruptions. • Ablations: Analysis of design choices (DREG coefficient, Fair Fight benchmark, scaling laws) to validate architectural decisions. 2 Related Work While existing gradient control techniques (Lipschitz constraints, Input Gradient Penalty, Spectral Normalization) provide important baselines, they operate via global constraints or terminal penalties. These approaches often incur computational overhead ( O(N) memory, high latency) or representational cost (reduced expressivity). We position ChainzRule differently: as a design philosophy that integrates derivative control into the architecture itself via forward-mode Jacobian accumulation. This enables layer-wise sensitivity management with the computational efficiency of a standard forward pass. 2.1 Parameter-Efficient Architectures and Modern MLPs While modern efficiency research has focused heavily on Parameter-Efficient Fine-Tuning (PEFT) via low-rank updates 1 or architectural bottlenecks 32 , these methods primarily optimize for the quantity of trainable weights. They often overlook the sensitivity of the resulting mapping. CR diverges from this trend; rather than simply compressing a standard MLP, we introduce a polynomial inductive bias that treats derivative stability as a primary optimization objective, offering a more favorable accuracy-sensitivity frontier than traditional bottleneck architectures. While efficiency will not be compared directly to these models, we will go onto show the extreme efficiency of CR later on. 2.2 Polynomial and Symbolic Neural Networks Neural ODE’s While Neural ODEs are mathematically elegant, they are ”black boxes” in terms of architectural transparency. The reliance on an external ODESolve component introduces a computational overhead that is often prohibitive for high-throughput tasks like real-time NLP or mobile CV. While the Adjoint Sensitivity Method allows Neural ODEs to maintain a constant O(1) memory cost during training, the requirement of an iterative ODESolve at inference time remains a prohibitive bottleneck for high-throughput applications. ChainzRule bridges this gap by utilizing polynomial expansions that offer the ”curvy,” continuous-like inductive bias of an ODE, but execute in a single, discrete, high-speed feed-forward pass 4 . Kolmogorov-Arnold Networks (KANs) These networks use learnable B-splines on edges instead of fixed activations on nodes. However, they suffer from Runge’s Phenomenon - high-order oscillations at the edges of the interval. Without explicit DREG ”governor,” KANs can become more unstable than MLPs as they scale 21 . Physics Informed Neural Network (PINNs) There is no denying the revolutionary nature of PINNs -integrating the laws of nature directly into the neural network’s loss function 29 . The Physics-Informed approach is highly specialized; it requires a known governing equation (like the Navier-Stokes or Burgers’ equation) to function. In general-purpose machine learning (e.g., sentiment analysis or object recognition), there is no ”physics” to enforce. CR takes the core technical intuition of PINNs - that derivatives should be part of the objective function - and generalizes it. By using Differential Regularization (DREG), we are enforcing a ”behavioral consistency” similar to a PINN, but for tasks where a universal, governing law is absent. 2.3 Jacobian Regularization and Sobolev Training Sobolev Training This is the ”gold standard” for training with derivatives. It requires a loss function that includes the difference between model derivatives and target derivatives. Sobolev Training treats derivatives as explicit training targets, which requires the existence of ground-truth derivative labels - a luxury rarely available in general-purpose NLP or CV tasks. Furthermore, retrieving these derivatives typically requires Double-Backpropagation to update weights 8 . This incurs a computational complexity of : \mathcal{O}(C_{f}+2C_{b})\approx\mathcal{O}(3C_{f}) where C_{f} and C_{b} are the costs of forward and backward passes, respectively. In contrast, ChainzRule utilizes forward-mode Jacobian accumulation. This approach is not only more computationally efficient, reducing overhead to \mathcal{O}(d\cdot C_{f}) , but it is also more numerically stable in compact regimes, as it avoids the floating-point instability often associated with second-order computational graphs. We can see that using CR provides the same derivative awareness at a fraction of the memory and compute cost. Input Gradient

Read the complete text