Holonorm
Daryl Noupa Yongueng, Hamidou Tembine · 2025
Read Holonorm by Daryl Noupa Yongueng, Hamidou Tembine (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10504.txt.
Introduction
Daryl Noupa Yongueng and Hamidou Tembine propose Holonorm, a new normalization function for training transformer networks. Normalization keeps signals inside a network at healthy sizes. A recent method called Dynamic Tanh showed a simple squashing curve could replace the standard layer normalization, though it suffers from distortion and loss of direction.
Holonorm generalizes the softsign function into a form suited to whole vectors and tensors rather than single numbers. It preserves a signal's direction and orthogonality, stays invertible, and maps every vector into the open unit ball, which prevents activations from exploding in deep models. Because its output lies between zero and one, it can also be read as a percentage, with one minus the value as its complement.
Catalog record
| CREATOR | Daryl Noupa Yongueng, Hamidou Tembine |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-10504
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Holonorm Daryl Noupa Yongueng, Hamidou Tembine Abstract Normalization is a key point in transformer training . In Dynamic Tanh (DyT), the author demonstrated that Tanh can be used as an alternative layer normalization (LN) and confirmed the effectiveness of the idea. But Tanh itself faces orthogonality, linearity and distortion problems. Due to that, his proposition cannot be reliable. So we propose a Holonorm (hn) which has residual connections and nonlinearity. Holonorm is suitable for replacing Tanh in the context of normalization. Although the HoloNorm expression could be similar to the softsign function in dimension one, softsign is a componentwise function which is not good for tensors and vectors of great dimension. Holonorm preserves the orthogonality, the direction, the invertibility of the signal. Holonorm is also a suitable metric, maps all vectors into the open unit ball. This prevents exploding activations and improves stability in deep Transformer models. In this work, we have meticulously examined the normalization in transformers and say that Holonorm, a generalized form of softsign function suited as a normalization function first.Second, defined between 0 and 1 hn serves as a percentage, and 1 − Holonorm 1-\text{Holonorm} is its complement, making it better understandable in evaluating a model. Keywords: Layer normalization, Tanh, Transformer, Orthogonality, Nonlinearity, Componentwise. 1 Introduction F inding the best method to train deep learning networks is a long-standing challenge. Layer normalization has became a fundamental aspect in machine learning model architecture. Scientists have proposed several expression to enhance the perfomance of a model for a specific task. In the literature, we have the Gaussian error linear unit (Gelu) [ 1 ] that surpasses Rectify error linear unit (Relu) [ 2 ] . But one of the most popular is Tanh [ 4 ] which got some symetric properties that make it very intersting but not enough in our point of view because of his limitations. In Hilbert spaces, applying componentwise means possibly an infinite number of coefficients. tanh is computationally ineffective compared to the calculation of a norm. A striking point of Tanh is that it strongly correlates certain vectors that are uncorrelated at the beginning of the training. When it comes to orthogonality between words (uncorrelated words), by applying tanh , it can make them correlated. But by applying holonorm , it stays uncorrelated. Holonorm keeps them uncorrelated if they were uncorrelated initially. Thus, Tanh does not maintain orthogonality and therefore is not suitable for tensors, vectors then textual data . Literature review Layer normalization is a technique used in neural networks to normalize the inputs across the features for each data point. Normalization plays a fundamental role in stabilizing training and improving the convergence of deep neural networks [ 9 ] . While various normalization techniques have been proposed, the use of Tanh-based normalization introduces several challenges in training Transformer architectures. The Tanh activation function, although bounded and smooth, suffers from saturation effects where the gradients become vanishingly small for large input values [ 7 ] . This leads to decrease in gradient flow in deep networks, significantly slowing down convergence and altering learning in early layers [ 8 ] . In Transformer models, which rely heavily on residual connections and multi-head attention mechanisms, such vanishing gradients can accumulate across layers, causing optimization instability [ 5 ] . Moreover, Tanh outputs are symmetric around zero, but the compressive nature of the function near its limits leads to information loss in hidden states [ 10 ] . These issues are particularly pronounced in sequence-to-sequence tasks where long-term dependencies are essential, making Tanh less suitable compared to other normalization techniques such as LayerNorm or recent adaptive methods [ 6 ] . Recent studies emphasize the importance of non-saturating activation and normalization mechanisms to preserve representational fidelity and training efficiency. As such, replacing Tanh in normalization schemes has become critical in addressing depth-related degradation in Transformers. Holonorm Holomorphic normalization (HoloNorm) is an activation function better than Hyperbolic Tangent (Tanh) to normalize a function during the training of a machine learning model due the relevant facts mention earlier 1 . Holonorm is defined as: \text{HN}_{p}(x)=\frac{x}{\left(1+\|x\|_{p}\right)} Where p is the range of the norm, typically p=1 for norm 1 and p=2 for euclidiean norm. The function is defined for all x\in\mathbb{R}^{D} and maps to the open unit ball in \mathbb{R}^{D} . In norm-1, softsign function is assimilated to holonorm but holonorm is defined in the open unit ball. Figure 1: Curve of hn(x)=\frac{x}{1+\|x\|} Figure 2: Primitive H(x) : H(x)=\begin{cases}x-\ln(1+x)&x\geq 0\\ x+\ln(1-x)&x<0\end{cases} Figure 3: Derivative of h^{\prime}(x)=\frac{1}{(1+|x|)^{2}} Figure 4: Inverse of h^{-1}(y)=\frac{y}{1-|y|},\quad y\in(-1,1) Table 1: Holonorm and her correlated curves This transformation is scale-sensitive but direction-preserving, unlike tanh, it scales the entire vector uniformly rather than compressing individual components. It preserves the direction by approximately maintaining the angle between vectors and keeps orthogonal signals nearly orthogonal after transformation. Additionally, it does not distort unrelated signals, as independent vectors remain uncorrelated, making it suited for reconstruction since the original signal geometry is largely preserved, allowing for better recovery. Considering the metric evaluation, it does the ratio between the actual value and the predicted value or vice versa. Either if the actual value is zero, the denominator will never be zero, then output will not tend toward infinity, allowing the reconstruction of the initial terms. Contribution In this paper, our aim is to propose a novel function that can serves for differents operations as a normalization, activation, and evaluation function. In section 2 we will present advantages of Holonorm on Tanh and the reasons why we should replace it by Holonorm. Section 3 we will do and experiment to prove our statement with a discussion and the last section is the conclusion. 2 Reason why HoloNorm is better than Tanh The mathematical properties of Tanh and HoloNorm are compared in Table 3 . The general Holonorm hn(x) function is defined as: Proposition 1: Holonorm is mathematical better than Tanh in every use case . Property Tanh Holonorm Remarks Expression \tanh(x)=\dfrac{e^{x}-e^{-x}}{e^{x}+e^{-x}} \text{HN}(x)=\dfrac{x}{1+|x|} Holonorm is simpler Inverse \tanh^{-1}(y)=\dfrac{1}{2}\ln\left(\dfrac{1+y}{1-y}\right) \text{HN}^{-1}(y)=\dfrac{y}{1-|y|} or \dfrac{y}{1+|y|} Holonorm is faster, easier Derivative \dfrac{d}{dx}\tanh(x)=1-\tanh^{2}(x) \dfrac{d}{dx}\text{HN}(x)=\dfrac{1}{(1+|x|)^{2}} Holonorm is cheaper Integral \int\tanh(x)\,dx=\ln(\cosh(x))+C \int\text{HN}(x)\,dx=\begin{cases}(1+x)-\ln(1+x)+C&\text{if }x\geq 0\\ x+\ln(1-x)+C&\text{if }x<0\end{cases} Holonorm integrable in parts Numerical Stability Poor near x\gg 1 (saturates) Stable everywhere Holonorm is more stable Computational Cost Involves \exp(x) and \ln(x) Simple algebraic ops only Holonorm is more efficient Figure 5: Differents curves of holonorm and Tanh 5 presents the different curves of Holonorm and Tanh. We can see the similarity and the differences of each of them in a same graphic. Vector Holonorm Approx. Tanh Approx. (1, 2, 3) (0.2108, 0.4216, 0.6324) (0.7616, 0.9640, 0.9951) (12, 3, -6) (0.8136, 0.2034, -0.4068) (1.0000, 0.9951, -0.9999877) (1, -2, 1) (0.2898, -0.5796, 0.2898) (0.7616, -0.9640, 0.7616) Table 2: Comparison of Holonorm and Tanh on example vectors In 2 , we see that the Tanh function compresses the values, leading to a loss of information about the original signal. In contrast, Holonorm retains the original signal geometry, preserving the relative magnitudes and directions of the vectors. Figure 6: Curves after applying holonorm The above figure 6 represent those 3 vectors after Holonorm and we see that the don’t change their direction after the transformation. This mean that Holonorm preserve the meaning of words, the direction and