Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks

Keigo Nishida [email protected], Eren Mehmet Kıral [email protected], Kenichi Bannai [email protected], Mohammad Emtiyaz Khan [email protected], Thomas Möllenhoff [email protected] · 2025

Read Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks by Keigo Nishida [email protected], Eren Mehmet Kıral [email protected], Kenichi Bannai [email protected], Mohammad Emtiyaz Khan [email protected], Thomas Möllenhoff [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17768.txt.

Introduction

Keigo Nishida and co-authors present a machine learning paper, published in 2025, borrowing a trick from biology to train large networks with low precision numbers. In the brain, synapse strengths follow a log-normal spread and change by multiplication plus noise, yet the system stays stable. The authors ask whether artificial networks can gain the same steadiness from multiplicative updates.

Their rule, derived from Bayesian reasoning, updates each weight by a factor rather than a step and needs just one extra stored vector. It proved as easy to use as the standard Adam optimizer. Vision and language models trained from scratch stayed stable and accurate under low precision arithmetic, hinting at future energy saving hardware that computes roughly by design.

Catalog record

CREATORKeigo Nishida [email protected], Eren Mehmet Kıral [email protected], Kenichi Bannai [email protected], Mohammad Emtiyaz Khan [email protected], Thomas Möllenhoff [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Log-Normal Multiplicative Dynamics for Stable Low-Precision Training of Large Networks Keigo Nishida [email protected], Eren Mehmet Kıral [email protected], Kenichi Bannai [email protected], Mohammad Emtiyaz Khan [email protected], Thomas Möllenhoff [email protected] Abstract Studies in neuroscience have shown that biological synapses follow a log-normal distribution whose transitioning can be explained by noisy multiplicative dynamics. Biological networks can function stably even under dynamically fluctuating conditions arising due to unreliable synaptic transmissions. Here we ask: Is it possible to design similar multiplicative training in artificial neural networks? To answer this question, we derive a Bayesian learning rule that assumes log-normal posterior distributions over weights which gives rise to a new Log-Normal Multiplicative Dynamics (LMD) algorithm. The algorithm uses multiplicative updates with both noise and regularization applied multiplicatively. The method is as easy to implement as Adam and only requires one additional vector to store. Our results show that LMD achieves stable and accurate training-from-scratch under low-precision forward operations for Vision Transformer and GPT-2. These results suggest that multiplicative dynamics, a biological feature, may enable stable low-precision inference and learning on future energy-efficient hardware. Code is available at https://github.com/team-approx-bayes/lmd 1 Introduction Biological synapses exhibit continuous dynamic activity accompanied by noise ( Choquet and Triller 2013 ) . The distribution of the synaptic spine size has been observed to follow a log-normal distribution, which is thought to arise from noisy multiplicative dynamics ( Loewenstein et al. 2011 ) . In addition to exhibiting activity-dependent plasticity, synaptic spines also show spontaneous fluctuations, with their dynamics considered to be approximately proportional to the spine size ( Loewenstein et al. 2011 ; Kasai et al. 2021 ) . Such dynamics are often regarded as a cause of unreliable or uncertain information transmission, but their importance in neural computation is being increasingly recognized ( Seung 2003 ; Bartol et al. 2015 ; Kappel et al. 2015 ; Aitchison et al. 2021 ) . To exploit the dynamics of biological synapses for artificial neural networks (ANNs), both the weight updates and the noise injection for the spontaneous fluctuations need to be multiplicative. However, contemporary ANN training employ only one of these two mechanisms. Bernstein et al. 2020 addressed the question of how the brain can stably learn despite the unreliability of biological synapses by adopting multiplicative weight updates (MWU) ( Arora et al. 2012 ) for ANNs. However, spontaneous fluctuations caused by noise were not addressed. In contrast, Bayesian neural networks assume a probability distribution over their weights and sample from this distribution ( Graves 2011 ; Blundell et al. 2015 ; Khan et al. 2018 ) . However, because the sampling noise is not inherently proportional to the weight magnitude, it is insufficient to reproduce the fluctuations observed in biological synapses. Previous works use multiplicative noise proportional to the weight norm, but still use additive gradient updates ( Wen et al. 2018 ; Bisla et al. 2022 ; Trinh et al. 2024 ) . Multiplicative fluctuation is highly compatible with data formats that emphasize dynamic range with limited bit width, making it promising for application to energy-efficient dedicated hardware required by low-precision data formats for both inference and training ( Bernstein et al. 2020 ; Zhao et al. 2022 ; Haghi et al. 2024 ) . This characteristic may also help improve computational throughput using the Microscaling (MX) data format ( Rouhani et al. 2023a ) , which has been considered for various LLM deployments ( Rouhani et al. 2023c ; Verrilli 2024 ; Tseng et al. 2025 ; Ramani et al. 2025 ) . However, multiplicative updates tend to cause exponential weight growth ( Bernstein et al. 2020 , Section 8) , and their use in training large-scale neural networks from scratch has not been successful. Stabilizing learning under multiplicative updates requires new regularization techniques. In this paper, we propose the Log-Normal Multiplicative Dynamics (LMD) algorithm. LMD is derived from the Lie-Group Bayesian Learning Rule (Lie-Group BLR) with the multiplicative group of positive reals and log-normal weight noise distribution ( Kiral et al. 2023 ) . LMD uses multiplicative weight updates with both noise and regularization applied multiplicatively, using which we can train large ANNs from scratch successfully, see Figure 1 . Figure 1: These two plots demonstrate that LMD enables full-scratch learning of Vision Transformer and GPT-2. Even when the forward pass is performed using low-precision formats based on the Microscaling data format (MXFP6), no performance degradation occurs. AdamW and Madam are unable to sufficiently train ViT. On GPT-2, AdamW shows perplexity similar to LMD, but we can observe that the optimizer can not stably learn using low-precision forward-passes. For GPT-2 training, LMD and Madam use a sequence length of 4096 and a batch size of 16. Note that AdamW uses a sequence length of 1024 and a batch size of 64. However, the number of tokens per step is the same. All training, except for the forward passes in MXFP6, was performed using bfloat16. Our contributions are as follows: 1. We achieve the first successful training from scratch of Vision Transformer (ViT) ( Dosovitskiy et al. 2021 ) and GPT-2 ( Radford et al. 2019 ) using multiplicative weight updates. Using low-precision emulation, we demonstrate that LMD maintains full model performance even when forward passes are performed in the low-precision MX data format (Figure 1 ). 2. We show that LMD effectively suppresses the excessive weight growth that has been an issue for standard MWU methods (Figure 2 ). Through ablation studies, we verify that multiplicative weight decay strongly suppresses weight growth and delivers superior regularization compared to conventional additive weight decay (Figure 3 ). 3. We show that multiplicative noise injection is important for stabilizing the weights when using low-precision MX data format for forward passes (Figure 4 ). (Backward passes use bfloat16 ( Kalamkar et al. 2019 ) rather than the MX data format.) Figure 2: These two plots show the dynamics of the \ell_{2} norm of weights during training of the models shown in Figure 1 when only using bfloat16. Madam takes a large norm, while LMD converges to a norm close to the initial value, clearly suppressing excessive weight increase. Table 1: ViT and GPT-2 optimizer comparison; mean ± standard deviation (n = 3 independent runs). ViT on ImageNet GPT-2 on OpenWebText Method Test Acc. (%) Weight Norm (seq-len) Val. Loss Weight Norm AdamW 68.11_{\pm 0.38} 260.7_{\pm 0.5} 1024 2.937_{\pm 0.001} 392.7_{\pm 0.4} 4096 4.790_{\pm 2.017} 931.7_{\pm 70.5} w/ Fwd MXFP6 67.99_{\pm 0.27} 261.5_{\pm 0.4} 1024 3.015_{\pm 0.000} 425.0_{\pm 0.4} Madam 60.14_{\pm 0.31} 577.3_{\pm 0.9} 4096 3.303_{\pm 0.003} 765.1_{\pm 3.4} LMD 77.06_{\pm 0.08} 55.2_{\pm 0.1} 1024 2.961_{\pm 0.002} 217.2_{\pm 4.5} 4096 2.925_{\pm 0.006} 212.9_{\pm 2.1} w/ Fwd MXFP6 \bf 77.15_{\pm 0.08} 55.5_{\pm 0.3} 4096 \bf 2.927_{\pm 0.002} 204.1_{\pm 11.2} 2 Background and Related Work 2.1 Multiplicative Weight Updates The low reliability of synaptic transmission is at the heart of neural computation ( Seung 2003 ) . The brain not only overcomes such unreliability, but can even use it to drive learning ( Kappel et al. 2015 ; Aitchison et al. 2021 ; Kasai et al. 2021 ) . Synaptic transmission is driven by dynamic processes among the molecules that make up the synapse, and probabilistic approaches are thought to be effective in understanding their robustness and

Read the complete text