From MNIST to ImageNet: Understanding the Scalability Boundaries of Differentiable Logic Gate Networks

Sven Brändle, Till Aczel, Andreas Plesner & Roger Wattenhofer · 2025

Read From MNIST to ImageNet: Understanding the Scalability Boundaries of Differentiable Logic Gate Networks by Sven Brändle, Till Aczel, Andreas Plesner & Roger Wattenhofer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25933.txt.

Introduction

Sven Brandle and co-authors test how far differentiable logic gate networks scale in a 2025 paper. Such networks replace neurons with learnable logic gates, giving very fast, energy light inference that suits hardware. They have mostly been proven on small tasks with ten classes or fewer, leaving the output layer design and large scale behavior open.

Synthetic and real data sets probe expressiveness and scaling, with class counts pushed toward two thousand. Temperature tuning proved key to output layer quality. The study maps when the standard group sum output works well and how to apply it at large scale, giving design guidance for this young network family.

Catalog record

CREATORSven Brändle, Till Aczel, Andreas Plesner & Roger Wattenhofer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25933

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From MNIST to ImageNet: Understanding the Scalability Boundaries of Differentiable Logic Gate Networks Sven Brändle, Till Aczel, Andreas Plesner & Roger Wattenhofer Abstract Differentiable Logic Gate Networks (DLGNs) are a very fast and energy-efficient alternative to conventional feed-forward networks. With learnable combinations of logical gates, DLGNs enable fast inference by hardware-friendly execution. Since the concept of DLGNs has only recently gained attention, these networks are still in their developmental infancy, including the design and scalability of their output layer. To date, this architecture has primarily been tested on datasets with up to ten classes. This work examines the behavior of DLGNs on large multi-class datasets. We investigate its general expressiveness, its scalability, and evaluate alternative output strategies. Using both synthetic and real-world datasets, we provide key insights into the importance of temperature tuning and its impact on output layer performance. We evaluate conditions under which the Group-Sum layer performs well and how it can be applied to large-scale classification of up to 2000 classes. 10^{1} 10^{2} 10^{3} 0 25 50 75 100 # Classes Test Accuracy (%) DLGN MLP Figure 1: DLGNs (blue) consistently outperform MLPs (red) across classification tasks with up to 2000 classes. The result illustrates the potential of logic-gate-based architectures to remain effective when applied to large-scale classification problems. 1 Introduction Deep artificial neural networks have improved immensely in the last few years, exhibiting impressive performance across a wide range of tasks ( Golroudbari & Sabour 2023 ; Noor & Ige 2024 ; Ekundayo & Ezugwu 2025 ) . However, these improvements come with rapidly growing computational costs ( Thompson et al. 2020 ; Rosenfeld 2021 ; Tripp et al. 2024 ) . This constrains their deployment in many real-world environments, particularly on edge devices and mobile phones ( Zhang et al. 2020 ; Zheng 2025 ) . Thus, there is increasing interest in developing neural networks with competitive performance and energy-efficient deployment. All computations on digital hardware are inherently built from Boolean operations such as AND, OR, and NOT ( Kukunas 2015 ) . This raises the question of whether machine learning models can be run directly on logic gates, the fundamental building blocks of digital computation. Logic Gate Networks (LGNs) provide one way to address this question. Instead of relying on traditional arithmetic operations, LGNs combine discrete logical operations, enabling extremely fast inference. While inference is efficient, training such discrete networks poses significant challenges. Differentiable LGNs (DLGNs) ( Petersen et al. 2022 ) resolve this issue by introducing continuous relaxations of logical operations, allowing LGNs to be trained with gradient-based optimization methods ( LeCun et al. 2015 ; Goodfellow et al. 2016 ) . Up to now, DLGNs have been evaluated mainly on small classification datasets. Designing an expressive yet trainable classification layer for DLGNs is not trivial. The most common approach is the Group-Sum layer, where a large set of output neurons represents each class. The activations of neurons within each set are summed to produce the logit for that class. Thus, every class requires its own dedicated group of neurons. Petersen et al. 2022 report using between 8’000 and 64’000 output neurons for MNIST (800–6’400 neurons per class) and up to 102’400 neurons per class for CIFAR-10. While effective for small-scale datasets, this design raises concerns about efficiency and scalability as the number of classes increases. The standard Group-Sum classification layer is believed to have limited capacity to handle larger numbers of classes, potentially restricting the scalability of DLGNs. Petersen et al. 2022 ; Petersen et al. 2024 ; Yousefi et al. 2025 mainly evaluated DLGNs on MNIST and CIFAR-10, arguing that training for a larger number of classes is infeasible when up to 102’400 neurons per output class is required. In this work, we provide the first large-scale evaluation of DLGNs on datasets with thousands of classes, systematically analyzing the expressiveness of the Group-Sum output layer. We show that the temperature parameter \tau is a key factor that controls redundancy and neuron utilization, directly influencing scalability. Beyond Group-Sum, we propose and evaluate alternative output layer designs, comparing their effectiveness across synthetic and real-world datasets. Together, these experiments shed light on the strengths and limitations of DLGNs in large-class settings and highlight open challenges for extending these architectures to more complex data. Open questions include how many output bits are needed to represent a class reliably and whether summing over large groups of output neurons provides an effective decoding strategy. 2 Background 2.1 Differentiable Logic Gate Networks Logic Gate Networks (LGNs) are composed of Boolean logic gates that process binary signals. Karakatic et al. 2013 proposed a genetic programming approach that constructs circuits from truth tables. While effective for small tasks, these methods scale poorly ( Ondas et al. 2005 ) . Differentiable Logic Gate Networks (DLGNs) ( Petersen et al. 2022 ) address this limitation by introducing continuous relaxations of discrete functions, enabling gradient-based training. A DLGN consists of LogicLayers , where each neuron receives two inputs and applies a learnable logical function. During training, a neuron’s output is computed as: o=\sum_{i=1}^{16}p_{i}\cdot f_{i}(a,b)=\sum_{i=1}^{16}\frac{e^{w_{i}}}{\sum_{j=1}^{16}e^{w_{j}}}\cdot f_{i}(a,b), (1) where a and b are inputs, f_{i} represent logical functions such as AND, OR, XOR (see Table 1 ), and w_{i} are learnable weights. The continuous formulation allows end-to-end training with gradient-based learning methods ( LeCun et al. 2015 ; Goodfellow et al. 2016 ) . Table 1: List of real-valued binary logic operators used in the neurons of a Differentiable Logic Gate Network. During training, the real-valued functions are used to allow gradient propagation, thus enabling gradient-based learning methods ( LeCun et al. 2015 ; Goodfellow et al. 2016 ) . ID Operator Real-valued equivalent 00 01 10 11 0 FALSE 0 0 0 0 0 1 a\wedge b a\cdot b 0 0 0 1 2 \neg(a\Rightarrow b) a-ab 0 0 1 0 3 a a 0 0 1 1 4 \neg(a\Leftarrow b) b-ab 0 1 0 0 5 b b 0 1 0 1 6 a\oplus b a+b-2ab 0 1 1 0 7 a\vee b a+b-ab 0 1 1 1 8 \neg(a\vee b) 1-(a+b-ab) 1 0 0 0 9 \neg(a\oplus b) 1-(a+b-2ab) 1 0 0 1 10 \neg b 1-b 1 0 1 0 11 a\Leftarrow b 1-b+ab 1 0 1 1 12 \neg a 1-a 1 1 0 0 13 a\Rightarrow b 1-a+ab 1 1 0 1 14 \neg(a\wedge b) 1-ab 1 1 1 0 15 TRUE 1 1 1 1 1 During inference, only the function with the largest weight is used: o=f_{i^{\ast}}(a,b),\quad i^{\ast}=\arg\max_{i\in\{1,\dots,16\}}w_{i}. (2) This reduces computation to binary logical operations, enabling highly efficient predictions. This is referred to as the discrete setting. Here, the inputs must also be binarized. 2.2 Group-Sum Layer The Group-Sum layer serves as the DLGN output layer. The output of the final layer ( \mathbf{o} ) is partitioned into k equal segments, one per class. The outputs in each segment are summed and passed through a softmax to form the predicted probability distribution: \mathbf{p}=softmax\Bigg(\frac{1}{\tau}\begin{bmatrix}\sum\limits_{j=0}^{\frac{n}{k}-1}o_{j},\sum\limits_{j=\frac{n}{k}}^{\frac{2n}{k}-1}o_{j},\ \dots,\sum\limits_{j=\frac{(k-1)n}{k}}^{n-1}o_{j}\end{bmatrix}\Bigg), (3) where n is the number of output neurons, k the number of classes, and \tau a temperature scaling. 2.3 The Role of \tau Temperature \tau strongly affects performance (see Section 5 ). Small \tau values produce sharper predictions and larger gradients,

Read the complete text