Mathematical Programming Models for Exact and Interpretable Formulation of Neural Networks
Masoud Ataei, Edrin Hasaj, Jacob Gipp, Sepideh Forouzi · 2025
Read Mathematical Programming Models for Exact and Interpretable Formulation of Neural Networks by Masoud Ataei, Edrin Hasaj, Jacob Gipp, Sepideh Forouzi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14356.txt.
Introduction
Masoud Ataei and three co-authors recast neural network training as a math programming problem in a 2025 paper. Standard training tunes weights by trial-like steps and gives no promise of the best result, nor a clear reason for the network’s shape. Their framework states training as one mixed-integer problem with exact rules.
Switch-like parts of the network are modeled with binary variables, and pruning rules at filter and layer level enforce a sparse shape. Learning, shape choice, and slimming all sit in a single problem whose solution is globally best for its stated goal. That goal balances accuracy, sparsity, and compactness. Domain rules can be added as hard constraints. The payoff is networks that are both lean and open to inspection.
Catalog record
| CREATOR | Masoud Ataei, Edrin Hasaj, Jacob Gipp, Sepideh Forouzi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14356
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mathematical Programming Models for Exact and Interpretable Formulation of Neural Networks Masoud Ataei, Edrin Hasaj, Jacob Gipp, Sepideh Forouzi Abstract This paper presents a unified mixed-integer programming framework for training sparse and interpretable neural networks. We develop exact formulations for both fully connected and convolutional architectures by modeling nonlinearities such as ReLU activations through binary variables and encoding structural sparsity via filter- and layer-level pruning constraints. The resulting models integrate parameter learning, architecture selection, and structural regularization within a single optimization problem, yielding globally optimal solutions with respect to a composite objective that balances prediction accuracy, weight sparsity, and architectural compactness. The mixed-integer programming formulation accommodates piecewise-linear operations, including max pooling and activation gating, and permits precise enforcement of logic-based or domain-specific constraints. By incorporating considerations of interpretability, sparsity, and verifiability directly into the training process, the proposed framework bridges a range of research areas including explainable artificial intelligence, symbolic reasoning, and formal verification. keywords Mixed-Integer Programming, Neural Networks, ReLU Modeling, Convolutional Networks, Network Pruning, Sparsity, Interpretability, Formal Verification, Explainable AI 1 Introduction Deep neural networks have achieved remarkable performance across a variety of tasks, yet their internal decision-making processes often remain opaque. In high-stakes domains such as healthcare and finance, this lack of transparency is a serious concern, as trust, accountability, and regulatory compliance demand interpretable models Rudin (2019) . Relying on black-box models in critical applications can lead to harmful outcomes, and there is a growing consensus that one should instead design inherently interpretable networks from the start. A rich body of research in explainable AI has explored methods to understand or constrain neural network behavior. Broadly, two paradigms exist: post-hoc explanation techniques versus intrinsically interpretable model design Lipton (2018) . Post-hoc methods (e.g., saliency maps Simonyan et al. (2013) , LIME Ribeiro et al. (2016) , SHAP Lundberg and Lee (2017) ) attempt to explain a trained network’s predictions without altering its structure. However, such explanations can be unreliable and do not guarantee understanding of the true underlying decision rules. In contrast, intrinsically interpretable approaches bake transparency into the model itself. Rule extraction algorithms aim to distill a trained neural network into human-understandable if-then decision rules or decision trees Andrews et al. (1995) . Neuro-symbolic methods incorporate logical reasoning or structured knowledge into neural architectures Garcez et al. (2019) , allowing models to reason over interpretable structures. Sparsity-driven methods (e.g., \ell_{1} -regularization, pruning) simplify networks by forcing many weights or neurons to zero, highlighting only the most relevant features Han et al. (2015) ; LeCun et al. (1990) . In parallel, mathematical programming, particularly mixed-integer linear programming (MILP), has emerged as a tool for encoding neural network behavior, enabling verification and certification tasks Tjeng et al. (2017) ; Serra et al. (2020) ; Anderson et al. (2020) ; Nair et al. (2020) ; Dua (2010) ; Thorbjarnarson and Yorke-Smith (2023) ; Schwan et al. (2023) . MILP formulations have been used to verify adversarial robustness Fischetti and Jo (2018) , enforce fairness Pérez et al. (2022) , and compress networks optimally ElAraby et al. (2020) . However, most existing formulations target specific objectives (e.g., robustness verification or post-hoc pruning) and do not simultaneously support interpretability constraints during training. This work presents a unified MILP framework for designing interpretable neural architectures. We encode feed-forward and convolutional layers using exact linear constraints for ReLU activations and affine transformations, while introducing binary variables to model neuron and layer selection. Our formulation supports structured pruning, sparsity promotion, and exact verifiability. By integrating training objectives and structural regularization into a single MILP, we achieve interpretable architectures optimized under verifiable conditions. The remainder of this paper is organized as follows. Section 2 presents a mixed-integer linear programming formulation for dense, fully connected feed-forward neural networks, with explicit modeling of nonlinear activation functions such as ReLU. This section also discusses considerations related to sparsity and interpretability of the resulting models. In Section 3 , the framework is extended to convolutional neural networks through the introduction of constraints that encode convolutional operations, max-pooling layers, and activation gating. Section 5 concludes the paper with a discussion of prospective applications and possible directions for extending the proposed MILP-based approach. 2 MILP Model of Dense Neural Networks We present an MILP formulation that captures the exact behavior of a fully connected feed-forward neural network, enabling simultaneous training, architecture selection, and structural sparsity enforcement. The model encodes affine transformations, piecewise-linear ReLU activations, and layer-level pruning through a unified system of linear constraints and binary variables. This allows the network to be optimized globally with respect to a composite objective balancing predictive accuracy, parameter sparsity, and architectural compactness. To begin, we define the core notation and structural elements of the network. Let the training dataset be denoted by \mathcal{D}=\{(\mathbf{x}_{i},\mathbf{t}_{i})\}_{i=1}^{n}\subset\mathbb{R}^{n_{0}}\times\mathbb{R}^{n_{L+1}}, where \mathbf{x}_{i}\in\mathbb{R}^{n_{0}} is the input and \mathbf{t}_{i}\in\mathbb{R}^{n_{L+1}} is the target output of the i -th sample. The network comprises L\in\mathbb{Z}_{>0} hidden layers and one output layer, indexed by l\in[L+1]:=\{1,2,\ldots,L+1\} . Each layer l contains n_{l} neurons and is parameterized by a weight matrix \mathbf{W}^{(l)}\in\mathbb{R}^{n_{l}\times n_{l-1}} and bias vector \mathbf{b}^{(l)}\in\mathbb{R}^{n_{l}} . For a given input \mathbf{x}_{i} , the network computes intermediate pre-activations \mathbf{z}_{i}^{(l)}\in\mathbb{R}^{n_{l}} and activations \mathbf{a}_{i}^{(l)}\in\mathbb{R}^{n_{l}} at each layer. The ReLU activation function is applied to hidden layers and is defined component-wise by \operatorname{ReLU}(z):=\max(0,z). To exactly model this nonlinearity within the MILP framework, we introduce binary variables that encode the activation regime of each neuron per sample, yielding a precise piecewise-linear characterization of the network’s forward computation. The full optimization model is then given as follows: \displaystyle\min\quad \displaystyle\sum_{i=1}^{n}\left\|\mathbf{a}_{i}^{(L+1)}-\mathbf{t}_{i}\right\|_{2}^{2}+\alpha\lambda\sum_{l=1}^{L+1}\sum_{j=1}^{n_{l}}\sum_{k=1}^{n_{l-1}}u_{j,k}^{(l)} \displaystyle+\tfrac{1}{2}\alpha(1-\lambda)\sum_{l=1}^{L+1}\left\|\mathbf{W}^{(l)}\right\|_{F}^{2}+\beta\sum_{l=1}^{L}\gamma_{l} (1a) s.t. \displaystyle\mathbf{a}_{i}^{(0)}=\mathbf{x}_{i}, \displaystyle\forall i\in[n] (1b) \displaystyle\mathbf{z}_{i}^{(l)}=\mathbf{W}^{(l)}\mathbf{a}_{i}^{(l-1)}+\mathbf{b}^{(l)}, \displaystyle\forall i\in[n],\ \forall l\in[L] (1c) \displaystyle\mathbf{a}_{i}^{(L+1)}=\mathbf{W}^{(L+1)}\mathbf{a}_{i}^{(L)}+\mathbf{b}^{(L+1)}, \displaystyle\forall i\in[n] (1d) \displaystyle a_{i,j}^{(l)}\geq 0, \displaystyle\forall i\in[n],\ \forall l\in[L],\ \forall j\in[n_{l}] (1e) \displaystyle a_{i,j}^{(l)}\geq z_{i,j}^{(l)}, \displaystyle\forall i\in[n],\ \forall l\in[L],\ \forall j\in[n_{l}] (1f) \displaystyle a_{i,j}^{(l)}\leq z_{i,j}^{(l)}-z_{\min}^{(l)}(1-\delta_{i,j}^{(l)}), \displaystyle\forall i\in[n],\ \forall l\in[L],\ \forall j\in[n_{l}] (1g) \displaystyle a_{i,j}^{(l)}\leq z_{\max}^{(l)}\delta_{i,j}^{(l)}, \displaystyle\forall i\in[n],\ \forall l\in[L],\ \forall j\in[n_{l}] (1h) \displaystyle 0\leq a_{i,j}^{(l)},\ z_{i,j}^{(l)}\leq M\gamma_{l}, \displaystyle\forall i\in[n],\ \forall l\in[L],\ \forall j\in[n_{l}] (1i) \displaystyle u_{j,k}^{(l)}\geq W_{j,k}^{(l)},\quad u_{j,k}^{(l)}\geq-W_{j,k}^{(l)}, \displaystyle\forall l\in[L+1],\ \forall j\in[n_{l}],\ \forall k\in[n_{l-1}] (1j) \displaystyle-M\gamma_{l}\leq W_{j,k}^{(l)}\leq M\gamma_{l}, \displaystyle\forall l\in[L],\ \forall j\in[n_{l}],\ \forall k\in[n_{l-1}] (1k) \displaystyle-M\gamma_{l}\leq b_{j}^{(l)}\leq M\gamma_{l}, \displaystyle\forall l\in[L],\ \forall j\in[n_{l}] (1l) \displaystyle\gamma_{l+1}\leq\gamma_{l}, \displaystyle\forall l\in[L-1] (1m) \displaystyle\gamma_{1}=1 (1n) \displaystyle\sum_{k=1}^{n_{l-1}}W_{j,k}^{(l)}\geq\sum_{k=1}^{n_{l-1}}W_{j+1,k}^{(l)}, \displaystyle\forall l\in[L],\ \forall j\in[n_{l}-1] (1o) \displaystyle\delta_{i,j}^{(l)}\in\{0,1\}, \displaystyle\forall i\in[n],\ \forall l\in[L],\ \forall j\in[n_{l}] (1p) \displaystyle\gamma_{l}\in\{0,1\}, \displaystyle\forall l\in[L] (1q) \displaystyle u_{j,k}^{(l)}\geq 0,\quad \displaystyle\forall l\in[L+1],\ \forall j\in[n_{l}],\ \forall k\in[n_{l-1}] (1r) The forward computation in a dense neural network is initialized by assigning each input vector \mathbf{x}_{i} to the activation of the input layer. This anchoring step is encoded in constraint ( 1b ) as follows: \mathbf{a}_{i}^{(0)}=\mathbf{x}_{i},\quad\forall i\in[n]. It establishes the input activations as the foundation for all subsequent propagation through the network layers. For every hidden layer l\in[L] , the pre-activation vector \mathbf{z}_{i}^{(l)}\in\mathbb{R}^{n_{l}} is computed via an affine transformation of the previous layer’s activations, as expressed by constraint ( 1c ) as follows: \mathbf{z}_{i}^{(l)}=\mathbf{W}^{(l)}\mathbf{a}_{i}^{(l-1)}+\mathbf{b}^{(l)},\quad\forall i\in[n]. Here, \mathbf{W}^{(l)} and \mathbf{b}^{(l)} denote the learnable weights and biases associated with layer l . This operation linearly aggregates the incoming signals to produce the pre-activation inputs for each neuron in the layer. The output of the final layer L+1 , which is typically linear to support both regression and classification tasks, is similarly computed by constraint ( 1d ) through \mathbf{a}_{i}^{(L+1)}=\mathbf{W}^{(L+1)}\mathbf{a}_{i}^{(L)}+\mathbf{b}^{(L+1)},\quad\forall i\in[n]. Together, constraints ( 1b