Identifiability of Deep Polynomial Neural Networks

Konstantin Usevich, Clara Dérand, Ricardo Borsoi, Marianne Clausel · 2025

Read Identifiability of Deep Polynomial Neural Networks by Konstantin Usevich, Clara Dérand, Ricardo Borsoi, Marianne Clausel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17093.txt.

Introduction

Konstantin Usevich and co-authors settle when deep polynomial networks can be pinned down from their behavior, in this 2025 paper. These networks use polynomial activations, giving them a rich algebra. Identifiability asks whether the inside weights are the only ones that could produce the observed input-output map, a trait that makes a model readable.

The team maps how activation degrees and layer widths trade off in deciding the answer, for networks with and without bias terms. Networks whose layers never widen turn out identifiable under mild conditions. Encoder-decoder shapes qualify too, as long as the decoder does not widen too fast against the degrees. The proofs lean on splitting tensors into low-rank parts. The paper also settles a standing conjecture about the size of the shape space these networks trace.

Catalog record

CREATORKonstantin Usevich, Clara Dérand, Ricardo Borsoi, Marianne Clausel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE112 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-17093

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifiability of Deep Polynomial Neural Networks Konstantin Usevich, Clara Dérand, Ricardo Borsoi, Marianne Clausel Abstract Polynomial Neural Networks (PNNs) possess a rich algebraic and geometric structure. However, their identifiability—a key property for ensuring interpretability—remains poorly understood. In this work, we present a comprehensive analysis of the identifiability of deep PNNs, including architectures with and without bias terms. Our results reveal an intricate interplay between activation degrees and layer widths in achieving identifiability. As special cases, we show that architectures with non-increasing layer widths are generically identifiable under mild conditions, while encoder-decoder networks are identifiable when the decoder widths do not grow too rapidly. Our proofs are constructive and center on a connection between deep PNNs and low-rank tensor decompositions, and Kruskal-type uniqueness theorems. This yields both generic conditions determined by the architecture, and effective conditions that depend on the network’s parameters. We also settle an open conjecture on the expected dimension of PNN’s neurovarieties , and provide new bounds on the activation degrees required for it to reach its maximum. 1 Introduction Neural network architectures which use polynomials as activation functions— polynomial neural networks (PNN)—have emerged as architectures that combine competitive experimental performance (capturing high-order interactions between input features) while allowing a fine grained theoretical analysis. On the one hand, PNNs have been employed in many problems in computer vision [ 1 , 2 , 3 ] , image representation [ 4 ] , physics [ 5 ] and finance [ 6 ] , to name a few. On the other hand, the geometry of function spaces associated with PNNs, called neuromanifolds , can be analyzed using tools from algebraic geometry. Properties of such spaces, such as their dimension, shed light on the impact of a PNN architecture (layer widths and activation degrees) on the expressivity of feedforward, convolutional and self-attention PNN architectures [ 7 , 8 , 9 , 10 , 11 ] . They also determine the landscape of their loss function and the dynamics of their training process [ 7 , 12 , 13 ] . Moreover, PNNs are also closely linked to low-rank tensor decompositions [ 14 , 15 , 16 , 17 ] , which play a fundamental role in the study of latent variable models due to their uniqueness properties [ 18 ] . In fact, single-output 2-layer PNNs are equivalent to symmetric tensors [ 7 ] . This connection provides a rich set of tools for studying the identifiability of PNNs. Identifiability—whether the parameters and, consequently, the hidden representations of a NN can be uniquely determined from its response—is a key question in NN theory [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 ] . Identifiability is critical to ensure interpretability in representation learning [ 32 , 33 , 34 ] , to provably obtain disentangled representations [ 35 ] , and in the study of causal models [ 36 ] . It is also critical to understand how the architecture affects the inference process and to support manipulation or “stitching” of pretrained models and representations [ 34 ] . The identifiability of deep PNNs is intimately linked to the dimension of neurovarieties and to the uniqueness of tensor decompositions, but few studies investigated such connections. Existing work has considered specific types of self-attention [ 9 ] and convolutional [ 8 ] layers, and feedforward PNNs without bias [ 11 ] . However, these results focus on local identifiability: the number of possible parametrizations was shown to become finite, which is in contrast to global identifiability or uniqueness (a single solution). Moreover, current results for feedforward networks only show that local identifiability holds for very high activation degrees, or for networks with the same widths in every layer [ 11 ] . A standing conjecture is that this holds for any PNN with degrees at least quadratic and non-increasing layer widths [ 11 ] , which parallels identifiability results of ReLU networks [ 28 ] . However, a general theory of (global) identifiability of deep PNNs is still missing. 1.1 Our contribution We provide a comprehensive analysis of the identifiability of deep PNNs. We prove that an L -layer PNN is (globally) identifiable if and only if every 2-layer block composed by a pair of two successive layers is identifiable. This surprising result tightly links the identifiability of shallow and deep polynomial networks, which is a key challenge in the general theory of NNs. Moreover, our results reveal an intricate interplay between activation degrees and layer widths in achieving identifiability. As special cases, we show that architectures with non-increasing layer widths (i.e., pyramidal nets) are generically identifiable, while encoder-decoder (bottleneck) networks are identifiable when the decoder widths do not grow too rapidly. We also show that the minimal activation degrees required to render a PNN identifiable (which is equivalent to its activation thresholds ) is only linear in the layer widths, compared to the quadratic bound in [ 11 , Theorem 18] . These results not only settle but generalize conjectures stated in [ 11 ] . Moreover, we also address the case of PNNs with biases (which was overlooked in previous theoretical studies) by leveraging a homogeneization procedure. Our proofs are constructive and are based on a connection between deep PNNs and partially symmetric canonical polyadic tensor decompositions (CPD). This allows us to leverage Kruskal-type uniqueness theorems for tensors to obtain global identifiability results. We obtain generic conditions for identifiability of some PNN architecture, as well as specific conditions that can certify the uniqueness of a given set of PNN weights. Our results also shed light on the geometry of the neurovarieties , as they lead to conditions under which its dimension reaches the expected (maximum) value. 1.2 Related works Polynomial NNs: Several works studied PNNs from the lens of algebraic geometry using their associated neuromanifolds and neurovarieties [ 37 , 7 ] and their connection to tensor decompositions. Kileel et al. [ 7 ] studied the expressivity or feedforward PNNs in terms of the dimension of their neurovarieties. An analysis of the neuromanifolds for several architectures was presented in [ 10 ] . Conditions under which training losses do not exhibit bad local minima or spurious valleys were also investigated [ 13 , 12 , 38 ] . The links between training 2-layer PNNs and low-rank tensor approximation [ 13 ] as well as the biases gradient descent [ 39 ] have been established. Recent work computed the dimensions of neuromanifolds associated with special types of self-attention [ 9 ] and convolutional [ 8 ] architectures, which is linked to local identifiability (a finite number of representations). For feedforward PNNs, local identifiability was demonstrated for networks with the same widths in every layer [ 11 ] , while stronger uniqueness results are available for the 2-layer case with more general polynomial activations [ 40 ] . Local identifiability also holds when the activation degrees are larger than a so-called activation threshold [ 11 ] . PNNs are also linked to factorization machines [ 41 ] ; this led to the development of efficient tensor-based learning algorithms [ 42 , 43 ] . Note

Read the complete text