Neural Network Symmetrisation in Concrete Settings
Rob Cornish · 2024
Read Neural Network Symmetrisation in Concrete Settings by Rob Cornish (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-09469.txt.
Introduction
Rob Cornish gives a plain-language overview of a general theory for making neural networks respect symmetries, first set out in abstract form in two thousand twenty-four. The overview spells out what the abstract results mean in concrete settings. Two cases get the focus: plain functions that give one fixed output, and random kernels that give output chances. Readers get the high-level picture without the heavy category machinery of the source work.
Catalog record
| CREATOR | Rob Cornish |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-09469
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Network Symmetrisation in Concrete Settings Rob Cornish Abstract Cornish 2024 recently gave a general theory of neural network symmetrisation in the abstract context of Markov categories. We give a high-level overview of these results, and their concrete implications for the symmetrisation of deterministic functions and of Markov kernels. † † year: 2024 † † workshop: Symmetry and Geometry in Neural Representations keywords Equivariance, Symmetrisation, Stochastic 1 Introduction It is often useful to ensure that a neural network f:X\to Y is equivariant with respect to the actions of some group. Recently there has been interest in doing so via symmetrisation techniques ( Yarotsky 2018 ) . Roughly speaking, these approaches take f\coloneqq\mathsf{sym}(f_{0}) where f_{0}:X\to Y is some neural network that is not equivariant, and \mathsf{sym} is a function that maps non-equivariant neural networks to equivariant ones. A variety of different choices of \mathsf{sym} have been considered in the literature to-date, including Janossy pooling ( Murphy et al. 2019 ) , frame averaging ( Puny et al. 2022 ) , canonicalisation ( Kaba et al. 2023 ) , and probabilistic averaging ( Kim et al. 2023 ; Dym et al. 2024 ) . Recently, Cornish 2024 gave a general theory of symmetrisation that characterises all possible choices of \mathsf{sym} , recovering these previous techniques as special cases, and extending to the novel setting of stochastic neural networks f , which had not previously been considered. This framework also streamlines the presentation of compositional and recursive usage of these techniques, and encompasses a range of complex situations such as noncompact translation groups and semidirect products in a uniform way. The results of Cornish 2024 were developed in terms of Markov categories ( Fritz 2020 ) , which provide a high-level algebraic framework for reasoning about probability in an intuitive yet rigorous way. However, Markov categories are currently not widely known in the machine learning community, and so in this paper we present special cases of Cornish 2024 in more familiar concrete settings. We begin in Section 2 with the purely deterministic case considered by Kaba et al. 2023 , and then extend to include randomness in Section 3 . We assume knowledge of only the basic definition of a category (see Appendix A ). Notation Given a category \mathsf{C} , we will denote by \mathsf{C}(X,Y) the set of morphisms X\to Y in \mathsf{C} . We also denote the category of sets and functions by \mathsf{Set} . 2 Deterministic symmetrisation Actions Recall that an action of a group G on a set X is a function \alpha_{X} that takes as input some g\in G and x\in X and returns an output \alpha_{X}(g,x) that also lives in X . We will usually write this output simply as g\cdot x when \alpha_{X} is clear from context. The category of G -sets For every group G , there is a category \mathsf{Set}^{G} . Each object of this category is a G -set , or in other words a set X equipped with a group action \alpha_{X} . The morphisms f:X\to Y in \mathsf{Set}^{G} are then functions that are equivariant with respect to \alpha_{X} and \alpha_{Y} , so that f(g\cdot x)=g\cdot f(x) holds for all x\in X and g\in G . Invariant functions are the special case where \alpha_{Y} is the trivial action with no effect, in which case we have f(g\cdot x)=f(x) . Given two G -sets X and Y , their product is always another G -set that we denote by X\otimes Y . This is simply the cartesian product of X and Y equipped with the diagonal action \alpha_{X\otimes Y} defined as g\cdot(x,y)\coloneqq(g\cdot x,g\cdot y) . Symmetrisation procedures Suppose H\subseteq G is a subgroup. Given any G -set X , we can always obtain an H -set RX with the same underlying set as X , and whose H -action is obtained via restriction , so that \alpha_{RX}(h,x)\coloneqq\alpha_{X}(h,x) . This allows us to define a symmetrisation procedure as any function of the form: {\lx@inpgf@ignorespaces\mathsf{Set}^{H}(RX,RY)} {\lx@inpgf@ignorespaces\mathsf{Set}^{G}(X,Y).} \scriptstyle{\lx@inpgf@ignorespaces\mathsf{sym}} (1) Notice that such a \mathsf{sym} takes a function that is equivariant only with respect to the subgroup H and “upgrades” it to become equivariant with respect to the whole group G . If H is the trivial subgroup consisting of just the identity element, then H -equivariance becomes trivial, and \mathsf{sym} may take as input any arbitrary function X\to Y . The latter case has received the majority of attention in the symmetrisation literature, although Section 3.3 of Kaba et al. 2023 considers the general setup here. Although we focus on ( 1 ), Definition 4.7 of Cornish 2024 formulates symmetrisation slightly more generally in terms of a homomorphism \varphi:H\to G . The case of ( 1 ) is then recovered by considering a subgroup inclusion \varphi:H\hookrightarrow G . As we explain in Appendix B , this approach can be more convenient for describing the composition of multiple symmetrisation procedures in sequence, which allows for “building up” complex equivariance properties in a structured way. A general characterisation The following result allows us to characterise all symmetrisation procedures, i.e. functions of the form ( 1 ). Here G/H denotes the set of H - cosets G/H\coloneqq\{[g]\mid g\in G\}, where [g]\coloneqq\{gh\mid h\in H\} . This set always comes equipped with the G -action g\cdot[g^{\prime}]\coloneqq[gg^{\prime}] , and hence may always be regarded as an object in \mathsf{Set}^{G} . We now have the following result. Theorem 1 . For all choices of the various components involved, there is a bijection {\lx@inpgf@ignorespaces\mathsf{Set}^{H}(RX,RY)} {\lx@inpgf@ignorespaces\mathsf{Set}^{G}(G/H\otimes X,Y)} \scriptstyle{\lx@inpgf@ignorespaces\cong} (2) that sends f\mapsto f^{\sharp} defined as f^{\sharp}([g],x)\coloneqq g\cdot f(g^{-1}\cdot x) . We give a self-contained proof in Appendix C , where we also explain how this result arises very naturally in the context of category theory. It now follows that the following two steps always constitute a symmetrisation procedure for every choice of the function \mathsf{func} shown: {\lx@inpgf@ignorespaces\mathsf{Set}^{H}(RX,RY)} {\lx@inpgf@ignorespaces\mathsf{Set}^{G}(G/H\otimes X,Y)} {\lx@inpgf@ignorespaces\mathsf{Set}^{G}(X,Y).} \scriptstyle{\lx@inpgf@ignorespaces\cong} \scriptstyle{\lx@inpgf@ignorespaces\mathsf{func}} (3) Here the first arrow denotes ( 2 ). Moreover, since ( 2 ) is a bijection, every symmetrisation procedure can be obtained in this way for some choice of \mathsf{func} . In other words, ( 3 ) fully characterises all possible functions of the desired form ( 1 ). Precomposition The question now is, how can we obtain \mathsf{func} ? If we want a “general purpose” strategy that works without further assumptions, then there is only one obvious choice. This is namely the mapping that sends f^{\sharp}:G/H\otimes X\to Y in \mathsf{Set}^{G} to the composition {\lx@inpgf@ignorespaces X} {\lx@inpgf@ignorespaces G/H\otimes X} {\lx@inpgf@ignorespaces Y} \scriptstyle{\lx@inpgf@ignorespaces\Gamma} \scriptstyle{\lx@inpgf@ignorespaces f^{\sharp}} (4) where \Gamma is any choice of morphism X\to G/H\otimes X in \mathsf{Set}^{G} . In other words, \mathsf{func}(f^{\sharp})\coloneqq f^{\sharp}\circ\Gamma. It follows directly from the fact that \mathsf{Set}^{G} is a category and hence closed under composition that \mathsf{func} here is a function of the type required in ( 3 ). A reasonable condition for a symmetrisation procedure \mathsf{sym} to satisfy is stability ( Cornish 2024 , Definition 4.11) : it should be the case that, if f is already G -equivariant, then \mathsf{sym}(f)=f. When \mathsf{func} is obtained via precomposition as in ( 4 ), this holds if and only if \Gamma:X\to G/H\otimes X can be written as \Gamma(x)=(\gamma(x),x) for some \gamma:X\to G/H in \mathsf{Set}^{G} ( Cornish 2024 , Proposition