Fusion of Graph Neural Networks via Optimal Transport
Weronika Ormaniec, Michael Vollenweider, Elisa Hoskovec Department of Computer Science, ETH Zürich · 2025
Read Fusion of Graph Neural Networks via Optimal Transport by Weronika Ormaniec, Michael Vollenweider, Elisa Hoskovec Department of Computer Science, ETH Zürich (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21579.txt.
Introduction
Weronika Ormaniec, Michael Vollenweider, and Elisa Hoskovec merge trained graph neural networks into one model with optimal transport, in work from twenty twenty-five. Model fusion can combine skills from separate training runs into a single network. Graph networks add structure that plain weight averaging ignores.
The team aligns weights layer by layer with transport plans and tests three cost choices. Fusion beat plain averaging in every setup tried. Yet gains were smaller and harder won than for simple feed forward networks. Adding graph structure into the transport step did not lift results further. The study maps where transport fusion helps and where graph models resist it.
Catalog record
| CREATOR | Weronika Ormaniec, Michael Vollenweider, Elisa Hoskovec Department of Computer Science, ETH Zürich |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21579
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fusion of Graph Neural Networks via Optimal Transport Weronika Ormaniec, Michael Vollenweider, Elisa Hoskovec Department of Computer Science, ETH Zürich Abstract In this paper, we explore the idea of combining GCNs into one model. To that end, we align the weights of different models layer-wise using optimal transport (OT). We present and evaluate three types of transportation costs and show that the studied fusion method consistently outperforms the performance of vanilla averaging. Finally, we present results suggesting that model fusion using OT is harder in the case of GCNs than MLPs and that incorporating the graph structure into the process does not improve the performance of the method. 1 Introduction In recent years, combining multiple models into one through model fusion has emerged as a new research topic. One of its main uses is the creation of efficient alternatives to ensemble methods that rely on expensive inference ( Singh & Jaggi 2020 ; Wortsman et al. 2022 ; Matena & Raffel 2022 ; Ainsworth et al. 2022 ; Nguyen et al. 2023 ; Kandpal et al. 2023 ; Imfeld et al. 2023 ) . Singh & Jaggi 2020 introduced the idea of combining neural networks by aligning neurons layer-wise using optimal transport (OT) and then averaging their weights. They were able to successfully fuse MLPs, CNNs, and RNNs. Imfeld et al. 2023 applied a similar approach to Transformers. Finally, Jing et al. 2024 explored the fusion of GNNs via deep graph mating (see appendix D ). In this work, we (1) successfully adapt the OT fusion approach to graph convolutional networks (GCNs), and show preliminary results suggesting that (2) GCNs are harder to fuse with OT than MLPs and (3) incorporating the graph structure into the fusion process is not beneficial. 2 Background Optimal Transport One of the main ingredients for the proposed approach is optimal transport (OT). OT addresses the problem of optimally transporting a given source distribution to a target distribution. The theory we outline here is strongly inspired by a comprehensive overview by Thorpe 2018 . We use variations of the problem statement for pairs of discrete probability measures \mu=\sum_{i=1}^{m}\alpha_{i}\delta_{x_{i}} and \nu=\sum_{j=1}^{n}\beta_{j}\delta_{y_{j}} , where \delta_{x} represents the unit mass of some point and \{x_{1},...,x_{m}\} and \{y_{1},...,y_{n}\} are the support of the source and target defined in some shared space S . Further, we have that \alpha and \beta live in the m and n dimensional probability simplex, respectively. EMD Algorithm The EMD algorithm solves the Kantorovich formulation of OT given as the linear program \textstyle T^{*}=\argmin_{T\in U(\alpha,\beta)}\sum_{i=1}^{m}\sum_{j=1}^{n}C_{ij}T_{ij}=\langle T,C\rangle,\vskip-5.0pt where C\in\mathbb{R}^{m,n}_{+} is the cost matrix with C_{ij} being the cost of transporting point x_{i} to y_{j} and T^{*} the optimal transport matrix (TM) living in U(\alpha,\beta):=\{T\in\mathbb{R}_{+}^{m,n}|T\mathbf{1}_{n}=\alpha,T^{T}\mathbf{1}_{m}=\beta }. The objective evaluated at T^{*} is sometimes called the Wasserstein distance between the two distributions. Sinkhorn Algorithm We require two variations of the Sinkhorn algorithm. The first version solves an entropy-regularized and unbalanced version of the classical linear Kantovorich formulation, allowing for a smoother solution space and some slackness in the balancedness constraint originally imposed on the TM by U(\alpha,\beta) . The according objective is given by \min_{T\in\mathbb{R}^{n,m}_{+}}\langle T,C\rangle+\rho_{\alpha}KL(T\mathbf{1}_{m}||\alpha)+\rho_{\beta}KL(T^{T}\mathbf{1}_{n}||\beta)-\epsilon KL(T,\alpha\beta^{T}), where \epsilon controls the regularization strength, parameters \rho_{\alpha} , \rho_{\beta} scale the penalization for deviations from full balancedness and KL(\cdot||\cdot) denotes the Kullbeck-Leibler divergence between two distributions. The second version solves a formulation of OT between two probability measures defined on graphs with feature vectors assigned to their vertices. The minimum of the corresponding objective is termed fused Gromov-Wasserstein distance and measure the distance between structured data with feature information. For a theoretical introduction, we refer the reader to Vayer et al. 2019 . OT Fusion Singh & Jaggi 2020 utilizes OT to fuse networks layer by layer, using either weights or activations as support of the underlying distributions. Formally, for a layer l , the TM of the previous layer is post-multiplied with the weight matrix of the current layer: \hat{W}^{(l,l-1)}:=W^{(l,l-1)}T^{(l-1)}\text{diag}(\frac{1}{\beta^{(l-1)}}) . The current layer is then aligned by pre-multiplying with the transpose of the TM of the current layer: \tilde{W}^{(l,l-1)}:=\text{diag}(\frac{1}{\beta^{(l)}})T^{(l)^{T}}\hat{W}^{(l,l-1)} . After a network is aligned to an anchor model, their weights are averaged. In this work, we assume that \beta_{i}=\beta_{j}\;\forall{i,j} . 3 Models and Methods 3.1 Graph Convolutional Networks A GCN is a type of neural network designed to operate on graph-structured data. Let G=(V,E) represent a graph, where V is the set of vertices and E the set of edges. The vertices can have feature vectors associated with them. The key idea of GCNs is to perform convolutional operations on the graph data, similar to how convolutional layers operate on regular grids in image data. The propagation rule for a single graph convolutional (GC) layer, as introduced by Kipf & Welling 2016 , can be expressed as \textstyle h^{(l+1)}_{i}=\text{ReLU}(W^{(l+1,l)}\frac{1}{\sqrt{\text{deg}_{i}}}\sum_{j\in\mathcal{N}_{i}}\frac{1}{\sqrt{\text{deg}_{j}}}h^{(l)}_{j}+b^{(l+1,l)}).\vskip-5.0pt Here, h^{(l)}_{i} is a feature vector of the i^{th} vertex at layer l , \mathcal{N}_{i} the set of neighbors of vertex i (including vertex i itself), and W^{(l,l+1)} and b^{(l,l+1)} the weights and biases for that layer. In the context of this work, it is important to note that the weight matrix is shared among vertices. Prior to applying the rectified linear unit (ReLU) activation function, one can incorporate batch normalization (BN). In this work we consider the GCN architecture proposed in Dwivedi et al. 2020 . Firstly, vertex features are processed with an embedding layer, then g GC layers with or without BN depending on the experiment are applied sequentially producing a hidden representation of each graph vertex feature. Finally, the hidden representation is averaged across graph vertices and p fully connected layers are applied sequentially. 3.2 GCN Fusion 3.2.1 Graph Convolutional Layer Due to weight sharing between graph vertices, the weights and biases of the GCN have exactly the same structure as those of the MLPs. Hence, the weight-based alignment for GCN layers works exactly the same as for MLPs. However, the pre-activations of graph convolutional layers are different from the ones in MLPs, namely, they are in the form of graphs. This leaves room for exploiting the graph structure to potentially improve OT fusion performance. During the fusion process we pass the same graphs through both networks and keep track of the matching graph vertices in between networks. This means that when we pass a batch of graphs to the network, each neuron of a layer has a set of graphs associated with its output. The structure of the graphs between neurons and between networks is the same, meaning that there is no need to additionally match graph vertices. For information on how to fuse other components of GCNs, i.e. biases, BN, and embedding layer see appendix B . 3.2.2 Optimal Transport Cost Solving the OT problem for fusing two GCN layers requires defining the cost of transporting a source point to a target point. Here, a point corresponds to a set of scalar activation graphs. To compute a distance between such sets, we first define a pairwise distance between scalar graphs. Let (G_{i},G_{j}) be a pair of graphs with identical sets of vertices V and edges E and a_{i}(u)\in\mathbb{R} and a_{j}(u)\in\mathbb{R} their vertex features, respectively. We consider three different ways to define a transport cost c_{ij} : (i) Euclidean Feature Distance (EFD) , where