Smoothness Errors in Dynamics Models and How to Avoid Them Edward Berman, Luisa Li, Jung Yeon Park, Robin Walters Abstract Modern neural networks have shown promise for solving partial differential equations over surfaces, often by discretizing the surface as a mesh and learning with a mesh-aware graph neural network. However, graph neural networks suffer from oversmoothing, where a node’s features become increasingly similar to those of its neighbors. Unitary graph convolutions, which are mathematically constrained to preserve smoothness, have been proposed to address this issue. Despite this, in many physical systems, such as diffusion processes, smoothness naturally increases and unitarity may be overconstraining. In this paper, we systematically study the smoothing effects of different GNNs for dynamics modeling and prove that unitary convolutions hurt performance for such tasks. We propose relaxed unitary convolutions that balance smoothness preservation with the natural smoothing required for physical systems. We also generalize unitary and relaxed unitary convolutions from graphs to meshes. In experiments on PDEs such as the heat and wave equations over complex meshes and on weather forecasting, we find that our method outperforms several strong baselines, including mesh-aware transformers and equivariant neural networks. Our code is available here . † † affiliationnotice: Equal contribution 1 Introduction Solving partial differential equations (PDEs) is crucial across many scientific and engineering domains, including acoustics, fluid dynamics, and electrodynamics. Recently, neural networks have been explored as alternatives to analytic and traditional numerical methods for solving PDEs. Neural networks offer faster inference ( 20 ) , discretization free solutions ( 48 ) , better robustness to partial observability ( 68 ; 36 ; 56 ) , and synergy with existing finite element methods ( 33 ) . Figure 1 : Qualitative comparison of autoregressive model predictions for the heat equation on the armadillo mesh at timestep T=190 . Our R-UniMesh model remains faithful to the ground truth during each step of the rollout, whereas the EMAN model over smooths and the Hermes model under smooths. A more complete comparison over several timesteps is in Section C.4 , Table 5 . However, neural network models often have architectural biases that hurt their ability to model certain dynamics. In particular, many deep learning methods solve PDEs by discretizing the domain into a grid or mesh and modeling the solution using a graph neural network ( 38 ; 61 , e.g.,) . Unfortunately, graph neural networks (GNNs) tend to oversmooth ( 46 ) , where adjacent node features become increasingly similar over successive iterations of message passing. The phenomenon of oversmoothing occurs in a variety of settings ( 12 ; 9 ; 40 ; 66 ; 41 ; 3 ; 72 ) and hampers the performance of deep GNNs. To address oversmoothing, 41 propose using unitary graph convolutions, which constrain weight matrices to be unitary. This ensures that the linear transformations preserve norms and remain invertible, improving network stability. They also prove that unitary convolutions prevent oversmoothing by preserving the Rayleigh quotient, a measure of graph smoothness. However, this poses a new problem: many dynamics problems commonly solved using GNNs require some amount of smoothing. For example, heat diffusion on graphs and meshes naturally smooths the input node features. Using unitary graph convolutions in such problems would result in undersmoothing and does not give a complete solution to smoothness errors. In this work, we first theoretically characterize the limitations of unitary convolutions for dynamics problems. In particular, we derive a lower bound on the approximation error of unitary functions and show that unitary functions are overconstrained for dynamical systems where the solution’s norm has high angular dependence. To address this issue, we propose relaxed unitary convolutions , which balance smoothness preservation with modeling fidelity, outperforming existing methods on dynamic systems that require natural smoothing. We also generalize both the Rayleigh quotient and unitary convolution framework from graphs to meshes so that relaxations can be applied in this setting. Finally, we systematically investigate smoothness tendencies of different mesh-GNN architectures and find that our mechanism for approximately preserving smoothness is key to successful modeling, providing equal or greater improvement to other inductive biases such as equivariance. In summary, our contributions are the following: 1. Derive a lower bound on the approximation error of unitary functions, demonstrating that they are overly restrictive when predicting dynamics with high angular dependence in the solution’s norm ( Section 4 ). 2. Introduce relaxed unitary convolutions that balance accuracy with smoothness preservation, and extend both the Rayleigh quotient and unitary convolution framework to meshes ( Section 5 ). 3. Empirically analyze the smoothness behavior of various GNN architectures on complex dynamical systems, showing that controlling smoothness can match or outperform strong baselines ( Section 6 ). 2 Related Works Oversmoothing and undersmoothing in GNNs. Like 41 , our work quantifies the effect of neural networks on the Rayleigh quotient ( 14 ) of a graph. 41 prove that unitary functions, and in particular the unitary convolution network, strictly preserve the Rayleigh quotient and therefore the smoothness of input graphs. However, we show both theoretically and empirically how this property can be overconstraining in GNNs. Other approaches to quantifying smoothness in PDE solutions have used the Matérn kernel ( 10 ; 21 ) or decay rate exponents ( 43 ) , but none have considered the Rayleigh quotient for dynamics models as we do. Our work is perhaps most similar to 40 , who also point out that some smoothing can be useful for certain regression tasks but do not consider dynamics modeling specifically. Similarly, 46 point out that GCNs ( 42 ) can be understood as a special case of Laplacian smoothing, which is a key reason why GCNs work at all. In fact, 42 argue that their architecture can be understood as a differentiable and parameterized generalization of the 1 -dim Weisfeiler-Lehman algorithm ( 45 ) , indicating that even randomly initialized GCNs can be performant due to the way they smooth information throughout the network. Despite these findings, there is comparatively less work studying the role of smoothness in spatio-temporal modeling tasks. While 50 study issues with message-passing based GNNs for spatio-temporal modeling, they focus on oversquashing , where information fails to propagate to distant nodes, whereas our work addresses oversmoothing and undersmoothing . Dynamics modeling over graphs and meshes. Our work focuses on dynamics modeling where PDE solutions are discretized as signals on graphs and meshes through the lens of smoothness. Many physical systems, such as wave propagation ( 26 ) , heat diffusion ( 6 ) , phase fields ( 11 ; 47 ) , fluid flows ( 16 ; 1 ) , and climate systems ( 30 ) can be described by systems of PDEs. Deep learning based approaches are increasingly used to solve these PDEs in these domains where numerical solving is difficult ( 78 ; 19 ; 1 ; 48 ; 57 ; 13 ; 51 ; 61 ; 49 ; 82 ; 22 ) . For PDE solving on meshes, these dynamics can be formulated extrinsically by embedding the manifold into Euclidean space ( 67 ; 62 ) , or intrinsically by defining evolution directly in the coordinates of local tangent spaces ( 15 ; 23 ; 55 ; 53 ; 7 ; 61 ; 73 ; 54 ) . While 15 , 62 , and 73 contain isolated experiments related to dynamics modeling, only our work and 61 study how the choice of Euclidean versus locally defined coordinate representations in the network can affect convergence to PDE solutions. Furthermore, our work is distinct from 61 in that only we directly assess how these design choices affect neural network smoothing behavior . Benchmarking PDE Surrogate Models. While the physical symmetries of many dynamical systems are well understood ( 59 ; 79 ; 10 ) , smoothness has received less attention. The performance of deep dynamics models is typically measured either via quantitative error metrics against the ground truth or their preservation of underlying physical laws, such as spectral energy errors ( 79 ) or equivariance errors ( 79 ; 80 ; 81 ) . Our work is novel in our application of the Rayleigh quotient in quantifying the smoothing effect of trained GNN dynamics models. Furthermore, we are among the first to design architectures with inductive biases that encourage the model to match the Rayleigh quotient of the labeled graphs. Other works have explored using the Rayleigh quotient as an auxiliary loss ( 65 ) , as positional encodings ( 25 ) , and as a hard constraint to preserve smoothness regardless of the true labels 41 . In contrast, only our work and the work of 70 use architectural inductive biases to match the smoothness of labeled graphs, and only our work evaluates how well the true smoothness of dynamical systems is recovered. 3 Background We first recall the definition of the Rayleigh quotient, a measure of smoothness on graphs, and provide background on unitary convolutions and their invariance to the Rayleigh quotient. We also introduce the mesh data type, which we later use to extend the unitary convolution framework from graphs to meshes. 3.1 Rayleigh quotient To measure the smoothness of a signal on a graph, we use the Rayleigh quotient as defined in 14 . Definition 1 (Rayleigh quotient, ( 14 ) ) . Given an undirected graph \mathcal{G}=(V,E) with |V|=n nodes and adjacency matrix \mathbf{A}\in\{0,1\}^{n\times n} , let \mathbf{D}\in\mathbb{R}^{n\times n} be a diagonal matrix where the i -th entry \mathbf{D}_{ii}=d_{i} the degree of node i . Let s\colon V\to\mathbb{C}^{d} be a function from nodes to features. Denote by \tilde{\mathbf{A}}=\mathbf{D}^{-1/2}\mathbf{AD}^{-1/2} the normalized adjacency matrix and \mathbf{X}\in\mathbb{C}^{n\times d} a matrix with the i -th row set to feature vector s(i) . The Rayleigh quotient is R_{\mathcal{G}}(\mathbf{X})=\frac{1}{2}\frac{{\sum_{(u,v)\in E}}\left\|\frac{s(u)}{\sqrt{d_{u}}}-\frac{s(v)}{\sqrt{d_{v}}}\right\|^{2}}{{\sum_{w\in V}}||s(w)||^{2}} (1) or \Tr{(\mathbf{X^{\dagger}(I-\tilde{A})X})}\cdot||\mathbf{X}||^{-2}_{F} in matrix form. We will often abbreviate the Laplacian as \mathbf{L=(I-\tilde{A})} . Intuitively, the Rayleigh quotient measures the mean difference in node features for adjacent nodes. A graph with identical node features has a Rayleigh quotient of zero. 3.2 Unitary Convolution 41 define two different models that preserve the Rayleigh quotient using unitary functions, which satisfy \mathbf{U^{\dagger}U=I} . In particular, they define the separable unitary convolution f_{\rm UniConv}^{\mathrm{sep}}(\mathbf{X;A})=\mathbf{\exp(iAt)XU},\quad\mathbf{U^{\dagger}U=I} (2) and the Lie unitary convolution f_{\rm UniConv}^{\mathrm{Lie}}(\mathbf{X;A})=\mathbf{\exp(AXW)},\quad\mathbf{W=-W^{\dagger}} (3) where \exp(\cdot) denotes the matrix exponential. We provide further background material on the matrix exponential and its relationship to unitary matrices in Section A.1 . The authors show that unitary convolutions are mathematically constrained to preserve the Rayleigh quotient: Proposition 1 (Invariance of Rayleigh quotient, Proposition 6 in 41 ) . Given an undirected graph \mathcal{G} on n nodes with normalized adjacency matrix \widetilde{\mathbf{A}}=\mathbf{D}^{-1/2}\mathbf{A}\mathbf{D}^{-1/2} , the Rayleigh quotient is invariant under normalized unitary or orthogonal graph convolution, i.e. R_{\mathcal{G}}(\mathbf{X})=R_{\mathcal{G}}(f_{\rm UniConv}(\mathbf{X})) where f_{\rm UniConv} is either seperable or Lie. 3.3 Mesh Data A (triangular) mesh \mathcal{M} consists of a set (\mathcal{V},\mathcal{E},\mathcal{F}) , where \mathcal{V} is a set of vertices, \mathcal{E}=\{(i,j)\} is a set of ordered vertex indices i , j connected by an edge, and \mathcal{F}=\{(i,j,k)\} is the set of ordered vertex indices i , j , k connected by a triangular face. The mesh generalizes graphs by including high order connectivity information via the inclusion of faces. We assume that the mesh is a 2- dimensional manifold embedded in \mathbb{R}^{3} , i.e. a manifold mesh. The definition of the manifold condition for a mesh is given in Definition 9 ( Section A.7 ). 4 Theory: Unitary Models are Overconstrained While unitary models on graphs can be useful because they preserve the Rayleigh quotient, this section illustrates how unitary models can be overly constrained. In particular, we derive an approximation error lower bound that clarifies the approximation limits of unitary models. We start by establishing our unitary approximation learning framework. 4.1 Preliminaries Let Z=\mathbb{C}^{n} be a domain with data probability density p\colon Z\to\mathbb{R} . Let u\colon\mathbb{C}^{n}\to\mathbb{C}^{n} be a unitary function and let f\colon\mathbb{C}^{n}\to\mathbb{C}^{n} be the target function. Denote the regression error by \text{err}_{\mathrm{reg}}(u)=\int_{Z}p(z)\lVert u(z)-f(z)\rVert_{2}^{2}dz. Group Invariance. Our main result relies on the theory of approximation error for group invariant functions h . We review these concepts in detail in Section A.4 and provide informal definitions in the paragraph that follows. A group invariant function h satisfies h(z)=h(gz) for all g\in G , z\in Z . Let Gz=\{gz\colon g\in G\} be the orbit of z . A fundamental domain F of a group G in Z is a set of orbit representatives. The domain Z can be written as the union of conjugates, Z=\cup_{g\in G}gF , where the conjugate is defined gF=\{gz\colon z\in F\} . Denote the integrated density on an orbit by p(Gz)=\int_{Gz}p(z)dz. Finally, denote the variance of a function f on an orbit Gz by \mathbb{V}_{Gz}[f] . The approximation error lower bound for an invariant function is given by the following proposition. Proposition 2 (Theorem 4.8 in 77 ) . For a G -invariant function h , the regression error is bounded below by \err\geq\int_{F}p(Gz)\mathbb{V}_{Gz}[f]dz . Proposition 2 was initially stated for real-valued functions in 77 , but can be applied to complex valued functions without loss of generality. Furthermore, 77 provide numerical evidence that Proposition 2 is a tight bound. 4.2 Unitary Approximation Error Lower Bound In this subsection, we state our main theoretical result, which demonstrates that unitary neural networks are overconstrained when the norm of the ground truth function has a high angular dependence. Recall the definition of \mathrm{SU}(n) , the group of rotations in \mathbb{C}^{n} : \mathrm{SU}(n)=\{U\in\mathbb{C}^{n\times n}\colon\det(U)=1\}. We can now give an approximation error lower bound for unitary models. See Section A.5 for the proof. Theorem 1 . Let F be a fundamental domain of \mathrm{SU}(n) in Z , e.g. F=\{te\colon t\in\mathbb{R}_{+}\} where e is a standard basis vector of \mathbb{C}^{n} . The approximation error of u of f has lower bound \int_{Z}p(z)\lVert u(z)-f(z)\rVert_{2}^{2}dz\geq\int_{F}p(\lVert te\rVert)\mathbb{V}_{Gz}[\rVert f\lVert]dz. The proof of Theorem 1 uses the reverse triangle inequality before applying Proposition 2 . Intuitively, the fundamental domain enumerates all concentric spheres S^{2n-1} embedded in \mathbb{C}^{n} . Unitary functions are complex valued rotations and reflections that preserve the norm of data points that live on each sphere. The error lower bound is given by the variance of the norm of f averaged over each concentric sphere where the norm of u is constant. Our result suggests that unitary functions can be particularly overly constraining when the norm of f has a high angular dependence. 5 Unitary Convolution Constraint Relaxation Since Theorem 1 informs us that a unitary convolution network may be overconstraining when the ground truth is not perfectly smoothness preserving, we propose two methods for relaxing unitary convolutions and describe how to extend these architectures from graphs to meshes. We name these methods Taylor truncation and encoder-decoder. The Taylor truncation method allows precise control of the extent of relaxation, whereas the encoder-decoder method scales more easily with the number of parameters in the network. The Taylor truncation method is especially useful in cases where the true smoothness is known a priori, in which case theoretical results from the literature (see Section A.8 ) can inform what truncation order \mathbf{T_{\rm max}} is needed to achieve enough relaxation. We will use the first relaxation for a motivating experiment in Section 6.1 and the second for a more challenging set of tasks in Section 6.2 . A comparison of the encoder-decoder and Taylor truncation methods is shown in Figure 2 . We will use f_{\rm Layer} to define individual layers and small caps text to define architectures constructed from those layers. Figure 2 : Top: After zero padding, individual unitary blocks are stacked and the output is fed into an unconstrained decoder. Bottom: Each block uses Taylor truncated unitary convolution. 5.1 Relaxed Unitary Convolution via Taylor Truncation We relax Lie unitary convolution by truncating the Taylor series approximations used in Equation 3 . We note that 41 do propose their own constraint relaxation for separable unitary convolutions Equation 2 . However, their approach conflates two sources of relaxation: Taylor series truncation of the matrix exponential and letting \mathbf{U} remain unconstrained, making it difficult to measure the individual contributions of each source and to tune the extent of the relaxation. By relaxing the Lie unitary convolution rather than the separable unitary convolution, we isolate the architectural component that alters the Rayleigh quotient. Instead of approximating the matrix exponential using enough Taylor series terms so that the truncation error is negligible, we truncate at some T=\mathbf{T_{\rm max}} where \mathbf{T_{\rm max}} controls the extent of the relaxation. Our Taylor-relaxed unitary convolution layer is defined f_{\rm Relaxed}(\mathbf{X};\mathbf{A},\mathbf{T}_{\rm max})=\overset{\mathbf{T}_{\rm max}}{\underset{i=0}{\sum}}\frac{1}{i!}\mathbf{L}^{i}(\mathbf{X}) (4) where \mathbf{L}(\mathbf{X)}=\mathbf{AXW} , \mathbf{W}=-\mathbf{W^{\dagger}} . While this approach does not preserve the Rayleigh quotient for small \mathbf{T_{\rm max}} , we recover the standard Lie unitary convolution in Equation 3 in the limit as T\to\infty . Motivated by the desire to find an appropriate \mathbf{T_{\rm max}} that applies only a small perturbation to the Rayleigh quotients of input graphs, in Section B.2 we conduct a sensitivity analysis of the Rayleigh quotient to different Taylor series truncations. Consistent with 41 , we find that \mathbf{T_{\rm max}}=10 is sufficient to preserve the Rayleigh quotient. We will refer to models constructed from relaxed layers in Equation 4 as R-UniGraph . 5.2 Relaxed Unitary Convolutional Models via Encoder-Decoder In this section, we note a limitation of relaxed unitary convolution via Taylor truncation that makes it difficult to scale, and propose an alternative relaxation method that addresses this. Since unitary layers cannot change the channel dimension of the node features, the only way to increase the number of parameters in the network is to increase the number of unitary layers. This means that scaling unitary convolutional models requires making the model deeper, which can make training unstable ( 5 ) . As an alternative, we propose an encoder-decoder method, which first zero-pads the input node features to the desired hidden dimension. This allows us to increase the parameter count without increasing depth. Zero padding also trivially preserves the Rayleigh quotient since it preserves norms. Concretely, we define our zero padding function f_{\rm pad}\colon\mathbb{R}^{n\times d_{\rm in}}\to\mathbb{R}^{n\times d_{\rm out}} by \mathbf{X}\mapsto\mathbf{X}\oplus 0 , where 0 is the \mathbb{R}^{n\times(d_{\rm out}-d_{\rm in})} zero matrix. We define our k -layer encoder E by E=f_{\rm UniConv}^{(k)}\circ\dots\circ f_{\rm UniConv}^{(1)}(f_{\rm pad}(\mathbf{X}),\mathbf{A}) (5) where f_{\rm UniConv} is either separable or Lie. The decoder layer D then serves two purposes: ( i ) map to the target node feature dimension and ( ii ) break the unitary constraint. The decoder layer can be any network, such as an MLP or GCN. 5.3 Relaxed Unitary Convolution on Meshes We now generalize unitary convolutional models, relaxed unitary convolutional models, and the mathematical definition of Rayleigh quotient from graphs to meshes. This enables us to solve dynamics problems on manifolds by discretizing them as meshes, such as testing the thermal stability of mechanical parts or weather forecasting on Earth. In particular, we prove that, under modest assumptions on the mesh triangulation, unitary convolution with a weighted adjacency matrix preserves the Rayleigh quotient on meshes ( Definition 3 ); enabling generalization of both unitary and relaxed unitary models to meshes. Mesh Rayleigh Quotient. We first generalize the Rayleigh quotient from graphs to meshes by using the mesh Laplacian instead of the graph Laplacian. The Laplacian on a mesh is typically defined as the symmetric cotangent Laplacian ( 64 ) given in Equation 6 , which approximates the Laplace-Beltrami operator for the continuous manifold which the mesh discritizes. For a scalar function s defined on nodes, (\mathbf{\tilde{L}}(s))_{i}=\frac{1}{2A_{i}}\underset{j\in\mathcal{N}(i)}{\sum}\left(\cot\alpha_{ij}+\cot\beta_{ij}\right)(s_{j}-s_{i}) (6) where \mathcal{N}(i) denotes the adjacent vertices of i , \alpha_{ij} and \beta_{ij} are the angles opposite edge (i,j) , and A_{i} is the vertex area of i . We use the barycentric cell area for A_{i} . We note that it is invalid to define the mesh Rayleigh quotient by replacing \mathbf{L} in Equation 1 with the symmetric cotangent Laplacian \mathbf{\tilde{L}} in Equation 6 . The cotangent weights in Equation 6 may be negative, which means that the Rayleigh quotient is no longer a valid measure of smoothness ( 66 , Definition 1,) . To address this, we use the Robust Laplacian ( 71 ) , which performs a minimal edge rewiring of the mesh so that the cotangent weights obey the intrinsic Delaunay criterion . Definition 2 (Intrinsic Delaunay Criterion, ( 8 ) ) . For all faces connected by an edge (i,j) with opposite angles \alpha_{ij} and \beta_{ij} , \alpha_{ij}+\beta_{ij}\leq\pi . Concretely, this means that our Laplacian weights are both symmetric and the off-diagonals are nonnegative; the edge rewiring simply provides an alternative discretization of the same manifold. Denote by \mathcal{W} the cotangent weights \mathcal{W}_{ij}=\begin{cases}\frac{1}{2}\left(\cot\alpha_{ij}+\cot\beta_{ij}\right),&j\in\mathcal{N}(i)\\ -\underset{k\in\mathcal{N}(i)}{\sum}\mathcal{W}_{ik},&i=j\\ 0,&\text{Otherwise.}\end{cases} (7) We define a novel Rayleigh quotient for meshes as follows. Definition 3 (Mesh Rayleigh Quotient) . Let \mathcal{M}=(\mathcal{V},\mathcal{E},\mathcal{F}) be a mesh with |V|=n nodes. Denote by \mathcal{W} the cotangent weights corresponding to the Robust Laplacian \tilde{\mathbf{L}} . Denote by \mathcal{E}^{\prime} the rewired edge set given by \mathcal{E}^{\prime}=\{(u,v)\colon\mathcal{W}_{uv}\neq 0\} . Let s and \mathbf{X} be the same as in Definition 1 . The mesh Rayleigh quotient is defined R_{\mathcal{M}}(\mathbf{X})=\frac{1}{2}\frac{\underset{(u,v)\in\mathcal{E}^{\prime}}{\sum}\mathcal{W}_{uv}\left\|\frac{s(u)}{\sqrt{d_{u}}}-\frac{s(v)}{\sqrt{d_{v}}}\right\|^{2}}{\underset{w\in V}{\sum}\left\|s(w)\right\|^{2}}=\frac{\Tr{(\mathbf{X^{\dagger}\tilde{L}X})}}{||\mathbf{X}||^{2}_{F}}. Unitary Mesh Convolution. We now make similar modifications to generalize unitary convolution from graphs to meshes. Specifically, we modify the functions in Equation 2 and Equation 3 by incorporating the cotangent weights ( Equation 7 ) into the normalized adjacency matrix \mathbf{\tilde{A}} . We note that this edge weighting been shown to improve PDE solving numerically ( 18 ; 71 ) and we are the first to use it in deep dynamics models. In order to prove that incorporating these weights preserves the Rayleigh quotient given by Definition 3 , we assume the mesh already satisfies the Delaunay criterion. Assumption 1 (Mesh Weights Obey the Delaunay Criterion) . For a mesh \mathcal{M} , the mesh is manifold and all angles obey the Delaunay Criterion given by Definition 2 . We note that there are existing triangulation strategies that a practitioner can use to ensure that mesh edges satisfy this criterion as a standard data preprocessing step ( 37 ; 71 ) , see Section A.7 for details. With this assumption, we will now define unitary mesh convolution. Let \mathbf{D} be the weighted degree matrix defined by \mathbf{D}_{ii}=\sum_{i\neq j}\mathcal{W}_{ij} . Let \odot represent the Hadamard product which performs element-wise matrix multiplication. Let \mathbf{\tilde{A}} be the normalized adjacency matrix \mathbf{\tilde{A}}=\mathbf{D^{-1/2}}\left(\mathcal{W}\odot\mathbf{A}\right)\mathbf{D^{-1/2}} . We define separable unitary mesh convolution as f_{\rm UniMeshConv}^{\mathrm{Sep}}(\mathbf{X;A,\mathcal{W}})=\exp(i\mathbf{\tilde{A}})\mathbf{XU} (8) and Lie unitary mesh convolution as f_{\rm UniMeshConv}^{\mathrm{Lie}}(\mathbf{X;A},\mathcal{W})=\exp(\mathbf{\tilde{A}}\mathbf{XW}) (9) where \mathbf{UU^{\dagger}=I} and \mathbf{W+W^{\dagger}=0} . The following Corollary (proven in Section A.6 ) states that Equation 8 and Equation 9 preserve the Rayleigh quotient on meshes. Corollary 1 (Corollary to Proposition 1 ) . Given a mesh \mathcal{M} with normalized adjacency matrix \mathbf{\tilde{A}}=\mathbf{D^{-1/2}}(\mathcal{W}\odot\mathbf{A})\mathbf{D^{-1/2}} that satisfies 1 , the mesh Rayleigh quotient is invariant under normalized unitary or orthogonal graph convolution, i.e. R_{\mathcal{M}}(\mathbf{X})=R_{\mathcal{M}}(f_{\rm UniMeshConv}(\mathbf{X})) where f_{\rm UniMeshConv} is either separable or Lie. Relaxed Unitary Mesh Convolution. We create a network architecture by coupling the encoder-decoder relaxation ( Section 5.2 ) with Lie unitary mesh convolution in Equation 9 . Concretely, nodes are first zero padded. An encoder E^{(k)} ( Equation 5 ) is constructed from k layers of Lie unitary mesh convolution ( Equation 9 ), and a MLP or GCN decoder layer D maps to the target. We name our relaxed model R-UniMesh . R-UniMesh uses the norm preserving GroupSort activation from 2 to introduce nonlinearity. R-UniMesh also uses orthogonal weights, since our modeling tasks on meshes in Section 6 are real valued. 6 Experiments 6.1 Motivating Experiment: Heat Flow on Grid Graphs In this section, we motivate the use of relaxed unitary models by showing that R-UniGraph is able to outperform both normal and Lie unitary graph convolution on predicting heat diffusion. The Taylor truncation method is key to balancing the smoothness preservation of unitary models with the flexibility to capture the true smoothness of the target heat graph. Heat Diffusion Dataset. We use PyGSP ( 24 ) to simulate heat diffusion on 10,000 two-dimensional grids, each initialized with 20 randomly placed heat sources. Denote by H\colon\mathbb{R}_{+}\to\mathbb{R}^{n} a function that maps time t to the heat distribution of the graph with n nodes. In other words, the heat field on the graph is represented by a feature vector in \mathbb{R}^{n\times 1} . In particular, H(t)=e^{-\tau t\mathbf{L}}H(0) where \tau is a diffusivity constant, \mathbf{L} is the graph Laplacian, and H(0) is the initial heat values across the graph. The task is to predict the heat distribution on the graph at time t=4 given the heat distribution at time t=3 . We denote the target heat field as \mathbf{Y}=H(4) . See Section B.1 for further dataset details. We compare the performance in terms of MSE loss and mean Rayleigh quotient error for three models: GCN, R-UniGraph , and a Lie unitary model. We use \mathbf{T_{\rm max}}=3 for the relaxed model. The mean Rayleigh quotient error is given by \mathrm{MRE}(f)=|\overline{R_{\mathcal{G}}(f(\mathbf{X}))}-\overline{R_{\mathcal{G}}(\mathbf{Y})}|. Results. As shown in Table 1 and Figure 3 , the relaxed model significantly outperforms the GCN and also outperforms the Lie unitary model. Moreover, the relaxed model is best able to produce graphs whose smoothness matches that of the true labels. Our results validate Proposition 7 in 41 (also provided in Section A.2 ), which states that GCNs with standard weight initialization tend to increase the smoothness of input signals on the graph as measured by the Rayleigh quotient. Crucially, our experiments reveal that GCNs not only smooth the input signal, but they oversmooth compared to the target signal, even when some smoothing is desirable, i.e. the output signal is smoother than the input. In contrast, R-UniMesh is often initialized in an undersmooth state and is able to learn to increase the smoothness of input signals during training to match the smoothness of the target. These insights extend to heat flow on more intricate mesh datasets, as we show in Section 6.2 . Model MSE (\downarrow) MRE (\downarrow) GCN 1.08\cdot 10^{-2} 5.99\cdot 10^{-2} Lie Uni 0.14\cdot 10^{-2} 8.86\cdot 10^{-2} R-UniGraph (Ours) \mathbf{0.11\cdot 10^{-2}} \mathbf{2.07\cdot 10^{-2}} Table 1 : MSE and mean Rayleigh quotient error (MRE) of GCN, a Lie unitary convolution network, and R-UniMesh . The best run for each method out of 5 runs is shown. The best performance is bold. Figure 3 : Left: The dotted lines indicate the mean Rayleigh quotient for input heat graphs at T=3 and target graphs at T=4 . We also show the mean Rayleigh quotient for the best performing GCN, R-UniGraph , and Lie unitary models. R-UniGraph is best at capturing the true smoothness. Right: Validation MSE of the same three models. R-UniGraph has the best performance. Results for the full set of runs are provided in Section B.3 . 6.2 Dynamics on Mesh Manifolds We now consider more challenging and realistic tasks and show that R-UniMesh is competitive with strong baselines and outperforms all other models on diffusive dynamics. Specifically, our experiments reveal the following practical conclusions: (i) R-UniMesh performs as well as strong baselines such as mesh-aware transformers and equivariant neural networks; (ii) R-UniMesh is especially strong on heat diffusion tasks; (iii) geometric inductive biases, such as unitarity or equivariance, are important for strong performance on unseen meshes with complex geometries. We support these conclusions with simulated and real-world dynamics datasets on mesh manifolds, including PDE solving on the PyVista ( 74 ) meshes from 61 and weather forecasting on the Earth mesh from WeatherBench 2 (WB 2 ) ( 63 ) . Metric GCN GemCNN R-UniMesh (Ours) EMAN Transformer MPNN EGNN Hermes Heat (\alpha=1) NRMSE (\downarrow) – – \mathbf{51.9\pm 3.6} 73.50\pm 3.8 92.5\pm 5.6 99.45\pm 4.8 344.25\pm 110.5 \underline{73.02\pm 4.7} SMAPE (\downarrow) – 375.4\pm 0.53 \mathbf{79.7\pm 5.6} 110.9\pm 13.3 213.9\pm 2.7 223.6\pm 1.5 319.33\pm 7.59 \underline{107.6\pm 7.4} RE (\downarrow) – 52.21\pm 9.4 \mathbf{9.1\pm 7.4} \underline{14.2\pm 1.4} 46.0\pm 3.7 76.06\pm 3.6 81.5\pm 8.77 39.76\pm 4.7 Wave (c=1) NRMSE (\downarrow) – – \mathbf{236.5\pm 6.4} \underline{281.3\pm 15.5} 864.9\pm 184.9 563.6\pm 7.75 2280.1\pm 559.9 458.5\pm 13.0 SMAPE (\downarrow) – 318.8\pm 3.9 385.2\pm 1.2 \mathbf{301.0\pm 4.2} 327.0\pm 4.4 318.0\pm 2.8 354.3\pm 11.0 \underline{316.4\pm 4.5} RE (\downarrow) – 107.9\pm 3.158 93.5\pm 25.4 73.57\pm 6.5 \mathbf{48.0\pm 7.9} 139.3\pm 10.1 157.2\pm 14.8 \underline{70.03\pm 6.1} Cahn-Hilliard NRMSE (\downarrow) – \mathbf{121.2\pm 1.8} 123.9\pm 2.6 137.5\pm 0.69 144.4\pm 0.8 147.4\pm 11.36 1001.04\pm 5.73 \underline{122.0\pm 7.8} SMAPE (\downarrow) – 204.3\pm 2.4 \underline{167.3\pm 10.6} \mathbf{143.7\pm 2.5} 191.7\pm 2.0 201.22\pm 32.79 336.5\pm 2.777 173.3\pm 4.3 RE (\downarrow) – \mathbf{10.68\pm 3.3} 18.9\pm 10.4 48.57\pm 3.49 27.42\pm 2.87 23.98\pm 6.51 41.8\pm 1.997 \underline{14.38\pm 11.5} Table 2 : NRMSE, SMAPE, and RE averaged over all rollouts on all test meshes for the heat, wave, and Cahn-Hilliard equations. The best values are in bold and second best are underlined. Errors and standard deviations are reported over all test meshes and initializations. Cells with a dash (–) indicate models that do not converge for a given metric. Errors are scaled by \times 196 , the number of rollout timesteps. R-UniMesh is competitive across all tasks and excels at solving the heat equation on unseen meshes. Baselines. We include as baselines standard GNN models without any specific inductive biases for working on meshes, including a GCN ( 42 ) and an MPNN ( 31 ) . Additionally, we study symmetry preserving equivariant models, including gauge and Euclidean equivariance (formally defined in Section A.3 ). Informally, Euclidean equivariant models are invariant to roto-translations of the mesh in Cartesian coordinates and Gauge Equivariant GNNs are invariant to a choice of reference angle for models that work in local coordinates of the mesh-manifold. These models have been shown to be particularly strong for PDE solving on meshes ( 61 ) . We benchmark a state-of-the-art (SOTA) Euclidean equivariant model ( 67 ) as well as different types of Gauge Equivariant GNNs, including convolutional with GemCNN ( 23 ) , attentional with EMAN ( 7 ) , and message passing with Hermes ( 61 ) . We also consider a SOTA mesh transformer ( 38 ) . We compare these baselines with R-UniMesh . Datasets. The first task is to auto-regressively predict the solution to the heat, wave, and Cahn-Hilliard equations on test meshes given an initial condition. These equations are defined formally in Section C.1 . We use the same PyVista meshes generated in 61 . These meshes are highly intricate and test the models’ ability to handle nonlinear dynamics on complicated geometries. We use five different initializations for each test mesh. Further dataset and training details are provided in Section C.1 and Section C.2 . We will refer to this dataset as MeshPDE in the remainder of the paper. The second task is weather forecasting using WB 2 ( 63 ) , a widely used benchmark for data-driven global weather forecasting based on historic data. Specifically, we auto-regressively predict future weather conditions on Earth given an initial condition. A formal problem statement is in Section D.1 . We train and evaluate our models on the ERA5 dataset from WB2, which is the curated version of the ERA5 reanalysis data provided by the European Centre for Medium-Range Weather Forecasts (ECMWF) ( 35 ) . We use 1.5 ( 240\times 120 ) degree spatial resolution data with a 6 hour temporal resolution, consistent with the evaluation performed in WB2. Further details on mesh construction can be found in Section D.3 . We evaluate on two variables, temperature at pressure level 850 (T850) and geopotential at pressure level 500 (Z500). We take data from 2013-01-01 to 2019-12-31 UTC as training data. We use a smaller subset of the ERA5 data that is commonly used for other large scale data-based weather models due to compute constraints, but remain consistent to WB2 in evaluating on data from 2020-01-01 to 2020-12-31. Evaluation. For MeshPDE, our metrics include normalized root mean squared error (NRMSE), symmetric mean absolute percentage error (SMAPE), and Rayleigh quotient errors aggregated over all time-steps. In particular, the Rayleigh error (RE) is given by: \text{RE}(f)=\frac{1}{\mathbf{T_{\rm max}}}\sum_{t}^{\mathbf{T_{\rm max}}}|R_{\mathcal{M}}(\mathbf{Y_{t}})-R_{\mathcal{M}}(f(\mathbf{X_{t}}))|. Further details on these metrics are provided in Section C.3 and we compare RE with more global smoothness metrics in Section C.5 . We supplement these metrics with qualitative diagnostic figures in Section C.4 and videos in the supplementary material , showing in particular that Rayleigh errors are consistent with visual assessments of smoothness. For WB 2 , we report the root mean squared error (RMSE) and the anomaly correlation coefficient (ACC), both latitude weighted as recommended by the benchmark authors. RMSE measures forecast accuracy, while ACC is the Pearson correlation coefficient between forecast anomalies and ground-truth anomalies relative to a climatological baseline. Precise definitions are provided in Appendix D . Figure 4 : The Rayleigh quotient for each timestep on an unseen mesh for Hermes, EMAN, and R-UniMesh models. The R-UniMesh is the best at capturing the true smoothness for heat. Figure 5 : RMSE and ACC as a function of lead time for all models temperature prediction. R-UniMesh has a competitive RMSE, especially at early lead time. R-UniMesh also maintains viability for lead times of roughly 2 days according to the ECMWF baseline. Exact values recorded in Table 7 ( Section D.4 ). Results. Our main result is that our R-UniMesh model outperforms all baselines at solving highly diffusive PDEs and captures the true smoothness while remaining competitive across all other tasks. On MeshPDE, as shown in Table 2 , R-UniMesh performs particularly well on heat modeling, where it achieves the lowest NRMSE, SMAPE, and RE. On heat, R-UniMesh matches the true smoothness almost exactly at every timestep, as seen in Figure 4 , illustrating that R-UniMesh is best at capturing the underlying differential structure of the PDE solution. The convergence and smoothness errors reported by our metrics also agree visually with our qualitative diagnostics in Figure 1 and in Section C.4 . Another important finding is that nearly all models are able to perform comparably well on the Cahn-Hilliard equation, where the test mesh (toroid) is simple. The only exceptions are GCN and EGNN, which struggle across all tasks. This suggests that stronger geometric inductive biases such as unitarity or gauge equivariance are necessary for strong performance on unseen meshes with complex geometries. On the real-world WB 2 benchmark ( Figure 5 ), we see that our best performing models are comparable with the state of the art ( 63 ) in RMSE and ACC, coming within a couple of degrees of SOTA even at 10 lead days. This is despite restricting our training set size and model parameters due to compute limitations. We show in Section D.4 that R-UniMesh is among the best models for the geopotential prediction task, though all models are below the SOTA in 63 . Our results also support our earlier finding that geometric inductive biases matter for complex geometries: the equivariant and unitary models show no significant advantages in this setting, where there is no cross mesh generalization. Finally, we show in Section D.5 that R-UniMesh is competitive in terms of RE on both WB 2 variables. 7 Conclusion Our work clarifies the approximation limits of smoothness preserving (unitary) functions and unitary convolution networks and shows how constraint relaxations can aid performance on various dynamics modeling tasks on graphs and meshes. We contribute R-UniGraph and R-UniMesh , which provide SOTA performance on diffusive dynamics problems and excel at capturing the true smoothness of the system. Future work will explore using approximately unitary networks for solving PDEs under partial observability by using them as backbones for generative models. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendix A Deferred Theory This section provides both theoretical background and deferred proofs from the main text. A.1 Lie Algebras and the Exponential Map In this section we review the formalism behind Lie algebras and the exponential map. A group is a mathematical structure that formalizes what it means for something to be symmetric . We say that a group is a matrix Lie group , if it is a differentiable manifold and a subgroup of the set of invertible n\times n matrices. Lie groups are equipped with a Lie algebra , which is the tangent space at the identity element. Our work encounters the orthogonal and unitary Lie groups \mathrm{O}(n)=\{O\in\mathbb{R}^{n\times n}\colon OO^{T}=I\},\qquad\mathrm{U}(n)=\{U\in\mathbb{C}^{n\times n}\colon UU^{\dagger}=I\} as well as the special unitary group \mathrm{SU}(n)=\{U\in\mathbb{C}^{n\times n}\colon\det(U)=1\}. The associated Lie algebras for O(n) and U(n) are given by \mathfrak{o}(n)=\{M\in\mathbb{R}^{n\times n}\colon M+M^{T}=0\},\qquad\mathfrak{u}(n)=\{M\in\mathbb{C}^{n\times n}\colon M+M^{\dagger}=0\}. The exponential map provides a mechanism of parameterizing Lie groups with elements in the Lie algebra. For matrix Lie groups, the exponential map is simply the matrix exponential: \exp(\mathbf{X})=\overset{\infty}{\underset{i}{\sum}}\frac{1}{i!}\mathbf{X}^{i}. Applying the exponential map to a linear operator is given by \exp(\mathbf{L})(\mathbf{X})=\overset{\infty}{\underset{i}{\sum}}\frac{1}{i!}\mathbf{L}^{i}(\mathbf{X})=\mathbf{X}+\mathbf{L}(\mathbf{X})+\frac{1}{2}\mathbf{L}\circ\mathbf{L}(\mathbf{X})+\frac{1}{6}\mathbf{L}\circ\mathbf{L}\circ\mathbf{L}(\mathbf{X})+\dots In the case of Equation 3 , \mathbf{L} is graph convolution, \mathbf{L}(\mathbf{X)}=\mathbf{AXW} . Further background on group theory and abstract algebra can be found in 4 , 34 , and 27 . A.2 Convolutional oversmoothing This section provides a result from 41 which establishes that Graph Convolution Networks ( 42 ) have a high probability to exhibit smoothing. Proposition 3 (Proposition 7 in 41 ) . Given a simple undirected graph \mathcal{G} on n nodes with normalized adjacency matrix \widetilde{\mathbf{A}}=\mathbf{D}^{-1/2}\mathbf{A}\mathbf{D}^{-1/2} and node degree bounded by D , let \mathbf{X}\in\mathbb{R}^{n\times d} have rows drawn i.i.d. from the uniform distribution on the hypersphere in dimension d . Let f_{conv}(\mathbf{X})=\widetilde{\mathbf{A}}\mathbf{X}\mathbf{W} denote convolution with orthogonal feature transformation matrix \mathbf{W}\in O(d) . Then, the event below holds with probability 1-\exp(-\Omega(\sqrt{n})) : R_{\mathcal{G}}(\mathbf{X})\geq 1-O\left(\frac{1}{n^{1/4}}\right)\quad\text{and}\quad R_{\mathcal{G}}(f_{conv}(\mathbf{X}))\leq 1-\frac{\Tr(\widetilde{\mathbf{A}}^{3})}{\Tr(\widetilde{\mathbf{A}}^{2})}+O\left(\frac{1}{n^{1/4}}\right). A.3 Gauge and Euclidean Equivariance In this section, we introduce the necessary background and formal definitions for the equivariance constraints commonly applied to tasks defined on meshes. While working with arbitrary meshes, many commonly used network architectures compute distances between node positions. One has the option of computing these distances in either global Cartesian coordinates or in local tangent spaces of the mesh. In both cases, we may exploit the symmetry of these coordinate systems by enforcing equivariance with respect to transformations from a certain symmetry group into the network architecture, which allows the network to automatically generalize across orbits. We now give precise definitions of equivariance and invariance. Definition 4 . Let f:\mathcal{X}\rightarrow\mathcal{Y} be a map between input and output vector spaces \mathcal{X} and \mathcal{Y} . Let G be a group with representations \rho^{\mathcal{X}} and \rho^{\mathcal{Y}} which transform vectors in \mathcal{X} and \mathcal{Y} respectively. Representations are group homomorphisms which map group elements to invertible linear transformations. The map f:\mathcal{X}\rightarrow\mathcal{Y} is equivariant if \displaystyle\rho^{\mathcal{Y}}(g)f(x)=f(\rho^{\mathcal{X}}(g)x)\ ,\ \text{for all }g\in G,x\in\mathcal{X}\ . Invariance is a special case of equivariance in which \rho^{\mathcal{Y}}=\rm{Id}^{\mathcal{Y}} for all g\in G . With an invariant operator, the output of f is unaffected by the transformations applied to the input. Definition 5 . A map f:\mathcal{X}\rightarrow\mathcal{Y} is invariant if \displaystyle f(x)=f(\rho^{\mathcal{X}}(g)x)\ ,\ \text{for all }g\in G,x\in\mathcal{X}. A.3.1 Euclidean Equivariance For a mesh defined over a global coordinate system, a common choice of symmetry constraint is equivariance to the Euclidean group in n dimensions, E(n) . In this setting, the mesh is treated as a graph with positional encodings, and the equivariance constraint ensures generalization to different roto-translations of the mesh. Definition 6 . Let t\in\mathbb{R}^{n} be a translation vector and Q\in\mathbb{R}^{n\times n} an orthogonal matrix representing a rotation or reflection. A function f is equivariant to the Euclidean group E(n) if for any t\in\mathbb{R}^{n} and Q\in\mathbb{R}^{n\times n} we have f(Qx+t)=Qf(x)+t. A.3.2 Gauge Equivariance We may also choose to embed coordinates locally, using coordinates that are intrinsic to the 2D mesh rather than the extrinsic 3D coordinates of the embedding space. This approach arises from the desire for a general convolution-like operator over arbitrary manifolds discretized as a mesh. To encode data over a mesh it is still necessary to make a choice of local coordinate frame at each vertex. In order to guarantee the equivalence of the features resulting from different choices of reference frames, the model should be invariant to change of coordinates frame at each vertex, i.e. gauge equivariant. We specifically adapt the strategy described in 23 and define the local coordinate frame at each vertex in terms of a reference neighboring vertex. Denote v_{a} as the reference neighbor for gauge A , in which the neighbors have angles \theta_{A} , and denote v_{b} as the reference neighbor for gauge B with angles \theta_{B} . Comparing the two gauges, we see that they are related by a rotation of angle \phi , so that \theta_{B}=\theta_{A}-\phi . This change of gauge is called a gauge transformation of angle g:=\phi . Definition 7 (Equations 3 and 4 in 23 ) . Let \rho_{\text{in}} and \rho_{\text{out}} be input and output types with dimensions C_{\text{in}} and C_{\text{out}} . Let K_{\text{self}}\in\mathbb{R^{C_{\text{out}}\times C_{\text{in}}}} and K_{\text{neigh}}\colon[0,2\pi)\rightarrow\mathbb{R}^{C_{\text{out}}\times C_{\textit{in}}} be two kernels. We say the kernels are gauge equivariant if for any gauge transformation g\in[0,2\pi) and for any angle \theta\in[0,2\pi) we have K_{\text{neigh}}(\theta-g)=\rho_{\text{out}}(-g)K_{\text{neigh}}(\theta)\rho_{\text{in}}(g),\qquad K_{\text{self}}=\rho_{\text{out}}(-g)K_{\text{self}}\,\rho_{\text{in}}(g). Finally, as features at different nodes live in different tangent spaces and thus have different gauges, it is invalid to sum them directly. Let f_{p} and f_{q} be node features of a pair of neighboring nodes p and q . Before performing gauge equivariant convolution, we must parallel transport each f_{q} to T_{p}M along the mesh edge that connects the two vertices for them to be in the same gauge. For more details, we refer the reader to 23 . A.4 Unitary Learning Framework This section provides rigorous definitions for the mathematical tools used in the main text and additionally clarifies necessary hypotheses. We start with the fundamental domain. Assume X has dimension n . Let d be the dimension of a generic orbit of G in X . Let \nu be the (n-d) dimensional Hausdorff measure in X . Definition 8 (Fundamental Domain, Definition 4.1 in 77 ) . A closed subset F of X is called a fundamental domain of G in X if X is the union of conjugates of F , i.e., X=\cup_{g\in G}gF , and the intersection of any two conjugates has measure 0 under \nu . Next, we note that our proof of Theorem 1 satisfies the integrability assumption on the fundamental domain F and orbits Gz established in 77 : Assumption 2 (Integrability Hypothesis, Sec. A in 77 ) . The fundamental domain F and orbit Gx are differentiable manifolds and the union of all pairwise intersections \cap_{g_{1}\neq g_{2}}(g_{1}F\cap g_{2}F) has measure zero. We now provide more formal definitions for \mathbb{E}_{Gx}[f] and \mathbb{V}_{Gx}[f] used in Proposition 2 . Denote by q(z)=\frac{p(z)}{p(Gx)} the density of the orbit Gx so that \int_{Gx}q(z)dz=1 . The mean and variance of a function f on Gx are given by \mathbb{E}_{Gx}[f]=\int_{Gx}q(z)f(z)dz,\qquad\mathbb{V}_{Gx}[f]=\int_{Gx}q(z)\lVert\mathbb{E}_{Gx}[f]-f(z)\rVert_{2}^{2}dz. A.5 Proof of Main Theorem We now provide proof of our main theoretical result in the main text. We repeat the theorem here for convenience. Theorem 1 . Let F be a fundamental domain of \mathrm{SU}(n) in Z . In particular, F=\{te\colon t\in\mathbb{R}_{+}\} where e is a standard basis vector of \mathbb{C}^{n} . The approximation error lower bound can be expressed as \int_{Z}p(z)\lVert u(z)-f(z)\rVert_{2}^{2}dz\geq\int_{F}p(\lVert te\rVert)\mathbb{V}_{Gz}[\rVert f\lVert]dz. Proof of Theorem 1 . By the reverse triangle inequality, \displaystyle\int_{Z}p(z)\lVert u(z)-f(z)\rVert_{2}^{2}dz \displaystyle\geq\int_{Z}p(z)\left(\lVert u(z)\rVert-\lVert f(z)\rVert\right)^{2}dz. Notice that \lVert u(z)\rVert is invariant under the action of \mathrm{SU}(n) on the sphere S^{2n-1} with radius \lVert te\rVert and recall that \mathrm{SU}(n) acts transitively on the sphere. Thus, F=\{te\colon t\in\mathbb{R}_{+}\} is a valid fundamental domain that indexes each orbit Gz , the spheres with radii \lVert te\rVert . Our theorem then follows from Proposition 2 . ∎ In the following example, we show how this bound may be computed. Example 1 (Variance on the Unit Disk) . Denote by D_{2} the unit disk D_{2}=\{(x,y)\colon x^{2}+y^{2}\leq 1\} . Let Z=\mathbb{R}^{2} be a domain with density p=\begin{cases}\frac{1}{\pi},&(x,y)\in D_{2}\\ 0,&\text{Otherwise}.\end{cases} Denote by f a function in polar coordinates given by \displaystyle f\colon(\theta,r) \displaystyle\to\mathbb{R}^{2} \displaystyle\theta \displaystyle\mapsto(\sin\theta+r,\cos\theta+r). On each orbit, we compute \mathbb{E}_{G_{z}}[f]=(r,r),\qquad\mathbb{V}_{G_{z}}[f]=1,\qquad p(r)=\begin{cases}2r,&r\leq 1\\ 0,&\text{otherwise}.\end{cases} The approximation error bound of a unitary function u of f is then \displaystyle\int_{Z}p(z)\lVert u(z)-f(z)\rVert_{2}^{2}dz\geq\int_{F}p(r)\mathbb{V}_{G_{z}}[f]dr \displaystyle=\int_{0}^{1}2r(1)dr=1. A.6 Unitary Convolution on Meshes In this section, we prove Corollary 1 stated in Section 5.3 and repeated here for convenience. Corollary 1 (Corollary to Proposition 1 ) . Given a mesh \mathcal{M} with normalized adjacency matrix \mathbf{\tilde{A}}=\mathbf{D^{-1/2}}(\mathcal{W}\odot\mathbf{A})\mathbf{D^{-1/2}} that satisfies 1 , the mesh Rayleigh quotient is invariant under normalized unitary or orthogonal graph convolution, i.e. R_{\mathcal{M}}(\mathbf{X})=R_{\mathcal{M}}(f_{\rm UniMeshConv}(\mathbf{X})) where f_{\rm UniMeshConv} is either separable or Lie. Our proof follows the same structure as the proof of Proposition 1 in 41 with modifications to account for the weighted adjacency matrix. Namely, we invoke 1 which ensures that f_{\rm UniMeshConv} is norm preserving and therefore the the strategy in 41 still holds. Proof. We first prove invariance for Equation 8 . By the circulant property of the trace, \Tr\left(\left(\exp(i\tilde{\mathbf{A}})\mathbf{XU}\right)^{\dagger}\left(\mathbf{I-\tilde{A}}\right)\left(\exp(i\tilde{\mathbf{A}})\mathbf{XU}\right)\right)=\Tr\left(\mathbf{X}^{\dagger}\exp(-i\tilde{\mathbf{A}})\left(\mathbf{I-\tilde{A}}\right)\exp(i\tilde{\mathbf{A}})\mathbf{X}\right). Because \exp(-i\mathbf{\tilde{A}}) , \exp(i\mathbf{\tilde{A}}) , and (\mathbf{I-\tilde{A}}) share an eigenbasis, they commute, so \Tr\left(\left(\exp(i\tilde{\mathbf{A}})\mathbf{XU}\right)^{\dagger}\left(\mathbf{I-\tilde{A}}\right)\left(\exp(i\tilde{\mathbf{A}})\mathbf{XU}\right)\right)=\Tr\left(\mathbf{X}^{\dagger}\left(\mathbf{I-\tilde{A}}\right)\mathbf{X}\right). For the denominator, we need to show that \lVert\exp(i\mathbf{\tilde{A}})\mathbf{XU}\rVert_{F}^{2}=\lVert\mathbf{X}\rVert_{F}^{2} . By 1 we have that \mathcal{W} is symmetric. Because \mathbf{A} is also symmetric, we have that i\tilde{\mathbf{A}} is skew-symmetric and therefore \exp(i\tilde{\mathbf{A}})\in SU(n) . Thus, \lVert\exp(i\mathbf{\tilde{A}})\mathbf{XU}\rVert_{F}^{2}=\lVert\mathbf{X}\rVert_{F}^{2} and finally R_{\mathcal{M}}(\mathbf{X})=R_{\mathcal{M}}(f_{\rm UniMeshConv}(\mathbf{X})) . We now show that Equation 9 also preserves the Rayleigh quotient. First, we need to show that \lVert\exp(\mathbf{AXW)}\rVert_{F}^{2}=\lVert\mathbf{X}\rVert_{F}^{2} . To do this, we note that Equation 9 can equivalently be viewed us a function that acts on a vector in \mathbb{C}^{nd} . By properties of the Kroneckor tensor product, f_{\rm UniMeshConv}(\mathbf{X;A})=\exp(\mathbf{AXW})\iff\text{vec}\left(f_{\rm UniMeshConv}(\mathbf{X;A})\right)=\exp\left(\mathbf{A\otimes W^{T}}\right)\text{vec}(\mathbf{X}). Since \left(\mathbf{A\otimes W^{T}}\right)+\left(\mathbf{A\otimes W^{T}}\right)^{\dagger}=\mathbf{A}\otimes\left(\mathbf{W+W^{\dagger}}\right)^{T}=0, we have that \left(\mathbf{A\otimes W^{T}}\right) is in the lie algebra of the unitary group and therefore preserves the norm of \text{vec}(\mathbf{X}) . This holds for any symmetric edge weighting \tilde{\mathbf{A}}=\mathcal{W}\odot\mathbf{A} , which is guaranteed by 1 . Thus, \lVert\exp(\mathbf{AXW)}\rVert_{F}^{2}=\lVert\mathbf{X}\rVert_{F}^{2} . Next, note that \exp\left(\mathbf{\tilde{A}\otimes W^{T}}\right) commutes with (\mathbf{\tilde{A}\otimes I}) . Thus, \displaystyle\Tr \displaystyle\left(f_{\rm UniMeshConv}(\mathbf{X;\tilde{A}})^{\dagger}(\mathbf{I-\tilde{A}})f_{\rm UniMeshConv}(\mathbf{X;\tilde{A}})\right) \displaystyle=\text{vec}(\mathbf{X})^{\dagger}\exp(\mathbf{\tilde{A}\otimes W^{T}})^{\dagger}\left[(\mathbf{I-\tilde{A}})\otimes\mathbf{I}\right]\exp(\mathbf{\tilde{A}\otimes W^{T}})\text{vec}(\mathbf{X}) \displaystyle=\text{vec}(\mathbf{X})^{\dagger}\left[(\mathbf{I-\tilde{A}})\otimes\mathbf{I}\right]\text{vec}(\mathbf{X}). Multipliying the above by \lVert\mathbf{X}\rVert_{F}^{-2} recovers R_{\mathcal{M}}(\mathbf{X}) . We conclude that R_{\mathcal{M}}(\mathbf{X})=R_{\mathcal{M}}(f(\mathbf{X})) . ∎ Remark 2 . Corollary 1 was applied to convolution with the symmetric cotangent weights in Equation 7 , but the proof extends without loss of generality to any set of symmetric weights. A.7 Discrete Differential Geometry We provide further details and visualizations for concepts in discrete differential geometry, including the mesh manifold condition, the cotangent Laplacian, and Delaunay criterion. We also review results from the literature that suggests that the mesh edge rewiring algorithm used by the Robust Laplacian is a safe choice for the task of PDE solving. For additional reference on these topics, see 52 and 17 . The Manifold Condition Our work makes use of the assumption that a given mesh is manifold. We recall the following definition of the mesh manifold condition: Definition 9 (Manifold Condition, ( 71 ) ) . An interior (or boundary) edge ij is manifold if it is contained in exactly two (or one) triangles; an interior (or boundary) vertex i is manifold if the boundary of all triangles incident on i forms a single loop (or path) of edges. Alternative definitions for the manifold condition often state that a mesh is manifold if interior vertices have local neighborhoods that are homeomorphic to the unit disk and boundary vertices are homeomorphic to the half disk. We refer specifically to Definition 9 when we talk about a mesh being manifold in the paper. Cotangent Laplacian Area Normalization. Recall that for a scalar function s on mesh vertices we the cotangent Laplacian is defined as (\mathbf{\tilde{L}}(s))_{i}=\frac{1}{2A_{i}}\underset{j\in\mathcal{N}(i)}{\sum}\left(\cot\alpha_{ij}+\cot\beta_{ij}\right)(s_{j}-s_{i}) where \mathcal{N}(i) denotes the adjacent vertices of i , \alpha_{ij} and \beta_{ij} are the angles opposite edge (i,j) , and A_{i} is the vertex area of i and that we use the barycentric cell area for A_{i} . In particular, let \mathcal{A}_{abc} be the area of a triangular face with vertices abc and let \mathcal{F}(i) be the set of faces containing vertex i . The barycentric cell area is defined A_{i}=\underset{abc\in\mathcal{F}(i)}{\sum}\mathcal{A}_{abc}/3. Normalization by the cell area was used for the dataset construction in 61 , but it is not used in the definition of the Robust Laplacian ( 71 ) . In fact, the cell area introduces asymmetry in the edge weights. This is undesirable, as unitary mesh convolution depends on symmetric edge weights in order to preserve the Rayleigh quotient. Cotangent Laplacian Edge Weights and Robust Rewiring. As noted in Section 5.3 , an arbitrary mesh may have negative cotangent weights. These cotangent weights have the following geometric meaning. For vertices ij connected by an edge, we say that the edge is primal . For manifold meshes, for each primal edge there exists a dual edge that connects the triangle circumcenters for the two triangles that share a primal edge. The cotangent weights correspond to the ratio of the primal and dual edge lengths for vertices ij ( 17 ) . The weights are positive when the angles \alpha_{ij}+\beta_{ij}\leq\pi . The Robust Laplacian applies two edge rewiring algorithms sequentially, the tufted cover algorithm and the Delaunay edge flip algorithm. The tufted cover algorithm ensures the mesh is manifold so that the Delaunay edge flip algorithm can be applied. The Delaunay edge flip algorithm then ensures that the mesh satisfies the intrinsic Delaunay criterion ( Definition 2 ). A sample edge flip is illustrated in Figure 6 . Figure 6 : Illustration of an intrinsic Delaunay edge flip performed by the Robust Laplacian edge rewiring algorithm. This figure is a reproduced version of Figure 7, 71 . 71 note that from a finite element perspective, changing the triangulation via Delaunay edge-flipping effectively just provides a different set of linear basis functions for the same polyhedral domain. Thus, the only practical concern is whether the tufted cover algorithm, which ensures that the mesh is manifold, does not dramatically change the connectivity. Empirical results from 71 find that the Robust Laplacian greatly improves performance computing geodesic distances with the heat method ( 18 ) , which depends on numerical PDE solving on highly nonmanifold meshes with the cotangent Laplacian. This provides confidence that the tufted cover algorithm improves the edge-weighting scheme for unitary mesh convolution. See 18 for additional reference on numerical PDE solving on nonmanifold meshes. A.8 Rayleigh Quotient Sensitivity We include results from 28 and 25 that illustrate the sensitivity of the Rayleigh quotient to small perturbations of the input, such as Taylor series truncation errors. While the hypotheses are stronger than what we may actually see in practice, the following proposition provides an intuition for the Rayleigh quotient sensitivity. Proposition 4 (Proposition 4 in 28 ) . Suppose \mathbf{u=x+e} is an approximate eigenvector corresponding to a simple eigenvalue \lambda\neq 0 of a symmetric A , with \lVert\mathbf{x}\rVert=1 , \mathbf{e\perp x} , and \varepsilon=\lVert\mathbf{e}\rVert . Then, up to \mathcal{O}(\varepsilon^{4}) -terms, for the sensitivity of the Rayleigh quotient (as a function of \mathbf{u} ) it holds that \underset{\lambda_{i}\neq\lambda}{\min}\frac{|\lambda_{i}-\lambda|}{|\lambda_{i}|}\varepsilon^{2}\lesssim\frac{|R_{\mathcal{G}}(\mathbf{u})-\lambda|}{|\lambda|}\lesssim\underset{\lambda_{i}\neq\lambda}{\max}\frac{|\lambda_{i}-\lambda|}{|\lambda_{i}|}\varepsilon^{2}. This indicates that the Rayleigh quotient sensitivity is quadratic in perturbations \varepsilon . For \varepsilon<1 , this means that the sensitivity of the Rayleigh quotient is even less than the truncation error. We also have the following results from 25 : Proposition 5 (Theorem 1 in 25 ) . For any given graph G , if there exists a perturbation \Delta on \mathbf{L} , the change of Rayleigh quotient can be bounded by \lVert\Delta\rVert_{2} . Proposition 6 (Theorem 2 in 25 ) . For any given graph G , if there exists a perturbation \delta on \mathbf{x} , the change of Rayleigh quotient can be bounded by 2\mathbf{x}^{T}\mathbf{L}\delta+o(\delta) . If \delta is small enough, in which case o(\delta) can be ignored, the change can be further bounded by 2\mathbf{x}^{T}\mathbf{L}\delta . The results from 25 state fewer hypotheses than 28 . Proposition 5 outlines a bound similar to Proposition 4 in that they are both related to the norm of the perturbing vector, and Proposition 6 states an alternative bound related to the graph’s Laplacian. Appendix B Simulated Heat Diffusion Further Details B.1 Simulated Heat Diffusion Dataset This section details dataset generation specifications for our experiment in Section 6.1 . We generate grid-graphs with an average of 10 nodes and a standard deviation of 2 nodes. On the grid we randomly set 20 nodes to be heat sources. They are given a heat value of 1 and all other nodes start at 0 . Using PyGSP , We simulate heat flow on 10,000 graphs for training, and the task is to predict the next time step given the previous one. The simulation proceeds until time T=10 in increments of \Delta T=0.5 time steps. A sample graph data point is given in Figure 7 . Figure 7 : Sample heat diffusion process on a grid discretized as a graph. Node neighbors are the nodes that sit adjacent in the grid. B.2 Taylor Series Sensitivity Analysis We conduct a sensitivity analysis of the Rayleigh quotient to different Taylor series truncations. For completeness, we also compare with standard GCNs and Separable unitary networks. We study these tendencies at initialization for the heat diffusion dataset that is used for the experiment in Section 6.1 , described further in Section B.1 . Our analysis echos a theme similar to 32 and 29 that practitioners should be more thorough in evaluating when numerical approximations break strict theoretical guarantees. Experimental Setup. We simulate heat diffusion on a grid graph and use time step 3 to conduct the sensitivity analysis. We evaluate on the models f_{\rm GCN} , f_{\rm SepUniConv} , and f_{\rm LieUniConv} . For each model f and truncation length \mathbf{T_{\rm max}}\in\{1,\dots,10\} , we compute the Rayleigh quotients R_{\mathcal{G}}(\mathbf{X}) and R_{\mathcal{G}}(f(\mathbf{X})) for all graph mini batches \mathbf{X} . We denote the distribution of Rayleigh quotients before applying the model by P_{\mathbf{X}} and after applying the model by P_{f(\mathbf{X})} . To quantify the deviation between these distributions, we compute the KL divergence D_{\text{KL}}\left(P_{\mathbf{X}}\parallel P_{f(\mathbf{X})}\right) , which measures the change in the distribution of Rayleigh quotients caused by the model at initialization. Results. We see in Figure 8 the effect of Taylor series truncation on the unitarity of the network. In particular, we observe that the KL divergence between the two distributions decreases exponentially with the number of terms. This is to be expected, we know from Taylor’s theorem that a truncation at term t gives truncation error \mathcal{O}\left(\frac{\left[\lVert\mathbf{AXW}\rVert_{\mathbf{O}}\right]^{t+1}\lVert\mathbf{X}\rVert_{2}}{(t+1)!}\right) where \mathbf{\lVert\cdot\rVert}_{\mathbf{O}} is the operator norm. Furthermore, works such as 28 and 25 show theoretically that small truncation errors will not compound into large deviations in the Rayleigh quotient. For details on the relevant propositions from 28 and 25 , see Section A.8 . Figure 8 : KL divergence between distribution of Rayleigh quotients before and after applying the model. Results are averaged over 10 runs. In the supplementary material we include a video that shows the evolving Rayleigh quotient distribution as we increase \mathbf{T_{\rm max}} . Figure 8 is also clickable and links to the same video in our anonymous artifact. B.3 Heat Diffusion Model Ensembles This section provides the results for Section 6.1 for a larger ensemble of models. In Figure 9 , we see that the training runs in Figure 3 occur frequently, with only a couple runs diverging. Figure 9 : Top: Validation MSE for an ensemble of 5 runs for a GCN (left), R-UniGraph (middle), and a Lie unitary convolution network (right) at timestep t=3 . R-UniGraph significantly outperforms the GCN and also outperforms the Lie unitary network. Bottom: The average Rayleigh quotient over all graphs for an ensemble of 5 runs for the same models at timestep t=3 . The GCN is under constrained and biased towards oversmoothing at initialization. R-UniGraph is able to roughly match the true smoothness of the labeled graphs. The Lie unitary network is overconstrained and can not model the Rayleigh quotient of the labels because it is forced to preserve the Rayleigh quotient of the input graphs. Appendix C PyVista Mesh Training Details, Evaluation Details, and Further Experiments This section provides extra experimental details and results for our dynamical systems modeling on PyVista meshes. C.1 PyVista Mesh Dynamical Systems This section provides details on the PDEs to be solved on the PyVista meshes. Before defining the PDEs to be solved, let us establish notation. Let \alpha be the thermal diffusivity, and c a constant. The heat and wave equations on the mesh are then given by \displaystyle\frac{\partial u}{\partial t}=\alpha\mathbf{\tilde{L}}u,\quad\text{(Heat)} \displaystyle\frac{\partial^{2}u}{\partial t^{2}}=c^{2}\mathbf{\tilde{L}}u,\quad\text{(Wave)}. We now define the Cahn-Hilliard equation ( 11 ) . Let c be the fluid concentration, M the diffusion coefficient, \mu the chemical potential, f the double-free energy function, and \lambda a positive constant. The Cahn-Hilliard equation is often represented by the following two coupled second order equations: \frac{\partial c}{\partial t}-M\mathbf{\tilde{L}}(\mu)=0,\quad\mu-\frac{\partial f}{\partial c}+\lambda\mathbf{\tilde{L}}(c)=0. Here, the double-free energy function is given by f(c)=100c^{2}(1-c^{2}) . Sample initial conditions for each equation on the PyVista meshes are shown in Table 3 and Table 4 . Mesh Heat T=0 Wave T=0 Armadillo Bunny Lucy Sphere Spider Urn Woman Table 3 : Sample initializations for the heat and wave equations on the PyVista meshes. Mesh Cahn-Hilliard T=0 Bunny Ellipsoid Sphere Super Toroid Table 4 : Sample initializations for the Cahn-Hilliard equation on the PyVista meshes. C.2 PyVista Mesh Training Details We extend the publicly available code base from 61 to train our baselines for the PyVista and WeatherBench 2 datasets: https://github.com/jypark0/hermes/ . For GemCNN, EMAN, and Hermes, we use the already available pretrained model checkpoints. For GCN, R-UniMesh , Mesh Transformer, MPNN, and EGNN we train our own models. We use the same train / test split for the meshes as in 61 . We performed ablations over learning rate and latent space sizes. Following 61 we keep models within a \sim 40,000-50,000 parameter budget. We note that this budget is relatively small, and that models that diverge in our experiments could potentially perform better under a more forgiving budget. All runs were performed on a single H 200 GPU ( 58 ) . We use the previous 5 time steps as input node feature vectors and backpropagate through 3 steps of auto-regressive inference. Hyperparameters are given in our artifact . Defaults are taken from 61 if provided and otherwise optimized via grid search. Considered hyperparameters include learning rate, optimizer, training epochs, latent size, and skip connections. We also consider z-scoring of normed edge lengths for EGNN, different decoder heads for R-UniMesh , and the number of clusters for the mesh transformer. As observed in 61 , we notice that residual connections can be key for performance with the gauge equivariant models. For R-UniMesh , we found that using a MLP readout with sinusoidal activation functions was a key ingredient for strong performance on MeshPDE. This supports previous work on how to train GNNs for long range tasks ( 75 ; 76 ) . However, the GCN decoder exhibited the best performance on WB 2 . C.3 PyVista Evaluation Details In order to aggregate smoothness errors over all time steps, we introduce a new metric. Define the Rayleigh Error (RE) by \int_{0}^{\infty}|R_{\mathcal{M}}(\mathbf{Y_{t}})-R_{\mathcal{M}}(f(\mathbf{X_{t}}))|dt . In practice we approximate this by summing over the time steps where we are able to perform inference and normalize to the max timestep: \text{RE}(f)=\frac{1}{\mathbf{T_{\rm max}}}\sum_{t}^{\mathbf{T_{\rm max}}}|R_{\mathcal{M}}(\mathbf{Y_{t}})-R_{\mathcal{M}}(f(\mathbf{X_{t}}))|. Following 38 and 60 , we also consider the scale invariant metrics NRMSE and SMAPE averaged over the entire rollout: \displaystyle\text{NRMSE}(f) \displaystyle=\frac{1}{\mathbf{T_{\rm max}}}\sum_{t}^{\mathbf{T_{\rm max}}}\sqrt{\frac{\frac{1}{n}\sum_{i=1}^{n}\left(f(\mathbf{X_{t}})_{i}-(\mathbf{Y_{t}})_{i}\right)^{2}}{\frac{1}{n}\sum_{i=1}^{n}(\mathbf{Y_{t}})_{i}^{2}}} \displaystyle\text{SMAPE}(f) \displaystyle=\frac{1}{\mathbf{T_{\rm max}}}\sum_{t}^{\mathbf{T_{\rm max}}}\frac{1}{n}\overset{n}{\underset{i=1}{\sum}}\frac{2|(\mathbf{Y_{t}})_{i}-f(\mathbf{X_{t}})_{i}|}{|(\mathbf{Y_{t}})_{i}|+|f(\mathbf{X_{t}})_{i}|+\varepsilon} where \varepsilon=10^{-8} is a stability constant. SMAPE is generally more robust than NMRSE in that it is less sensitive to outliers, but it is also more sensitive to small values. The scale invariant property of these metrics is crucial especially for heat diffusivity because solutions tend to decrease proportionally to e^{-t} . Thus, we need to consider deviations across several orders of magnitude in order to see how accurately we are modeling the decay. C.4 PyVista Mesh Qualitative Diagnostics In this section, we validate the superior performance of R-UniMesh on solving the heat equation with qualitative diagnostics. In Table 5 we show that R-UniMesh is the best at capturing the true smoothness of an unseen mesh during each step of the rollout. In the supplementary material we include a video corresponding to the rollout in Table 5 over all timesteps. Time Truth R-UniMesh (Ours) EMAN Hermes 10 50 100 150 190 Table 5 : Qualitative comparison of model performance for the heat equation on the armadillo mesh. Our R-UniMesh model remains faithful to the ground truth during each step of the rollout, whereas the EMAN model over smooths and the Hermes model under smooths. C.5 Beyond \mathbf{1-} hop Smoothness Since the Rayleigh quotient is a 1- hop metric, this section performs additional comparisons with a more global smoothness metric and finds that our 1- hop smoothness tendencies also hold more generally for the gauge equivariant models we study. In particular, we define smoothness according to the 2- point correlation function. Let \delta:\mathbb{R}^{3}\to\mathbb{R} be a function that maps a point \mathbf{x} on a mesh to the scalar solution u(\mathbf{x}) (or approximation thereof) to the PDE at that point. The smoothness is then defined by the 2- point correlation function \xi given in Equation 10 : \xi(r;\delta)=\mathbb{E}\left[\delta(\mathbf{x})\delta(\mathbf{x}+r)\right]. (10) Intuitively, if node features are similar at a distance of r apart, the correlation will be high. This allows us to study smoothness beyond 1- hop neighbors by considering larger r . Figure 10 shows an example correlation function for Hermes at a given time step. We note that this characterization of smoothness is common in the weak gravitational lensing literature for point-cloud datasets ( 69 ) and are easily computed with the TreeCorr library ( 39 ) . Let \delta_{ij} be the scalar field for the ground truth on a mesh \mathcal{M}_{i} at time step j and \widehat{\delta_{ij}} be the approximation thereof. We define our smoothness error by \text{err}_{\rm smooth}(\widehat{\delta_{ij}})=\frac{1}{r_{\rm bins}}\frac{1}{\mathbf{T_{\rm max}}}\frac{1}{N}\underset{i=1}{\overset{N}{{\sum}}}\overset{\mathbf{T_{\rm max}}}{\underset{j=1}{\sum}}\overset{r_{\rm bins}}{{\underset{k=1}{\sum}}}|\xi(r_{k};\delta_{ij})-\xi(r_{k};\widehat{\delta_{ij}})|. We note that the correlation function is related to the Fourier space power spectrum P(k) by \xi(r)=\frac{1}{2\pi^{2}}\int k^{2}P(k)\frac{\sin(kr)}{kr}dk. (11) Thus, Equation 11 informs us that our metric for smoothness as a function of r is related to traditional energy spectrum errors ( 79 , e.g.,) . We leave a more systematic comparison between the measures as an opportunity for future work. As seen in Table 6 , the more expressive attention and message passing based models are much better at capturing the underlying smoothness. The CNN model diverges for the heat and wave datasets, but performs reasonably well on Cahn-Hilliard. This is mirrored by our results in Table 2 in the main text for the Gauge Equivariant models. Figure 10 : Smoothness for a Hermes model as measured by the 2- point correlation function. The plot indicates undersmoothing in each radial bin. Heat ( \alpha=1 ) Model \text{err}_{\rm smooth} (\downarrow) GemCNN – EMAN \mathbf{4.04\times 10^{-3}} Hermes 9.71\times 10^{-3} Wave ( c=1 ) Model \text{err}_{\rm smooth} (\downarrow) GemCNN – EMAN \mathbf{1.78\times 10^{-3}} Hermes 1.38\times 10^{-2} Cahn–Hilliard Model \text{err}_{\rm smooth} (\downarrow) GemCNN 1.89\times 10^{-1} EMAN 4.59\times 10^{-1} Hermes \mathbf{9.61\times 10^{-3}} Table 6 : \text{err}_{\rm smooth} for Gauge Equivariant models on the PyVista Mesh datasets. Dashes (–) indicate non-convergence. Best performing model is indicated with bold text . Appendix D WeatherBench \mathbf{2} Further Details We lay out the relevant training, evaluation, and dataset details for WB 2 . D.1 Training Details and Problem Statement Our WB 2 problem statement is as follows. Given a dataset \mathcal{D}=\{X_{i}\}^{N}_{i=1} of historical weather data, the task of weather forecasting is to predict future weather conditions X_{T}\in\mathbb{R}^{V\times H\times W} given initial conditions \{X_{i}\}^{K}_{i=1},X_{i}\in\mathbb{R}^{V\times H\times W} , where T is the target lead time, K is the number of input time steps to the model, V is the number of atmospheric variables, and H\times W is the spatial resolution of the data, which depends on how densely we grid the globe. We follow the same training and hyper parameter optimization strategy as in Section C.2 . The only difference is that we use the 3 previous time steps as input instead of 5 . All models are given a consistent compute budget of 8 hours on an NVIDIA H 200 GPU for up to 100 epochs. D.2 Evaluation Details Here we give precise definitions of the evaluation and metrics omitted in the main text. We begin by establishing some notation common to the subsections, and consistent with the notation used in ( 63 ) . Let f denote the forecast, o the ground-truth observation, and c the climatology. Let t\in\{1,\ldots,T\} denote the verification time, l\in\{1,\ldots,L\} the lead time, i\in\{1,\ldots,I\} the latitude index, and j\in\{1,\ldots,J\} the longitude index. Forecasts are indexed as f_{t,l,i,j} , while observations and climatology are indexed by absolute time as o_{t,i,j} and c_{t,i,j} . D.2.1 Latitude weighting In an equiangular latitude-longitude grid, grid cells at the poles have a much smaller area compared to grid cells at the equator. Weighting all cells equally in the computation of RMSE and ACC would result in an inordinate bias towards the polar regions. As a result both metrics are latitude-weighted with weights computed as follows: w(i)=\frac{\sin\theta^{u}_{i}-\sin\theta^{l}_{i}}{\frac{1}{I}\sum^{I}_{i}(\sin\theta^{u}_{t}-\sin\theta^{l}_{i})}, where \theta^{u}_{i} and \theta^{l}_{i} indicate upper and lower latitude bounds, respectively. D.2.2 Climatology The climatology c is a function of the day of year and time of day, it is computed by taking the mean of ERA5 data from 1990 to 2019 (inclusive) for each grid point. A sliding window of 61 days is used around each day of year and time of day combination with weights linearly decaying to zero from the center. For notational consistency, we also define the lead-time–indexed climatology c_{t,l,i,j}:=c_{t+l,i,j}, corresponding to the climatology at the forecast valid time. D.2.3 Root mean squared error (RMSE) Following the WB 2 convention, our work measures error in terms of RMSE. For each variable and level pair, the RMSE at lead time l is defined as: \RMSE_{l}=\sqrt{\frac{1}{TIJ}\sum^{T}_{t}\sum^{I}_{i}\sum^{J}_{j}w(i)(f_{t,l,i,j}-o_{t,i,j})^{2}}. This choice is important for temperature forecasting, as we are invariant to choice of unit (e.g., temperature in terms of Kelvin and Celsius will have the same RMSE). Moreover, the change in scale over time is less dramatic as it was for the PyVista meshes, where we considered NRMSE. D.2.4 Anomaly correlation coefficient (ACC) The ACC is computed as the Pearson correlation coefficient of the anomalies with respect to the climatology c . Denote the differences between forecast and climatology and between observation and climatology by f^{\prime}_{t,l,i,j}=f_{t,l,i,j}-c_{t,l,i,j};\quad o^{\prime}_{t,i,j}=o_{t,i,j}-c_{t,i,j}. The ACC at lead time l is then defined as \ACC_{l}=\frac{1}{T}\sum^{T}_{t}\frac{\sum^{I}_{i}\sum^{J}_{j}w(i)f^{\prime}_{t,l,i,j}\,o^{\prime}_{t,i,j}}{\sqrt{\sum^{I}_{i}\sum^{J}_{j}w(i){f^{\prime}_{t,l,i,j}}^{2}\sum^{I}_{i}\sum^{J}_{j}w(i){o^{\prime}_{t,i,j}}^{2}}}. ACC ranges from 1 , indicating perfect correlation, to -1 , indicating perfect anti-correlation. The ECMWF states that when the ACC value falls below 0.6, it is considered that the positioning of synoptic scale features ceases to have value for forecasting purposes. D.3 Earth Mesh Discretization We construct a spherical mesh of the Earth by directly projecting the latitude–longitude grid points onto the unit sphere, and define mesh connectivity according to the original grid neighborhood structure. In order to obtain triangular faces, we further subdivide each cell into two triangles. The resulting mesh has 29040 nodes, 57600 faces, and 86640 edges. We note that this mesh construction is simpler than those used in other graph-based models, such as GraphCast ( 44 ) , which employs a subdivided icosahedron as the underlying mesh. However, our approach has the advantage that it operates directly on the native latitude–longitude grid and therefore does not require interpolation or regridding of the ERA5 data. Although more elaborate mesh constructions is likely to improve performance in real weather forecasting applications, our focus is on methodological experimentation rather than optimized weather prediction, and we therefore leave mesh optimization as an opportunity for future work. D.4 Temperature and Geopotential Extended Results We report the RMSE and ACC curves for all lead times. We see in Figure 11 that R-UniMesh stays valid for the longest lead time in terms of ACC, and is rivaled only by GemCNN in terms of RMSE for Geopotential. The results of Figure 5 and Figure 11 are also given as a table in Table 7 . Figure 11 : RMSE and ACC as a function of lead time for all models geopotential prediction. R-UniMesh has a competitive RMSE, especially at early lead time. R-UniMesh also maintains viability for lead times of roughly 2 days according to the ECMWF baseline. Model Metric 6h 12h 18h 24h 30h 36h 42h 48h 240h EGNN Z500 ACC 0.909 0.946 0.948 0.912 0.835 0.747 0.639 0.538 -0.134 Z500 RMSE 347.83 266.96 261.09 340.43 467.68 581.20 706.01 812.70 3769.31 T850 ACC 0.840 0.923 0.894 0.817 0.725 0.637 0.541 0.456 0.093 T850 RMSE 1.94 1.32 1.54 2.02 2.46 2.81 3.17 3.47 6.71 EMAN Z500 ACC 0.114 0.125 0.121 0.123 0.120 0.122 0.123 0.123 0.127 Z500 RMSE 39530.01 52028.53 65037.68 89210.70 103179.47 119529.81 139314.85 156786.23 4898180.46 T850 ACC -0.016 -0.011 -0.025 -0.029 -0.029 -0.028 -0.028 -0.028 -0.007 T850 RMSE 19.87 26.34 29.95 33.25 35.42 37.73 39.95 43.95 1097293305.30 GCN Z500 ACC 0.280 0.155 0.068 0.020 -0.011 -0.032 -0.052 -0.070 -0.101 Z500 RMSE 1897.27 2205.72 2491.37 2616.58 2798.23 3009.40 3272.80 3552.62 13147.85 T850 ACC 0.287 0.170 0.087 0.030 -0.006 -0.025 -0.033 -0.031 0.093 T850 RMSE 6.15 7.02 8.29 10.90 21.75 61.01 173.60 466.71 930146229837458.25 GemCNN Z500 ACC 0.934 0.984 0.962 0.910 0.818 0.742 0.650 0.584 -0.007 Z500 RMSE 304.11 156.00 242.86 364.36 521.75 615.11 724.60 785.92 1461.45 T850 ACC 0.879 0.995 0.901 0.776 0.670 0.581 0.477 0.392 -0.025 T850 RMSE 1.71 0.34 1.51 2.30 2.77 3.10 3.49 3.80 6.05 Hermes Z500 ACC 0.919 0.948 0.888 0.795 0.670 0.568 0.472 0.402 0.129 Z500 RMSE 339.23 289.93 463.62 667.90 927.73 1157.17 1416.43 1653.81 25644.40 T850 ACC 0.842 0.906 0.756 0.586 0.447 0.340 0.252 0.187 -0.025 T850 RMSE 1.96 1.50 2.55 3.59 4.45 5.20 5.99 6.76 293.82 MPNN Z500 ACC 0.911 0.943 0.916 0.839 0.726 0.617 0.512 0.429 0.139 Z500 RMSE 349.58 283.77 360.45 519.00 721.06 908.69 1116.36 1310.80 11191.46 T850 ACC 0.846 0.933 0.898 0.817 0.724 0.638 0.547 0.469 0.043 T850 RMSE 1.90 1.23 1.52 2.03 2.48 2.82 3.17 3.44 5.99 Transformer Z500 ACC -0.150 -0.150 -0.150 -0.150 -0.150 -0.150 -0.149 -0.149 -0.147 Z500 RMSE 55368.55 55368.50 55368.45 55520.22 55520.17 55520.12 55558.25 55558.18 55570.73 T850 ACC 0.884 0.884 0.884 0.553 0.553 0.553 0.304 0.304 -0.068 T850 RMSE 1.62 1.62 1.62 3.13 3.13 3.13 3.96 3.96 7.37 R-UniMesh (Ours) Z500 ACC 0.950 0.985 0.947 0.882 0.786 0.703 0.614 0.546 0.147 Z500 RMSE 260.31 140.60 271.28 405.01 556.83 662.19 773.67 853.87 2821.03 T850 ACC 0.888 0.964 0.889 0.774 0.679 0.598 0.504 0.428 0.134 T850 RMSE 1.63 0.91 1.63 2.37 2.85 3.20 3.65 4.04 11.68 Table 7 : Weather Forecasting Results, ACC and RMSE at Different Lead Times for Temperature and Geopotential. D.5 Smoothness Extended Results We provide smoothness errors for all models on WB 2 temperature and geopotential datasets. As seen in Table 8 , R-UniMesh is competitive across both temperature and geopotential, and is within statistical significance of the best performing EGNN model on geopotential. Model RE Temperature (\downarrow) RE Geopotential (\downarrow) GCN 6.8\cdot 10^{-2}\pm 1.7\cdot 10^{-2} 1.2\cdot 10^{-3}\pm 1.0\cdot 10^{-3} MPNN 3.0\cdot 10^{-3}\pm 2.7\cdot 10^{-3} 1.3\cdot 10^{-3}\pm 8.5\cdot 10^{-4} Hermes 3.1\cdot 10^{-1}\pm 7.5\cdot 10^{-1} 1.0\cdot 10^{-1}\pm 3.6\cdot 10^{-1} GemCNN \mathbf{8.9\cdot 10^{-4}\pm 7.0\cdot 10^{-4}} 9.4\cdot 10^{-4}\pm 6.1\cdot 10^{-4} EMAN 7.7\cdot 10^{0}\pm 5.2\cdot 10^{0} 4.1\pm 2.6\cdot 10^{-2} EGNN 1.1\cdot 10^{-3}\pm 7.3\cdot 10^{-4} \mathbf{8.3\cdot 10^{-4}\pm 1.0\cdot 10^{-3}} Transformer 9.4\cdot 10^{-4}\pm 8.2\cdot 10^{-4} 2.0\cdot 10^{-3}\pm 1.1\cdot 10^{-3} R-UniMesh (Ours) 2.2\cdot 10^{-3}\pm 1.6\cdot 10^{-3} 9.8\cdot 10^{-4}\pm 7.2\cdot 10^{-4} Table 8 : Rayleigh error for all models for all initializations on WB 2 temperature and geopotential. Best performing model is indicated with bold text . Errors scaled up by \times 40 .