Geometry-Preserving Neural Architectures on Manifolds with Boundary Karthik Elamvazhuthi, Shiba Biswal, Kian Rosenblum, Arushi Katyal, Tianli Qu, Grady Ma, Rishi Sonthalia Abstract Preserving geometric structure is important in learning. We propose a unified class of geometry-aware architectures that interleave geometric updates between layers, where both projection layers and intrinsic exponential-map updates arise as discretizations of projected dynamical systems on manifolds (with or without boundary). Within this framework, we establish universal approximation results for constrained neural ODEs. We also analyze architectures that enforce geometry only at the output, proving a separate universal approximation property that enables direct comparison to interleaved designs. When the constraint set is unknown, we learn projections via small-time heat-kernel limits, showing diffusion/flow-matching can be used as data-based projections. Experiments on dynamics over 𝕊 2 \mathbb{S}^{2} and SO ⁡ ( 3 ) \mathrm{SO}(3) , and diffusion on 𝕊 d − 1 \mathbb{S}^{d-1} -valued features demonstrate exact feasibility for analytic updates and strong performance for learned projections. 1 Introduction Many learning problems impose hard geometric constraints on their outputs. For example, data on matrix groups arise naturally in applications such as protein backbone modeling ( 42 ) and visual-inertial or drone-based pose estimation ( 43 ) , where predictions live on nonlinear manifolds. Similarly, covariance operators, required to be positive semi-definite with fixed trace ( 3 ) , define a constrained subset of Euclidean space. In such settings, predictions must lie in a prescribed constraint set M\subset\mathbb{R}^{d} in order to be physically valid or semantically meaningful. Standard machine learning (ML) architectures, including transformers, provide no guarantees that their outputs (or intermediate representations) respect these constraints. In this work, we propose geometry-aware architectures whose computations are constrained by construction . Our guiding principle is to perform expressive ambient-space transformations while interleaving them with constraint-preserving updates that map network states back onto the target set M . This design enforces invariance of the constraint set throughout the network, rather than enforcing feasibility only as a post-processing step. Problem formulation. Our objective is to define a class of functions that preserves the geometry of a given domain M\subseteq\mathbb{R}^{d} . We assume that M is a smooth manifold with boundary. Formally, we seek a family of maps \mathcal{F}=\{f:M\to M\} such that each f\in\mathcal{F} is well-defined on M and keeps M invariant, while remaining sufficiently rich to approximate broad families of transformations on M . That is, for any target mapping g:M\to M in a suitable regularity class (e.g., continuous or Lipschitz) and any \varepsilon>0 , we require a universal approximation property: there exists f\in\mathcal{F} such that d_{\mathcal{F}}(f,g)<\varepsilon for an appropriate metric d_{\mathcal{F}} on functions. We aim to adapt standard ML architectures so that their layers are intrinsically constrained to produce outputs that remain in M . The approximation property then ensures that the resulting geometry-preserving models are universally expressive. Recent works by 14 ; 10 develop manifold-valued neural networks, including VAEs with manifold latent spaces such as hyperspheres, Lie groups, and tori, and VAEs using compositions with exponential maps are considered in 37 . Manifold extensions of neural ODEs building on the adjoint method are considered in 8 , while 15 ; 34 consider intrinsic constructions of neural ODEs. There has also been significant work on building hyperbolic architectures ( 17 ; 31 ; 40 ) , as well as architectures that explore spaces with varying curvature or combinations of geometries ( 44 ; 47 ; 46 ; 33 ; 32 ) . Riemannian residual networks using the exponential map are studied in 23 , and universal approximation for manifold-valued neural ODEs of limited width is analyzed in 13 . Geometric Deep Networks (GDNs) with log-Euclidean network-exp structure and associated topological obstructions are developed in 24 , and geometry-preserving transformers with approximation guarantees under constraints are treated in 25 . A complementary optimization-theoretic line enforces feasibility by embedding projections or optimization problems into architectures. OptNet ( 2 ) differentiates through QP KKT conditions, while 9 consider repair layers that enforce constraints via closed-form or convex-analytic operations. DC3 ( 11 ) combines functional parameterizations for equalities with iterative corrections for inequalities. HardNet ( 36 ) provides closed-form differentiable projections for input-dependent affine sets, enforcing exact feasibility while preserving approximation guarantees. Despite this progress, two gaps remain central for the settings we study here. First, neural ODEs have not previously been studied in a form that guarantees invariance of a prescribed set M (including manifolds with boundary) under the learned dynamics. Second, existing architectures that enforce feasibility typically do so either via specialized geometric operations (e.g., exponential-map residual updates) or via projection/repair mechanisms tailored to restricted constraint families, leaving open a unified framework that supports both analytic and data-driven constraint-preserving updates and enables a direct comparison of where in the network constraints should be enforced. Our contributions. We develop a unified framework for geometry-preserving architectures on constraint sets (including manifolds with or without boundary). Our main contributions are as follows. (1) Projected neural ODEs and Intermediate Augmented Architectures (IAA). We introduce projected neural ODEs. Neural ODEs were originally introduced by 8 ; however, they have not previously been studied in the setting where the corresponding ODEs keep a set invariant. Our framework, based on projected dynamical systems ( 38 ) , guarantees invariance with respect to a given manifold M\subset\mathbb{R}^{d} with boundary. Through numerical discretization, constraints are enforced at every layer; we refer to this family as Intermediate Augmented Architectures (IAA) . In comparison with 23 ; 13 , we consider a larger class of constraint sets (allowing boundaries) and allow for arbitrary width. Moreover, 23 does not provide theoretical approximation results, while 13 considers approximation for a much smaller class of neural ODEs. We also consider different choices of augmentations, such as using a projection map as an alternative to the exponential map used in 23 ; 13 . (2) Final Augmented Architectures (FAA) and approximation guarantees. We consider architectures where constraints are enforced only at the output layer, as a simpler alternative to IAA and provide an empirical study comparing the two. We also provide approximation results for FAA under standard regularity assumptions for prox-regular constraint sets that may be non-convex. In this sense, our approximation results generalize those of 36 , which focus on constraint sets representable as input-dependent affine inequalities and hence exclude non-convex or non-affine settings. Additionally, we provide universal approximation results when the final augmentation is the Riemannian exponential map rather than a projection map. In comparison, 37 prove a similar result under the assumption that the underlying manifold is homeomorphic to Euclidean space, which generally need not hold. (3) Efficient intrinsic updates for Lie groups. We provide efficient intrinsic updates for a class of IAAs. When the augmentations use exponential maps, as in 23 ; 13 , we introduce simplified exponential-map layers for matrix Lie groups via Lie-algebra coordinates. These avoid the need to project network outputs onto the tangent space, simplifying implementation considerably. (4) Learning projections via flow matching. When projections are unavailable, we learn projections via a reverse-time flow-matching construction ( 30 ) using the projection-like behavior of diffusion models ( 41 ) . To the best of our knowledge, this is the first instance of using diffusion models to enforce constraints. (5) Empirical study. We conduct empirical comparisons of IAAs and FAAs on synthetic examples arising as flows of dynamics on \mathbb{S}^{2} , \mathrm{SO}(3) , the closed disk in \mathbb{R}^{2} , and Cucker–Smale consensus dynamics on (\mathrm{SO}(3))^{10} , as well as on a real-world protein dataset. We also consider the architecture of 26 that preserves constraints, but is fundamentally different from ours in structure. Figure 1 provides a summary of the cases considered in this work. Notation. In Appendix A , we provide a table of notations introduced in this paper. Throughout, \|\cdot\| denotes the standard Euclidean \|\cdot\|_{2} norm. Geometry Preserving Architectures ( Section 2 ) Intermediate Layer Augmented (IAA) ( Section 2.2 ) Final Layer Augmented (IAA) ( Section 2.2 ) Projection onto Domain ( Section 2.1 ) Projection onto Tangent Space ( Section 2.1 ) Projection onto Domain ( Section 2.2 ) Projection onto Tangent Space ( Section 2.1 )) Analytical Projection ( Section 2.1 ) Data-Based Projection ( Section 2.3 ) Analytical Projection ( Section 2.2 Analytical Projection ( Section 2.2 Data-Based Projection ( Section 2.3 ) Analytical Projection ( Section 2.2 ) Figure 1 : Overview of the geometry-preserving architectures, illustrating interleaved geometric augmentations, intrinsic updates, and final-layer projections. 2 Geometry Preserving Architectures We consider two classes of geometry-preserving architectures. The first class, IAA, introduces augmentation within the intermediate layers of the network, drawing inspiration from projected dynamical systems . The second class, FAA, applies augmentation only at the output layer. When explicit information about the underlying manifold is not available, we develop an algorithm to learn the projection map directly from the data. We do this in both IAA and FAA cases. 2.1 Intermediate Augmented Architectures To construct geometry-preserving architectures, we adopt the perspective of neural Ordinary Differential Equations (neural ODEs) ( 8 ) , which view deep networks as discretizations of continuous-time dynamical systems. This viewpoint provides a principled framework for designing architectures that respect geometric structure. In particular, by leveraging classes of dynamical systems whose flows are known to preserve prescribed constraint sets, we can derive neural architectures that inherit these invariance properties by construction, rather than enforcing them post hoc. We first review the idea of neural ODEs. Neural ODEs. Let T\geq 0 , and let f_{\theta}:[0,T]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} be a vector field parameterized by \theta\in\Theta , a class of weight parameters. A neural ODE is defined by \dot{x}(t)=f_{\theta}(t,x(t)),\qquad x(0)=x_{0}\in\mathbb{R}^{d}. (1) Neural ODEs can be interpreted as the infinite-depth limit of residual networks (ResNets) via a forward Euler discretization of ( 1 ), x^{\ell+1}=x^{\ell}+\Delta t\,f_{\theta}\left(\ell\frac{T}{L},x^{\ell}\right), (2) where \ell\in\{0,1,\ldots,L-1\} indexes the network layers. When L is large and f satisfies appropriate smoothness assumptions, the discrete dynamics ( 2 ) converge to the continuous-time neural ODE ( 1 ). Consequently, properties of deep neural networks with many layers can be analyzed through a lens of dynamical systems. This perspective also motivates the use of alternative numerical discretizations, beyond forward Euler, to design more stable neural architectures ( 18 ) . We extend the neural ODE framework to Projected Neural ODEs . To motivate this construction, we first recall the notion of projected dynamical systems ( 38 ) , drawing on tools from nonsmooth analysis ( 4 ) . Let M\subseteq\mathbb{R}^{d} be a be a closed subset. We remind the reader that the definitions and notations are provided in Appendix A . Let T_{M}(x) denote the (Clarke) tangent cone to M at a point x\in M . Let P_{A}:\mathbb{R}^{d}\rightarrow A be the (possibly set-valued) metric projection onto a closed set A\subseteq\mathbb{R}^{d} . And let N^{P}_{M}(x) denote the proximal normal cone to M at x\in M . The reach of a closed set A\subseteq\mathbb{R}^{d} is defined as \operatorname{reach}(A):=\sup\Big\{\alpha\geq 0\ \Big|\ \begin{aligned} &\text{every }z\text{ with }\operatorname{dist}(z,A)<\alpha\\ &\text{has a unique projection in }A\end{aligned}\Big\}. We will say that M is uniformly prox-regular if it has positive reach \alpha>0 , such that for each x\in M and v\in N^{P}_{M}(x) , the following inequality holds \langle v,y-x\rangle\leq\frac{\|v\|^{2}}{2\alpha}\,\|y-x\|^{2},\qquad\forall\,y\in M. (3) It is known that C^{2} manifolds have uniformly positive reach. Given these notions, let F:[0,T]\times M\rightarrow\mathbb{R}^{d} be a C^{1} vector field that is uniformly bounded. A projected dynamical system is defined by \dot{x}(t)=P_{T_{M}(x(t))}F(t,x(t)),\quad x(0)=x_{0}\in M, (4) where P_{T_{M}(x)}:\mathbb{R}^{d}\rightarrow T_{M}(x) denotes the orthogonal projection onto the tangent cone at x . Furthermore, we assume that F satisfies the following assumption. Assumption 2.1 . F(t,\cdot) is \mathscr{L} -Lipschitz on M uniformly in t\in[0,T] . That is, \|F(t,x)-F(t,y)\|\leq\mathscr{L}\|x-y\| \forall t\in[0,T] and \forall x,y\in M , with \mathscr{L}>0 . Let x_{0}\mapsto\Phi_{F}(x_{0})=x(T) denote the associated end-point map , or the solution of ( 4 ) starting from x_{0} evaluated at T . By construction, the solution of a projected dynamical system remains in M for all times, and thus preserves invariance of the constraint set. This observation motivates the definition of projected neural ODEs , obtained by replacing the vector field F in ( 4 ) with a learnable neural parameterization f_{\theta}:[0,T]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} as, \dot{y}(t)=P_{T_{M}(x(t))}f_{\theta}(t,y),\qquad y(0)=x_{0}\in M. (5) In the Theorem 2.2 below, we show that if the target vector field F satisfies suitable regularity assumptions and the admissible neural vector fields are sufficiently expressive, then the flows induced by projected neural ODEs can approximate the end-point map \Phi_{F} arbitrarily well. Theorem 2.2 . Suppose M is uniformly prox-regular. Let F,f_{\theta}:[0,T]\times M\to\mathbb{R}^{d} be such that it is continuous, satisfy 2.1 and 1. The vector fields satisfy: \|F-f_{\theta}\|_{\infty}\leq\delta , i.e., uniformly close in sup-norm, 2. \|F\|_{\infty}\leq U and \|f_{\theta}\|_{\infty}\leq U for some U<\infty . Then, with \mathscr{L}_{\alpha}:=\mathscr{L}+\frac{4U^{2}}{\alpha}+1 , where \alpha is the reach of M , one has \sup_{x_{0}\in M}\ \|\Phi_{F}(x_{0})-\Phi_{f_{\theta}}(x_{0})\|\leq\frac{\delta}{\mathscr{L}_{\alpha}}\left(e^{\mathscr{L}_{\alpha}T}-1\right). The proof is provided in Section B.1 . The class of maps representable by projected dynamical systems is very large. For example, let \Psi:M\times[0,T]\to M be a diffeotopy , i.e. each \Psi_{t}:=\Psi(\cdot,t) is a diffeomorphism and (x,t)\mapsto\Psi_{t}(x) is smooth. The associated (time-dependent) vector field whose solution realizes \{\Psi_{t}\} is X_{t}=\dot{\Psi}_{t}\circ\Psi_{t}^{-1} . Equivalently, X_{t} is the unique time-dependent vector field satisfying \dot{\Psi}_{t}(x)=X_{t}(\Psi_{t}(x))\qquad\text{for all }x\in M. Hence, any diffeotopy can be realized as a solution of a projected dynamical system. On the downside, there are some maps that cannot be realized using flows of ODEs, unless the space is augmented ( 12 ) or the norm for approximation is relaxed ( 7 ) . To implement a projected neural ODE, analogously to the standard neural ODE case, we require a discrete-time scheme (e.g., an Euler-type method) that yields a realizable network architecture. We therefore consider two recipes, depending on the chosen discretization of the underlying projected dynamical system. Projected IAAs. In this construction, we interleave a projection step after each layer \ell\in\{0,\dots,L-1\} , h^{0}=x_{0},\quad h^{\ell+1}=h^{\ell}+\Delta t\,P_{M}\!\left(f_{\theta}(\ell,h^{\ell})\right). (6) The network output is given by P_{M}\left(f_{\theta_{\ell}}(\ell,h^{\ell}))\right) . This approach assumes that the projection operator P_{M} onto M is available in closed form (or can be computed efficiently). For example, this is the case for the sphere embedded in \mathbb{R}^{3} (:= SO(3) ). Exponential IAAs. When M is a smooth manifold without boundary, we can employ an intrinsic geometric discretization ( 19 ) based on the Riemannian exponential map \exp_{x}:T_{x}M\rightarrow M where we use the differential geometric notation T_{x}M for the tangent space, which coincides with the tangent cone T_{M}(x) . In this setting, the network update is defined directly on the manifold via a geometric Euler scheme. Specifically, let {f_{\theta}}:\{0,\ldots,L-1\}\times M\rightarrow TM be a neural network that outputs a tangent vector at each point, where TM=\cup_{x\in M}T_{x}M . The update rule is h^{\ell}=\exp_{h^{\ell-1}}\!\left(\Delta t\,f_{\theta}({\ell},h^{\ell-1})\right),\quad h^{\ell}\in M. (7) By construction, the exponential update ( 7 ) guarantees that all intermediate states remain on M , removing the need for explicit projection operations while preserving geometric invariance. This sequence of updates is also considered in 23 ; 13 . Exponential IAAs on Lie Groups. While the exponential map based update can be found in 23 ; 13 , one contribution of this work is to specialize this construction to Lie Groups, where the architecture can be simplified considerably. Let G be a n -dimensional matrix Lie group embedded in \mathbb{R}^{d\times d} . The tangent space of G at g\in G , T_{g}G , can be characterized via the tangent space of G at the identity element e\in G , T_{e}G , which is canonically identified with the Lie algebra \mathfrak{g} of G . Let R_{g}:G\rightarrow G , R_{g}(h)=hg , be the right translation map. Its differential at the identity, (dR_{g})_{e}:T_{e}G\rightarrow T_{g}G, provides the identification T_{g}G=(dR_{g})_{e}(\mathfrak{g}). For matrix Lie groups, this differential is explicitly given by (dR_{g})_{e}X=Xg,\qquad X\in\mathfrak{g}. Let \{E_{1},\dots,E_{n}\} be a basis of \mathfrak{g} ( =T_{e}G ). Then \{E_{1}g,\dots,E_{n}g\} forms a basis of T_{g}G . Given any matrix A\in\mathbb{R}^{d\times d} , the orthogonal projection onto T_{g}G can be written as P_{T_{g}G}(A)=\sum_{i=1}^{n}a^{g}_{i}(A)\,E_{i}g, (8) where the i th coefficient a^{g}_{i}(A)\in\mathbb{R} depend on both A and g . Equivalently, the projection induces a coordinate map \mathbb{R}^{d\times d}\ni A\;\longmapsto\;\left(a^{g}_{1}(A),\dots,a^{g}_{n}(A)\right)^{\top}\in\mathbb{R}^{n}. The Riemannian exponential map at g\in G admits the closed-form expression \exp_{g}(X)=\exp_{e}(Xg^{-1})\,g,\quad X\in T_{g}G, where \exp_{e}:\mathfrak{g}\rightarrow G denotes the exponential map at the identity, which coincides with the matrix exponential for matrix Lie groups. Applying this formula to the projected direction ( 8 ) yields \exp_{g}\!\left(P_{T_{g}G}(A)\right)=\exp_{e}\!\left(\sum_{i=1}^{n}a^{g}_{i}(A)\,E_{i}\right)g. We may therefore express the exponential IAA update for a projected neural ODE on G as g^{\ell+1}=\exp_{e}\!\left(\Delta t\sum_{i=1}^{n}a^{g^{\ell}}_{i}\!\left(f_{\theta}(\ell,g^{\ell})\right)\,E_{i}\right)g^{\ell}, (9) where f_{\theta} is a neural network producing an ambient matrix-valued update. Rather than explicitly computing the projection coefficients a^{g}_{i}(\cdot) , we introduce a neural network \hat{f}_{\theta} with range \mathbb{R}^{n} that directly predicts the Lie algebra coordinates. The update then takes the simplified form g^{\ell+1}=\exp_{e}\!\left(\Delta t\sum_{i=1}^{n}\left(\hat{f}_{\theta}(\ell,g^{\ell})\right)_{i}\,E_{i}\right)g^{\ell}, (10) where (\cdot)_{i} denotes the i^{\text{th}} coordinate. Here, \hat{f}_{\theta} is trained to approximate the coordinate map g\;\mapsto\;\left(a^{g}_{1}(f_{\theta}(g)),\ldots,a^{g}_{n}(f_{\theta}(g))\right). Therefore, for Lie groups we do not need to project onto T_{g}G at every layer . Instead of producing a matrix and then projecting it onto the tangent space, the network directly outputs coefficients in a fixed basis of the Lie algebra (i.e., Lie algebra coordinates), which are then mapped back to the group via the exponential map. 2.2 Final Augmented Architectures Building on the architectures introduced in the previous sections, a natural question arises: what happens if the projection or exponential map is applied only at the final layer of the network, rather than interleaved throughout the architecture? In this section, we introduce FAA and establish basic approximation guarantees. Projected FAAs. We consider architectures that enforce geometric constraints only at the output layer via an explicit projection. Conceptually, these models first compute an unconstrained approximation in the ambient space and then apply a single geometric correction to map the output back onto the target manifold. While this approach does not guarantee that intermediate representations remain feasible, it is often simpler to implement and computationally cheaper than enforcing constraints at every layer. Let P_{M}:\Omega\rightarrow M be a well-defined projection onto a manifold M\subset\mathbb{R}^{d} , defined on an open neighborhood \Omega\subset\mathbb{R}^{d} containing M . Suppose f_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} is a parametrized approximation function. The corresponding constrained approximation is defined by composition with the projection map, \tilde{f}_{\theta}:=P_{M}\circ f_{\theta}. This construction raises a natural question: to what extent does projecting only at the final layer preserve approximation accuracy? Intuitively, if the unconstrained approximation f_{\theta} is “close” to the target map F in the ambient space and remains within the region where the projection is well defined, then the projection step should not significantly distort the approximation. The following theorem formalizes this intuition under standard regularity assumptions on the constraint set. Theorem 2.3 . Let M\subset\mathbb{R}^{d} be a uniformly prox-regular set with positive reach, and let P_{M}:\Omega\rightarrow M denote the metric projection onto M , where \Omega\subset\mathbb{R}^{d} is a bounded neighborhood of M contained within the reach of M . Suppose F:M\rightarrow M is a continuous target map, and let f_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d} satisfy \|F(x)-f_{\theta}(x)\|\leq\varepsilon , for all x\in\Omega . Additionally, assume that \varepsilon>0 is small enough, so that the image of f_{\theta} lies within the reach of M . Then the projected approximation satisfies \|F(x)-P_{M}(f_{\theta}(x))\|\leq 2\varepsilon,~~\forall x\in\Omega. The proof is in Section B.2 . A drawback of the result is that one requires f_{\theta} to have already well approximated F so that the composition with the projection map is valid. Exponential FAAs Similar to IAA, we consider the case where M is a manifold without boundary. And, instead of the projection map, we use the exponential map from a base point p i.e., approximation classes of the form \exp_{p}(\cdot) for some fixed base point p\in M . The corresponding constrained approximation is defined by composition with the projection map, \tilde{f}_{\theta}:=\exp_{p}\circ f_{\theta}. Using the compositions with the exponential map can prevent approximations of continuous functions on manifolds. For example, if F:M\rightarrow M is a homeomorphism and F=\exp_{x}\circ f and f:M\rightarrow T_{x}M would imply that f:M\rightarrow T_{x}M is a topological embedding. Since M is of same dimension as T_{x}M=\mathbb{R}^{d} , this is not always possible. For example, if M=S^{2} . Such topological obstructions have been identified for approximation properties of autoencoders ( 5 ; 27 ) . These obstructions arise from the inherent geometry of the underlying manifold and from the fact that one is seeking universal approximation in the uniform norm. However, if one relaxes the requirements of continuity or approximation in the uniform norm it is possible to establish an approximation result in a weaker norm. Expressibility of autoencoders in weaker norms has been shown in ( 27 ) , despite topological obstruction in stronger uniform norm. A similar idea works in our setting. Toward this end, we establish a representation result in the next theorem. Here M is not required to be embedded into a Euclidean space. We note that the following theorem requires concepts of geodesics , particularly a geodesically complete manifold, and the geodesic distance; definitions of which can be found in ( 29 , Chapters 5,6) . Theorem 2.4 . Let (M,g) be a geodesically complete finite‑dimensional connected Riemannian manifold with geodesic distance \operatorname{dist}_{M} . Fix a base point p\in M , and denote by \exp_{p}\colon T_{p}M\to M the exponential map at p . Let \Omega\subset\mathbb{R}^{d} be compact, and let F\colon\Omega\to M be continuous. Then there exists a measurable function f\colon\Omega\to T_{p}M with bounded range such that for all x\in\Omega , F(x)=\exp_{p}\left(f(x)\right). Suppose further that f_{\theta}:\Omega\to T_{p}M satisfies \|f_{\theta}-f\|_{L^{2}_{\mu}(\Omega)}<\delta , for some \delta>0 and a Borel measure \mu on M . Let K\subset T_{p}M be a compact set containing the images of both f_{\theta} and f , and let \mathscr{L}>0 be a Lipschitz constant for \exp_{p} restricted to K . Then \left\|\operatorname{dist}_{M}\left(F(x),\,\exp_{p}(f_{\theta}(x))\right)\right\|_{L^{2}_{\mu}(\Omega)}<\mathscr{L}\,\delta. The proof is given in Section B.3 . The assumption that an architecture produces a map f_{\theta}:\Omega\to T_{p}M with \|f_{\theta}-f\|_{L^{2}_{\mu}(\Omega)}<\delta is mild and allows the application of standard approximation results. In particular, since f is measurable and bounded almost everywhere, one may first approximate f in L^{2}_{\mu}(\Omega) by a smooth map \tilde{f}:\Omega\to T_{p}M (using density of smooth functions in L^{2}_{\mu} ; see ( 6 , Corollary 4.2.2) , viewing \mu as a measure on \mathbb{R}^{d} , that is supported on the embedded manifold M ). Next, one can approximate \tilde{f} uniformly on the compact set \Omega by a chosen function class (e.g., multi-layer perceptrons via classical universal approximation theorems). Combining these two steps yields parameters \theta such that \|f_{\theta}-f\|_{L^{2}_{\mu}(\Omega)} is arbitrarily small. 2.3 Learned Projections via Flow Matching While the methods discussed in previous sections allow us to define projected transformers when an explicit projection map onto a set is available, there are many situations where this projection map is not known in closed form and must instead be learned from data. We present such a method for constructing an approximate projection map using diffusion models and flow matching, based on ideas on projection like behavior of diffusion models ( 41 ) . This is closely connected to Varadhan’s asymptotics of the heat kernel approximating the distance function ( 45 ; 35 ) . We use this method to learn a (metric) projection P_{M}:\mathbb{R}^{d}\to M . We learn a time-dependent vector field v_{\theta}:\mathbb{R}^{d}\times[0,T]\to\mathbb{R}^{d} , parameterized by \theta , to recover a projection \tilde{P}_{M} that is an approximation of P_{M} , by integrating the reverse-time dynamics. For x,y\in\mathbb{R}^{d} and t\in\mathbb{R} , let K(t,x,y) denote the heat kernel, K(t,x,y)=\frac{1}{(4\pi t)^{d/2}}\exp\left(-\frac{\|x-y\|^{2}}{4t}\right). It defines the transition density of the Brownian motion W(t) starting at y , \mathbb{P}(W(t)\in dx\mid W(0)=y)=K(t,x,y)\,dx, From a generalization of Varadhan’s formula ( 39 ; 21 ) , we know that for any measurable set A -\lim_{t\to 0}4t\log\mathbb{P}_{x}(W(t)\in A)=\operatorname{dist}^{2}(x,A) where \operatorname{dist}(x,A)=\inf_{y\in A}\|x-y\| is the Euclidean distance from x to A . Therefore, for small t , the heat kernel smoothed density \mu_{t}(x)=\int_{\Omega}K(t,x,y)\,d\mu(y) approximately satisfies \log\mu_{t}(x)\approx-\frac{\operatorname{dist}^{2}(x,M)}{4t}+\text{const}. Differentiating both sides gives \nabla_{x}\log\mu_{t}(x)\approx-\frac{1}{2t}\nabla_{x}\operatorname{dist}^{2}(x,M). Since \nabla_{x}\operatorname{dist}^{2}(x,M)=2(x-P_{M}(x)) , we have \nabla_{x}\log\mu_{t}(x)\approx-\frac{x-P_{M}(x)}{t} , and hence, P_{M}(x)\approx x+t\nabla_{x}\log\mu_{t}(x). This shows that, for small t , the gradient of the smoothed density \mu_{t} provides a good approximation to the displacement from x to P_{M}(x) . Rather than estimate \nabla_{x}\log\mu_{t}(x) directly via score matching, we propose to estimate it through a flow matching framework ( 30 ) . In this approach, perturbed samples are generated as: Y_{t}=X+vt,\quad v\sim\mathcal{N}(m,\mathbb{I}), where X\sim\mu . The distribution \mu_{t} is precisely the distribution of Y_{t} . Although these paths are random, \mu_{t} can equivalently be realized as the solution to a deterministic flow equation: \dot{\hat{Y}}_{t}=v(t,\hat{Y}_{t}), where \hat{Y}_{t} has the same law as Y_{t} and v(t,x) is the conditional mean velocity: v(t,x):=\mathbb{E}[v\mid Y_{t}=x]. It can be shown that, v(t,x)=\nabla_{x}\log\mu_{t}(x). Once v(t,x) is estimated, we can approximate the projection map by solving the backward ODE: \dot{x}(t)=-v(t,x(t)). In practice, we learn v(t,x) using a flow-matching loss: \mathcal{L}(\theta):=\int_{0}^{T}\mathbb{E}_{x\sim\mu_{t}}\mathbb{E}_{v\mid x}\left[\|v_{\theta}(x,t)-v\|^{2}\right]dt. In particular, as t\to 0 , the learned backward flow: \dot{x}(s)=-v_{\theta}(t,x(s)) approximates the displacement from x to \Omega , providing an approximate projection mechanism. In the following theorem we prove an approximation result justifying this method of constructing the projection. This can be viewed as a deterministic analogue of ( 41 , Proposition 3.1) and gives a uniform small-noise ( t\downarrow 0 ) guarantee that the score-induced map x\mapsto x+t\nabla\log u_{t}(x) returns the metric projection P_{M}(x) up to O(t^{1/2}) on neighborhoods of the manifold. Theorem 2.5 . Let M\subset\mathbb{R}^{d} be a C^{\infty} compact embedded m -dimensional submanifold without boundary and positive reach \alpha>0 , with the Riemannian metric inherited by the embedding. For x\in\mathbb{R}^{d} and t>0 , define u_{t}(x)=\int_{S}(2\pi t)^{-d/2}\exp\!\left(-\frac{\|x-y\|^{2}}{2t}\right)\,d\mu(y), where d\mu is the induced Riemannian measure on M . Then for every compact set K\subset\{x\in\mathbb{R}^{d}:\operatorname{dist}(x,M)<\alpha\} , there exists a constant C_{K}<\infty such that for all x\in K and all sufficiently small t>0 , \left\|\nabla_{x}\log u_{t}(x)+\frac{x-P_{M}(x)}{t}\right\|\leq C_{K}, where P_{M}(x) denotes the (unique) metric projection of x onto M . Equivalently, on compact subsets of the tubular neighborhood of M , as t\downarrow 0 , x+t\,\nabla_{x}\log u_{t}(x)=P_{M}(x)+O(t^{1/2}),~\text{uniformly.} The proof is provided in Section B.4 . (a) Projection MSE vs. \sigma (b) Distance-to-manifold vs. \sigma Figure 2 : Analysis of projection performance: (a) shows the Mean Squared Error, and (b) shows the distance to the manifold relative to the noise standard deviation \sigma . More details on the distance used can be found in Appendix D.7 Sphere Disk SO(3) CS Protein Model Test loss Mean dist Test loss Mean dist Test loss Mean dist Test loss Mean dist Test loss Mean dist Regular 4.97\mathrm{e}{-3} 2.44\mathrm{e}{-2} 4.95\mathrm{e}{-4} 7.36\mathrm{e}{-3} 1.10\mathrm{e}{-1} 1.40\mathrm{e}{-0} 3.33\mathrm{e}{-1} 2.72\mathrm{e}{-0} 1.19\mathrm{e}{-1} 2.28\mathrm{e}{-0} Proj IAA 6.29\mathrm{e}{-3} 2.35\mathrm{e}{-8} 4.23\mathrm{e}{-7} 1.09\mathrm{e}{-9} 2.22\mathrm{e}{-1} 8.00\mathrm{e}{-7} – – 1.90\mathrm{e}{-1} 8.73\mathrm{e}{-7} Exp IAA 8.86\mathrm{e}{-3} 2.19\mathrm{e}{-9} – – 3.10\mathrm{e}{-1} 3.14\mathrm{e}{-6} 6.39\mathrm{e}{-1} 4.78\mathrm{e}{-6} 2.39\mathrm{e}{-1} 7.63\mathrm{e}{-3} Flow IAA 9.69\mathrm{e}{-3} 2.70\mathrm{e}{-2} 1.45\mathrm{e}{-3} 7.93\mathrm{e}{-3} 1.12\mathrm{e}{-1} 1.21\mathrm{e}{-0} 3.35\mathrm{e}{-1} 2.71\mathrm{e}{-0} 1.19\mathrm{e}{-1} 2.28\mathrm{e}{-0} Prob 7.48\mathrm{e}{-3} 1.50\mathrm{e}{-2} 2.58\mathrm{e}{-3} 0.00\mathrm{e}{-0} 1.68\mathrm{e}{-1} 1.48\mathrm{e}{-0} 3.32\mathrm{e}{-1} 2.72\mathrm{e}{-0} 5.50\mathrm{e}{-1} 2.15\mathrm{e}{-0} Proj FAA 5.03\mathrm{e}{-3} 2.21\mathrm{e}{-8} 7.33\mathrm{e}{-7} 1.89\mathrm{e}{-9} 2.52\mathrm{e}{-1} 8.26\mathrm{e}{-7} 6.38\mathrm{e}{-1} 8.42\mathrm{e}{-7} 1.88\mathrm{e}{-1} 8.36\mathrm{e}{-7} Exp FAA 2.16\mathrm{e}{-1} 1.99\mathrm{e}{-9} – – 2.47\mathrm{e}{-1} 1.47\mathrm{e}{-6} 6.36\mathrm{e}{-1} 1.89\mathrm{e}{-6} 2.41\mathrm{e}{-1} 7.63\mathrm{e}{-3} Flow FAA 7.69\mathrm{e}{-3} 2.53\mathrm{e}{-2} 8.52\mathrm{e}{-4} 5.44\mathrm{e}{-3} 1.31\mathrm{e}{-1} 1.43\mathrm{e}{-0} 3.34\mathrm{e}{-1} 2.72\mathrm{e}{-0} 1.19\mathrm{e}{-1} 2.28\mathrm{e}{-0} Table 1: Test loss and mean distance to the constraint manifold. Dashes indicate model/dataset combinations not run, either due to computational cost (CS Proj) or not defined (Disk Exp). More details on the distance used can be found in Appendix D.7 . 3 Numerical Experiments In this section, we test our models on a number of examples. For each example, Appendix C provides the details of the characterization of the tangent space, projection, exponential map for each of the examples below. The relevant code used to generate the results can be found at the following Anonymous Github . Example 1: The Special Orthogonal Group \mathrm{SO}(d)=\{O\in\mathbb{R}^{d\times d}:O^{\top}O=I,\,\det O=1\}. Example 2: The Sphere S^{d}=\{x\in\mathbb{R}^{d+1}:\|x\|_{2}=1\} Example 3: Manifold with boundary, closed unit disk K=\{x\in\mathbb{R}^{2}:\|x\|\leq 1\}. Example 4: Cucker-Smale Dynamics on SO(3) . We consider a synthetic sequence-to-sequence learning problem on a nonlinear manifold, where the underlying dynamics evolve on the rotation group SO(3) . The goal of this example, is to show that our approach for constructing geometry preserving architectures also works for approximating constrained sequence-to-sequence maps using transformers. Example 5: Protein dataset on SE(3) . We use the ProteinNet tertiary structure field, which specifies, for each residue i (each amino acid in the protein chain), the Cartesian coordinates of the backbone atoms N_{i} , C_{\alpha i} , and C_{i} . Experiment 1: Learned projection via flow matching. The first experiment we run, learn the projection map onto the manifold using flow matching and the training data (see Section 2.3 ). Specifically, for each dataset, we learn an approximate projection map \tilde{P}_{M}:\mathbb{R}^{d}\to\Omega , to simulate the absence of an analytical projection map. We do this by fitting a time-dependent velocity field v_{\theta}:\mathbb{R}^{d}\times[0,T]\to\mathbb{R}^{d} in a flow-matching framework and then integrating the reverse-time dynamics. More details can be found in Appendix D . After training a flow-matching projector for each dataset, we selected a single model per dataset by choosing the checkpoint with the lowest validation loss over the full hyperparameter sweep (Appendix D ). To assess robustness and projection quality, we constructed noisy test inputs by corrupting clean held-out test samples with isotropic Gaussian noise: \tilde{x}=x+\varepsilon , \varepsilon\sim\mathcal{N}(0,\sigma^{2}I) . For each noise level \sigma , we compared two projection operators applied to \tilde{x} : ( i ) an analytic projection available for the corresponding constraint set, and ( ii ) the learned flow-matching projection obtained by integrating the learned reverse-time dynamics. We then evaluated (a) reconstruction error via mean-squared error (MSE) as a function of \sigma ( Figure 2(a) ) and (b) post-projection constraint satisfaction via a dataset-specific distance-to-manifold diagnostic ( Figure 2(b) ). The resulting curves summarize how projection error and constraint violation evolve with increasing corruption across the sphere, \mathrm{SO}(3) , protein, disk, and contextual sequence (cs) datasets. Here we see that when the points are close to the manifold, we approximate the projection well and as we get further away the approximation quality degrades. This is expected due to the O(\sqrt{t}) error term in Theorem 2.5 . Experiment 2: Learning dynamics on manifolds. For each of the five datasets, we compare a family of geometry-aware architectures designed to preserve (or recover) the underlying constraint set M . Concretely, we evaluate: ( i ) an unconstrained residual network ( regular ResNet ), ( ii ) intrinsic–ambient architectures based on iterative alignment updates ( IAA ) with either analytic projection ( projected IAA ), exponential-map updates ( exp IAA ), or a learned projector obtained via flow matching ( flow IAA ), ( iii ) analogous feedforward architectures ( FAA ) with projected/exp/flow variants, and (iv) a probabilistic constrained transformer baseline following ( 25 ) . Across all model classes, we sweep over network depth (ResNet depth 4 , 6 , and 8 ) and weight decay (enabled/disabled). Each configuration is trained for 10{,}000 epochs with an AdamW optimizer and learning-rate reduction on validation-loss plateaus. We select the best checkpoint per configuration by validation loss, and then report test performance in terms of mean-squared error (MSE) on the prediction target and a dataset-specific distance-to-manifold diagnostic measuring constraint violation. The aggregate results across datasets and model classes are summarized in Table 1 . More details can be found in Appendix E . Discussion: Across datasets, the projected variants display a favorable Pareto tradeoff: they achieve simultaneously low test MSE and low distance-to-manifold, whereas other approaches typically improve one metric without matching the other. We also observe that the exponential variants improve constraint satisfaction relative to the regular (ambient) baseline, indicating that intrinsic updates can reduce off-manifold drift. By contrast, among methods that do not explicitly encode knowledge of the constraint set, namely the regular , flow , and probabilistic baselines, performance is broadly similar, both in test loss and in constraint violation, suggesting that without geometric information these models behave comparably. In light of Theorem 2.5 , one might expect the flow-matching projection to yield performance closer to that of the analytical projection-based models. However, the experiments suggest that realizing this behavior is not straightforward. Together with Figure 2 , this points to two plausible explanations: either the trained flow-matching model has not yet learned the projection with sufficient accuracy, or the intermediate layers produce states that lie too far from the manifold for the learned projection to correct reliably. Comparing the exponential-based verses projection-based architectures, we note that projected IAA performs better than the exponential IAA, and the results are comparable in the FAA case. Moreover, we remark that we are missing data for the projected IAA case in the CS example as it exceeded compute time. Moreover,the results for both FAA and IAA are very close, however FAA enjoys a significant computational advantage over IAA. A careful comparison was missing from existing literature. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendix A Notation Symbol Description C^{m} , m\in\mathbb{Z}_{>0} Space of m -times continuously differentiable functions T_{M}(x) Tangent cone of the set M at x : \left\{v\in\mathbb{R}^{d}\;\Bigg|\;\begin{aligned} &\forall\,x_{k}\in M,\;x_{k}\to x,\forall\,t_{k}\downarrow 0,\;\\ &\exists\,v_{k}\to v,x_{k}+t_{k}v_{k}\in M\end{aligned}\right\} T_{x}M Tangent space of the manifold M at the point x . TM Tangent bundle of M : \cup_{x\in M}T_{x}M P_{A}(x) Metric projection of x\in\mathbb{R}^{d} onto the set A : \arg\min_{y\in A}\|x-y\|_{2}~~\forall x\in\mathbb{R}^{d} N^{P}_{M}(x) Proximal normal cone to M at x : \left\{v\in\mathbb{R}^{d}\,\middle|\,\exists\,r>0\text{ s.t. }x\in P_{M}(x+rv)\right\} \|g\|_{\infty} , g\in C([0,T]\times M;\mathbb{R}^{d}) \sup_{\begin{subarray}{c}t\in[0,T],\\ x\in M\end{subarray}}\|g(t,x)\| B(x,r) Open Euclidean ball of radius r>0 centered at x\in\mathbb{R}^{d} . Df , \partial_{i}f Jacobian (or total derivative) of f and partial derivative of f with respect to the i -th coordinate. Appendix B Proofs B.1 Proof of Theorem 2.2 See 2.2 Proof. Fix x_{0}\in M and set x(t):=x(t;x_{0}) , y(t):=y(t;x_{0}) , and e(t):=x(t)-y(t) . For a.e. t , define \xi_{x}(t):=F(t,x(t))-P_{T_{M}(x(t))}\left(F(t,x(t))\right),\qquad\xi_{y}(t):=f_{\theta}(t,y(t))-P_{T_{M}(y(t))}\left(f_{\theta}(t,y(t))\right). It is known (for instance, from ( 20 , Lemma 4.6) that \xi_{x}(t)\in N^{P}_{M}(x(t)),\qquad\xi_{y}(t)\in N^{P}_{M}(y(t)), and the dynamics can be rewritten as \dot{x}(t)=F(t,x(t))-\xi_{x}(t),\qquad\dot{y}(t)=f_{\theta}(t,y(t))-\xi_{y}(t). In particular, since projections onto the Tangent cone are non-expansive, \|\xi_{x}(t)\|\leq\|F(t,x(t))\|\leq U and \|\xi_{y}(t)\|\leq\|f_{\theta}(t,y(t))\|\leq U . Differentiate \frac{1}{2}\|e(t)\|^{2} : \displaystyle\frac{1}{2}\frac{d}{dt}\|e(t)\|^{2} \displaystyle=\langle e(t),\dot{x}(t)-\dot{y}(t)\rangle \displaystyle=\langle e(t),F(t,x(t))-f_{\theta}(t,y(t))\rangle-\langle e(t),\xi_{x}(t)-\xi_{y}(t)\rangle \displaystyle=\langle e(t),F(t,x(t))-F(t,y(t))\rangle+\langle e(t),F(t,y(t))-f_{\theta}(t,y(t))\rangle-\langle e(t),\xi_{x}(t)-\xi_{y}(t)\rangle. ( * ) By Lipschitz continuity and uniform approximation, respectively, the first two terms satisfy \langle e(t),F(t,x)-F(t,y)\rangle\leq\mathscr{L}\|e(t)\|^{2},\qquad\langle e(t),F(t,y)-f_{\theta}(t,y)\rangle\leq\|e(t)\|\,\delta. For the normal term, using the positive reach inequality ( 3 ) we get, \langle\xi_{x}(t),y(t)-x(t)\rangle\leq\frac{\|\xi_{x}(t)\|^{2}}{2\alpha}\|e(t)\|^{2},\qquad\langle\xi_{y}(t),x(t)-y(t)\rangle\leq\frac{\|\xi_{y}(t)\|^{2}}{2\alpha}\|e(t)\|^{2}. The third term in (*) therefore satisfies, -\langle e(t),\xi_{x}(t)-\xi_{y}(t)\rangle=\langle\xi_{x}(t),y(t)-x(t)\rangle+\langle\xi_{y}(t),x(t)-y(t)\rangle\leq\frac{\|\xi_{x}(t)\|^{2}+\|\xi_{y}(t)\|^{2}}{2\alpha}\|e(t)\|^{2}\leq\frac{U^{2}}{\alpha}\|e(t)\|^{2}. Therefore, for a.e. t , \frac{1}{2}\frac{d}{dt}\|e(t)\|^{2}\leq\left(\mathscr{L}+\frac{U^{2}}{\alpha}\right)\|e(t)\|^{2}+\delta\|e(t)\|. Using Young’s inequality, we have \delta\|e\|\leq\frac{1}{2}\|e\|^{2}+\frac{1}{2}\delta^{2} . We obtain \frac{d}{dt}\|e(t)\|^{2}\leq 2\left(\mathscr{L}+\frac{U^{2}}{\alpha}+\frac{1}{2}\right)\|e(t)\|^{2}+\delta^{2},\qquad\|e(0)\|=0. Then the differential inequality can be written as \frac{dg}{dt}\leq a\,g(t)+b(t),\qquad g(0)=0. with a=2\mathscr{L}+2\frac{U^{2}}{\alpha}+1 By Grönwall’s inequality, we obtain g(t)\leq\delta^{2}t+\int_{0}^{t}a\,(\delta^{2}s)\exp\!\left(\int_{s}^{t}a\,du\right)\,ds. Since \int_{s}^{t}a\,du=a(t-s) , this becomes g(t)\leq\delta^{2}t+a\delta^{2}\int_{0}^{t}s\,e^{a(t-s)}\,ds. Moreover, \int_{0}^{t}s\,e^{a(t-s)}\,ds=-\frac{t}{a}+\frac{e^{at}-1}{a^{2}}. Plugging this back in gives g(t)\leq\delta^{2}t+a\delta^{2}\left(-\frac{t}{a}+\frac{e^{at}-1}{a^{2}}\right), g(t)\leq\delta^{2}t-\delta^{2}t+\frac{\delta^{2}}{a}\left(e^{at}-1\right), g(t)\leq\frac{\delta^{2}}{a}\left(e^{at}-1\right). Therefore, at t=T , \|e(T)\|^{2}=g(T)\leq\frac{\delta^{2}}{a}\left(e^{aT}-1\right)=\frac{\delta^{2}}{2\left(\mathscr{L}+\frac{U^{2}}{\alpha}+\frac{1}{2}\right)}\,\exp{\left(2T\left(\mathscr{L}+\frac{U^{2}}{\alpha}+\frac{1}{2}\right)-1\right)}.\qed B.2 Proof of Theorem 2.3 See 2.3 Proof. Fix x\in\Omega and set z:=f_{\theta}(x) . By assumption P_{M}(z) is well-defined. Since F(x)\in M , by the the fact that P_{M} is the projection, \|z-P_{M}(z)\|=\operatorname{dist}(z,M)\leq\|z-F(x)\|. Using the hypothesis that \|z-F(x)\|\leq\varepsilon , we obtain \|f_{\theta}(x)-P_{M}(f_{\theta}(x))\|\leq\varepsilon. Finally, the triangle inequality gives \|F(x)-P_{M}(f_{\theta}(x))\|\leq\|F(x)-f_{\theta}(x)\|+\|f_{\theta}(x)-P_{M}(f_{\theta}(x))\|\leq\varepsilon+\varepsilon=2\varepsilon. Since x\in\Omega was arbitrary, the bound holds for all x\in\Omega . \square ∎ B.3 Proof of Theorem 2.4 See 2.4 Proof. Since \Omega\subset\mathbb{R}^{d} is compact and F:\Omega\to M is continuous, the image F(\Omega)\subset M is compact. The distance function q\mapsto\operatorname{dist}_{M}(p,q) is continuous on M , hence the composition x\mapsto\operatorname{dist}_{M}\left(p,F(x)\right) is continuous on \Omega . By compactness of \Omega , this function attains its maximum, so there exists R:=\max_{x\in\Omega}\operatorname{dist}_{M}\left(p,F(x)\right)<\infty . For each x\in\Omega , define the set of minimizing initial velocities \Phi_{\min}(x):=\left\{v\in T_{p}M\;\middle|\;\exp_{p}(v)=F(x)\ \text{and}\ \|v\|=\operatorname{dist}_{M}\left(p,F(x)\right)\right\}. By the Hopf-Rinow theorem ( 29 , Corollary 6.15) , there exists a minimizing geodesic from p to F(x) , hence \Phi_{\min}(x)\neq\varnothing for all x\in\Omega . Moreover, for every v\in\Phi_{\min}(x) we have \|v\|\leq R , so \Phi_{\min}(x)\subset\overline{B(0,R)}\subset T_{p}M . The closed ball \overline{B(0,R)} is compact. The set \Phi_{\min}(x) is closed as it is the intersection of the closed sets \exp_{p}^{-1}(\{F(x)\})\quad\text{and}\quad\{v\in T_{p}M:\|v\|=\operatorname{dist}_{M}(p,F(x))\}. Hence \Phi_{\min}(x) is compact for every x\in\Omega . Define the graph \Gamma_{\min}:=\left\{(x,v)\in\Omega\times T_{p}M\;\middle|\;v\in\Phi_{\min}(x)\right\}. Using continuity of F , continuity of \exp_{p} , and continuity of the function x\mapsto\operatorname{dist}_{M}(p,F(x)) , it follows that \Gamma_{\min} is closed in \Omega\times T_{p}M , hence Borel measurable. Therefore, the set-valued function \Phi_{\min}:\Omega\to 2^{T_{p}M} , where 2^{T_{p}M} is the power set of all subsets of T_{p}M , is measurable, nonempty, and compact-valued. By the measurable selection theorem, ( 6 , Theorem 8.1.3) , there exists a measurable function f:\Omega\to T_{p}M such that f(x)\in\Phi_{\min}(x) for all x\in\Omega . In particular, \exp_{p}(f(x))=F(x)\quad\text{and}\quad\|f(x)\|\leq R\quad\text{for all }x\in\Omega. Consequently, f(\Omega)\subset\overline{B(0,R)} , which is compact. This yields a representation of F using the exponential map and the function f with range in T_{p}M . Since \exp_{p} is Lipschitz on the compact set K , we have \operatorname{dist}_{M}\left(\exp_{p}(f(x)),\exp_{p}(f_{\theta}(x))\right)\leq\mathscr{L}\|f(x)-f_{\theta}(x)\|. Integrating and using \|f_{\theta}-f\|_{L^{2}_{\mu}}<\delta yields the stated bound. ∎ B.4 Proof of Theorem 2.5 See 2.5 Proof. Fix a compact set K\subset\{x\in\mathbb{R}^{d}:\operatorname{dist}(x,M)<\alpha\} . For each x\in K , let y^{*}=P_{M}(x),\qquad n=x-y^{*},\qquad\mathsf{d}=\|n\|. We will reserve the symbol C for a generic constant, that is independent of x\in K . Recall u_{t}(x)=(2\pi t)^{-d/2}\int_{M}e^{-\|x-y\|^{2}/(2t)}\,d\mu(y), and define D_{t}(x)=\int_{M}e^{-\|x-y\|^{2}/(2t)}\,d\mu(y),\quad A_{t}(x)=\int_{M}(x-y)e^{-\|x-y\|^{2}/(2t)}\,d\mu(y). (11) Then \nabla_{x}u_{t}(x)=-\frac{1}{t}A_{t}(x),\qquad\nabla_{x}\log u_{t}(x)=-\frac{1}{t}\,\frac{A_{t}(x)}{D_{t}(x)}. Thus it suffices to prove \frac{A_{t}(x)}{D_{t}(x)}=n+O_{K}(t^{1/2})\quad\text{uniformly for }x\in K,\ t\downarrow 0. (12) Local graph representation and local geometry near y^{*} . Choose an orthogonal splitting \mathbb{R}^{d}=T_{y^{*}}M\oplus N_{y^{*}}M with T_{y^{*}}M\simeq\mathbb{R}^{m} . It is well-known (for instance, by combining implicit function theorem and Proposition 5.16 of ( 28 ) ) that there exist r_{0}>0 and a map y:B(0,r_{0})\subset T_{y^{*}}M\to M defined by y(s)=y^{*}+s+h(s), (13) where h:B(0,r_{0})\subset T_{y^{*}}M\longrightarrow N_{y^{*}}M is C^{\infty} map satisfying h(s)=\tfrac{1}{2}D^{2}h(0)[s,s]+R_{3}(s),\quad\|R_{3}(s)\|\leq C\,\|s\|^{3}, with h(0)=0 , Dh(0)=0 . The bilinear map D^{2}h(0):T_{y^{*}}M\times T_{y^{*}}M\to N_{y^{*}}M is the Hessian of h at 0 , and in these coordinates it coincides with the second fundamental form of M at y^{*} : \mathrm{II}_{y^{*}}(s_{1},s_{2})=D^{2}h(0)[s_{1},s_{2}],\quad s_{1},s_{2}\in T_{y^{*}}M. It satisfies \|D^{2}h(0)\|\leq C , where C depends continuously on y^{*} . Since M has positive reach \alpha>0 , the projection map P_{M} is continuous on K . Hence P_{M}(K)\subset M is compact. Therefore, all the constants r_{0} and C s can be chosen uniformly for all y^{*}\in P_{M}(K) . Surface measure in local coordinates The Riemannian surface measure \mu on M induces a pullback measure on T_{y^{*}}M via the parametrization y ( 13 ). For any measurable set A\subseteq B(0,r_{0}) , \mu(y(A))=\int_{A}J(s)\,ds,\qquad J(s)=\sqrt{\det g(s)}, where g is the induced Riemannian metric on T_{y^{*}}M . According to of ( 29 , Chapter 3) , it is defined through the parametrization y(s) as g_{s}(u,v):=\big\langle Dy(s)\,u,\;Dy(s)\,v\big\rangle_{\mathbb{R}^{d}},\qquad u,v\in T_{s}U\simeq\mathbb{R}^{m}. In the present y coordinates ( 13 ) , one has \partial_{i}y(s)=e_{i}+\partial_{i}h(s), where \{e_{1},\dots,e_{m}\} is an orthonormal basis of T_{y^{*}}M . Since h\in C^{\infty} and Dh(0)=0 , the map Dh\in C^{1} , and hence \|Dh(s)\|\leq C\|s\| for some C and \|s\| small, i.e., \|\partial_{i}h(s)\|=O(\|s\|),\qquad\|\partial_{i}h(s)\|\;\|\partial_{j}h(s)\|=O(\|s\|^{2}), and hence g_{ij}(s)=\delta_{ij}+\langle\partial_{i}h(s),\,\partial_{j}h(s)\rangle=\delta_{ij}+O(\|s\|^{2}). Since \det g(s) is a smooth function of the entries of g(s) and g(s)\to I_{m} as s\to 0 , we obtain the uniform expansion J(s)=\sqrt{\det g(s)}=1+J_{2}(s),\quad J_{2}(s):=O(\|s\|^{2}). (14) And the surface measure satisfies d\mu(y(s))=J(s)\,ds . Second-order geometry: shape operator. In the local graph chart y , the second fundamental form of M at y^{*} is encoded by the quadratic term D^{2}h(0)[s,s]\in N_{y^{*}}M , s\in T_{y^{*}}M . Fix a normal vector n\in N_{y^{*}}M and define the associated (weighted) shape operator S_{n}:T_{y^{*}}M\to T_{y^{*}}M by the relation \langle S_{n}s,\,s\rangle:=\langle n,\,D^{2}h(0)[s,s]\rangle,\qquad s\in T_{y^{*}}M. From Federer’s reach theorem ( 1 , Proposition A.1.) it follows that \|S_{n}\|_{\operatorname{op}}=\max_{i}\frac{\|n\|}{{\alpha}}\leq\frac{\mathsf{d}}{\alpha}<1. Define the linear operator Q_{n}:=I-S_{n}. Since S_{n} is self-adjoint, we obtain the uniform coercivity estimate for s\in T_{y^{*}}M , \langle Q_{n}s,\,s\rangle\geq\left(1-\frac{\mathsf{d}}{\alpha}\right)\|s\|^{2}\quad\text{or,~}Q_{n}\succeq\delta I.\quad\delta:=\left(1-\frac{\mathsf{d}}{\alpha}\right) We can see that we can take the constants \mathsf{d} and \delta to be uniform. Since K\subset\{x:\operatorname{dist}(x,M)<\alpha\} is compact, we can define \mathsf{d}<\sup_{x\in K}\operatorname{dist}(x,M)<\alpha Domain decomposition. We decompose M into two regions: M=\underbrace{y(B(0,r_{0}))}_{\mathrm{near}}\;\cup\;\underbrace{M\setminus y(B(0,r_{0}))}_{\mathrm{far}}. Let D_{t}^{\mathrm{near}},D_{t}^{\mathrm{far}} and A_{t}^{\mathrm{near}},A_{t}^{\mathrm{far}} denote the corresponding contributions to D_{t} and A_{t} ( 11 ) from these regions. Far region computation. Since y^{*}=P_{M}(x) is the unique minimizer of z\mapsto\|x-z\| on M and y(s) is a local graph around y^{*} , there exists \varepsilon_{0}>0 (uniform for x\in K ) such that \|x-z\|\,\geq\,\mathsf{d}+\varepsilon_{0},\qquad z\in M\setminus y(B(0,r_{0})). Consequently, for all t>0 sufficiently small, |D_{t}^{\mathrm{far}}(x)|+\|A_{t}^{\mathrm{far}}(x)\|\;\leq\;C\exp\left(-\frac{(\mathsf{d}+\varepsilon_{0})^{2}}{2t}\right)=\exp\left(-\frac{\mathsf{d}^{2}}{2t}\right)\,O\left(\exp\left(-\frac{c}{t}\right)\right), (15) where the implicit constant C (that results from the integration) depends only on K . Therefore, the far-region term is exponentially smaller than the leading factor e^{-\mathsf{d}^{2}/(2t)} as t\downarrow 0 . Hence the effect of the far-region contributions to D_{t} and A_{t} will be negligible, as we will see further ahead. Computing \|x-y(s)\|^{2} on the chart. Towards the near-region computation, we first compute \|x-y(s)\|^{2} . On the local graph chart y(s) , x-y(s)=x-y^{*}-s-h(s)=n-s-\tfrac{1}{2}D^{2}h(0)[s,s]-R_{3}(s), (16) where R_{3}(s)=O(\|s\|^{3}) uniformly over y^{*}\in P_{M}(K) . Using the orthogonal decomposition \mathbb{R}^{d}=T_{y^{*}}M\oplus N_{y^{*}}M , we note that s\in T_{y^{*}}M , n\in N_{y^{*}}M , and given h\in N_{y^{*}}M , \tfrac{1}{2}D^{2}h(0)[s,s]+R_{3}(s)\in N_{y^{*}}M . We obtain \displaystyle\|x-y(s)\|^{2} \displaystyle=\left\|n-\tfrac{1}{2}D^{2}h(0)[s,s]-R_{3}(s)\right\|^{2}+\|s\|^{2} \displaystyle=\|n\|^{2}-\left\langle n,D^{2}h(0)[s,s]\right\rangle-2\langle n,R_{3}(s)\rangle+\left\|\tfrac{1}{2}D^{2}h(0)[s,s]+R_{3}(s)\right\|^{2}+\|s\|^{2}. ( * ) We evaluate this expression term-by-term. First consider, \|s\|^{2}-\langle n,D^{2}h(0)[s,s]\rangle=\|s\|^{2}-\langle S_{n}s,s\rangle=\langle Q_{n}s,s\rangle. Next consider, \left\|\tfrac{1}{2}D^{2}h(0)[s,s]+R_{3}(s)\right\|^{2}=\left\|\tfrac{1}{2}D^{2}h(0)[s,s]\right\|^{2}+2\left\langle\tfrac{1}{2}D^{2}h(0)[s,s],R_{3}(s)\right\rangle+\|R_{3}(s)\|^{2}. Since \tfrac{1}{2}D^{2}h(0)[s,s] is quadratic in s and R_{3}(s) is cubic, therefore \langle\tfrac{1}{2}D^{2}h(0)[s,s],R_{3}(s)\rangle=O(\|s\|^{5}) . The other two contributions are O(\|s\|^{4}) and O(\|s\|^{6}) , respectively. Hence, \left\|\tfrac{1}{2}D^{2}h(0)[s,s]+R_{3}(s)\right\|^{2}=O(\|s\|^{4}) Lastly, since \|n\|\leq\mathsf{d} on K and R_{3}(s)=O(\|s\|^{3}) , we have \langle n,R_{3}(s)\rangle=O(\|s\|^{3}). Putting it all together, (*) evaluates to, \|x-y(s)\|^{2}=\mathsf{d}^{2}+\langle Q_{n}s,s\rangle+E(s),\quad\text{where}~E(s)=O(\|s\|^{3}). (17) Gaussian normalization on the chart. Introduce the Gaussian normalization Z_{Q_{n},t}:=\int_{\mathbb{R}^{m}}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,ds=(2\pi t)^{m/2}(\det Q_{n})^{-1/2}, (18) and the associated probability measure d\gamma_{Q_{n},t}(s):=\frac{1}{Z_{Q_{n},t}}\,e^{-\frac{1}{2}\langle Q_{n}s,s\rangle/t}\,ds. (19) Let S\sim\gamma_{Q_{n},t} . The covariance of \gamma_{Q_{n},t} is \mathrm{Cov}(S)=t\,Q_{n}^{-1}. (20) Moreover, for any fixed k\geq 1 , \mathbb{E}_{\gamma_{Q_{n},t}}\big[\|S\|^{k}\big]\leq C_{k}\,t^{k/2}, (21) where the constant C_{k} is uniform for x\in K , since Q_{n}\succeq\delta I uniformly. Asymptotics of D_{t} We first consider D_{t}^{\text{near}} , D_{t}^{\text{near}}=\int_{y(B(0,r_{0}))}e^{-\|x-z\|^{2}/(2t)}d\mu(z)=\int_{B(0,r_{0})}e^{-\|x-y(s)\|^{2}/(2t)}J(s)\,ds Using ( 17 ), \displaystyle D_{t}^{\text{near}}=e^{-\mathsf{d}^{2}/(2t)}\int_{B(0,r_{0})}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds We extend E(s) and J_{2}(s) to measurable functions on \mathbb{R}^{m} that agree with the original ones on B(0,r_{0}) and satisfy the growth bounds |E(s)|\leq C\|s\|^{3},\qquad|J_{2}(s)|\leq C\|s\|^{2},\qquad\forall s\in\mathbb{R}^{m} with C independent of x\in K . This guarantees integrability against the Gaussian weight. With these extensions, the integral over B(0,r_{0}) may be written as \displaystyle\begin{aligned} D_{t}^{\text{near}}&=e^{-\mathsf{d}^{2}/(2t)}\int_{\mathbb{R}^{m}}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds\\ &~-e^{-\mathsf{d}^{2}/(2t)}\int_{\mathbb{R}^{m}\setminus B(0,r_{0})}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds\end{aligned} (22) We rewrite the first integral above as \int_{\mathbb{R}^{m}}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds=Z_{Q_{n},t}\,\mathbb{E}_{\gamma_{Q_{n},t}}\left[e^{-E(S)/(2t)}\,(1+J_{2}(S))\right], (23) where S\sim\gamma_{Q_{n},t} . Using \exp({-E(S)/2t})=1+\frac{1}{2t}O(\|S\|^{3}) and J_{2}(S)=O(\|S\|^{2}) , e^{-E(S)/2t}(1+J_{2}(S))=1+\frac{1}{2t}O(\|S\|^{3})+O(\|S\|^{2})+\frac{1}{2t}O(\|S\|^{5}). (24) where we have neglected the higher-order mixed terms . Using ( 21 ), evaluating the expectation of these terms one-by-one, \displaystyle\mathbb{E}_{\gamma_{Q_{n},t}}\left[\frac{O(\|S\|^{3})}{2t}\right]=O\left(\frac{1}{2t}t^{3/2}\right)=O(t^{1/2}) \displaystyle\mathbb{E}_{\gamma_{Q_{n},t}}[O(\|S\|^{2})]=O(t) \displaystyle\mathbb{E}_{\gamma_{Q_{n},t}}\left[\frac{1}{2t}O(\|S\|^{5})\right]=\frac{1}{2t}O(t^{5/2})=O(t^{3/2}) Putting these together, we have \mathbb{E}_{\gamma_{Q_{n},t}}\left[e^{-E(S)/(2t)}\,(1+J_{2}(S))\right]=(1+O(t^{1/2})). (25) Hence, the first integral in D_{t}^{\text{near}} ( 22 ) evaluates to: e^{-\mathsf{d}^{2}/(2t)}\int_{\mathbb{R}^{m}}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds=Z_{Q_{n},t}\,e^{-\mathsf{d}^{2}/(2t)}(1+O(t^{1/2})) (26) Next, we compute the second integral in D_{t}^{\text{near}} ( 22 ), e^{-\mathsf{d}^{2}/(2t)}\int_{\mathbb{R}^{m}\setminus B(0,r_{0})}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds. Consider the integral only, \displaystyle\int_{\mathbb{R}^{m}\setminus B(0,r_{0})}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}e^{-E(s)/(2t)}(1+J_{2}(s))\,ds \displaystyle\hskip 28.45274pt=\int_{\mathbb{R}^{m}\setminus B(0,r_{0})}e^{\left(-\tfrac{1}{4t}\langle Q_{n}s,s\rangle\right)}e^{\left(-\tfrac{1}{4t}\langle Q_{n}s,s\rangle\right)}e^{-E(s)/(2t)}(1+J_{2}(s))\,ds Recall that Q_{n}\succeq\delta I uniformly for x\in K , and since T_{y^{*}}M\simeq\mathbb{R}^{m} , we have \langle Q_{n}s,s\rangle\geq\delta\|s\|^{2} for all s\in\mathbb{R}^{m} . In particular, for all \|s\|\geq r_{0} , \exp\left(-\tfrac{1}{4t}\langle Q_{n}s,s\rangle\right)\leq\exp\left(-\tfrac{\delta}{4t}\,\|s\|^{2}\right)\leq\exp\left(-\tfrac{\delta}{4t}\,r_{0}^{2}\right). Continuing the integral computation from above, \displaystyle\int_{\mathbb{R}^{m}\setminus B(0,r_{0})}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}e^{-E(s)/(2t)}(1+J_{2}(s))\,ds \displaystyle\hskip 56.9055pt\leq e^{-\frac{\delta r_{0}^{2}}{4t}}\int_{\mathbb{R}^{m}}e^{\left(-\tfrac{1}{4t}\langle Q_{n}s,s\rangle\right)}e^{-E(s)/(2t)}(1+J_{2}(s))\,ds \displaystyle\hskip 56.9055pt=2^{m/2}Z_{Q_{n},t}e^{-\frac{\delta r_{0}^{2}}{4t}}\left(1+O(t^{1/2})\right),~~\text{(using \eqref{eq:temp1}, \eqref{eq:temp})} where the constant 2^{m/2} is due to Z_{Q_{n},2t}=2^{m/2}Z_{Q_{n},t} . In summary, the second integral in D_{t}^{\text{near}} in ( 22 ) can be bounded as \displaystyle\begin{aligned} &e^{-\mathsf{d}^{2}/(2t)}\int_{\mathbb{R}^{m}\setminus B(0,r_{0})}e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds\\ &\hskip 56.9055pt\leq\,2^{m/2}Z_{Q_{n},t}\,e^{-\mathsf{d}^{2}/(2t)}\,e^{-\delta r_{0}^{2}/(4t)}\,\left(1+O(t^{1/2})\right).\end{aligned} (27) Lastly, we recall that D_{t}^{\text{far}} , ( 15 ), satisfies |D_{t}^{\text{far}}|\leq\exp\left(-\frac{\mathsf{d}^{2}}{2t}\right)\,O\left(\exp\left(-\frac{c}{t}\right)\right). Putting together ( 26 ), the tail bound ( 27 ), and the far-region estimate ( 15 ), we obtain \displaystyle D_{t}(x)=Z_{Q_{n},t}\,e^{-\mathsf{d}^{2}/(2t)}\left(1+O(t^{1/2})\right)+2^{m/2}Z_{Q_{n},t}\,e^{-\mathsf{d}^{2}/(2t)}\,e^{-\delta r_{0}^{2}/(4t)}+e^{-\mathsf{d}^{2}/(2t)}\,O\!\left(e^{-c/t}\right), uniformly for x\in K and t\downarrow 0 . Absorbing the latter two exponentially small remainder into the error term yields \displaystyle D_{t}(x)=e^{-\mathsf{d}^{2}/(2t)}(2\pi t)^{m/2}(\det Q_{n})^{-1/2}\left(1+O\left(t^{1/2}\right)\right). (28) Asymptotics of A_{t}(x) Similar to D_{t}^{\text{near}} , we first consider A_{t}^{\text{near}} , A_{t}^{\text{near}}=\int_{y(B(0,r_{0}))}(x-z)\,e^{-\|x-z\|^{2}/(2t)}d\mu(z)=\int_{B(0,r_{0})}(x-y(s))\,e^{-\|x-y(s)\|^{2}/(2t)}J(s)\,ds Using ( 17 ), \displaystyle A_{t}^{\text{near}}=e^{-\mathsf{d}^{2}/(2t)}\int_{B(0,r_{0})}(x-y(s))\,e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds Similar to D_{t}^{near} , by extending E(s) and J_{2}(s) to measurable functions on \mathbb{R}^{m} , we split the domain of integration. \displaystyle\begin{aligned} A_{t}^{\text{near}}&=e^{-\mathsf{d}^{2}/(2t)}\int_{\mathbb{R}^{m}}(x-y(s))\,e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds\\ &~-e^{-\mathsf{d}^{2}/(2t)}\int_{\mathbb{R}^{m}\setminus B(0,r_{0})}(x-y(s))\,e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds.\end{aligned} (29) Rewrite ( 16 ) as x-y(s)=n-s-\tfrac{1}{2}D^{2}h(0)[s,s]-R_{3}(s)=n+G(s) where G(s):=-s-\tfrac{1}{2}D^{2}h(0)[s,s]-R_{3}(s) . Consider the first integral in ( 29 ), \displaystyle e^{-\mathsf{d}^{2}/(2t)}\int_{\mathbb{R}^{m}}(x-y(s))\,e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-R(s)/(2t)}\,(1+J_{2}(s))\,ds=Z_{Q_{n},t}\,\mathbb{E}_{\gamma_{Q_{n},t}}\left[(n+G(S))\,e^{-E(s)/(2t)}(1+J_{2}(S))\right] where S\sim\gamma_{Q_{n},t} . Computing the expectation, \displaystyle\mathbb{E}[(n+G(S)) \displaystyle e^{-E(s)/(2t)}(1+J_{2}(S))] \displaystyle=\mathbb{E}\left[\left(n-s-\tfrac{1}{2}D^{2}h(0)[s,s]-R_{3}(s)\right)\left(1+\frac{1}{2t}O(\|S\|^{3})+O(\|S\|^{2})\right)\right] \displaystyle=\mathbb{E}\left[n\left(1+\frac{1}{2t}O(\|S\|^{3})+O(\|S\|^{2})\right)\right]-\mathbb{E}\left[s\left(1+\frac{1}{2t}O(\|S\|^{3})+O(\|S\|^{2})\right)\right] \displaystyle-\tfrac{1}{2}\mathbb{E}\left[D^{2}h(0)[s,s]\left(1+\frac{1}{2t}O(\|S\|^{3})+O(\|S\|^{2})\right)\right] From ( 25 ), the first term evaluates to n(1+O(t^{1/2})) . The second term is bounded by \displaystyle\mathbb{E}_{\gamma_{Q_{n},t}}\left[s\left(1+\frac{1}{2t}O(\|S\|^{3})+O(\|S\|^{2})\right)\right] \displaystyle\leq\mathbb{E}_{\gamma_{Q_{n},t}}[S]+\frac{1}{2t}\mathbb{E}_{\gamma_{Q_{n},t}}[O(\|S\|^{4})+\mathbb{E}_{\gamma_{Q_{n},t}}(O(\|S\|^{3}))=O(t) where we have used the fact that \gamma_{Q_{n},t} is centered, hence \mathbb{E}_{\gamma_{Q_{n},t}}[S]=0 . Similarly, we bound the third term using the fact that \|D^{2}h(0)\|\leq C , and hence \tfrac{1}{2}\mathbb{E}_{\gamma_{Q_{n},t}}\left[D^{2}h(0)[s,s]\left(1+\frac{1}{2t}O(\|S\|^{3})+O(\|S\|^{2})\right)\right]\leq\tfrac{1}{2}\|D^{2}h(0)\|\,\mathbb{E}_{\gamma_{Q_{n},t}}(\|S\|^{2}+\frac{1}{2t}O(\|S\|^{5})+O(\|S\|^{4})\leq O(t) Therefore, we obtain \displaystyle Z_{Q_{n},t}\,\mathbb{E}_{\gamma_{Q_{n},t}}\left[(n+G(S))\,e^{-E(s)/(2t)}(1+J_{2}(S))\right]=Z_{Q_{n},t}(n+O(t^{1/2})). (30) Consider the second integral in ( 29 ). Similar to ( 27 ), this term can be bounded, using the expression ( 16 ) as \displaystyle\begin{aligned} -e^{-d^{2}/(2t)}\int_{\mathbb{R}^{m}\setminus B(0,r_{0})}(x-y(s))\,e^{\left(-\tfrac{1}{2t}\langle Q_{n}s,s\rangle\right)}\,e^{-E(s)/(2t)}\,(1+J_{2}(s))\,ds.\\ \leq 2^{m/2}Z_{Q_{n},t}\,e^{-\mathsf{d}^{2}/(2t)}\,e^{-\delta r_{0}^{2}/(4t)}\,\left(1+O(t^{1/2})\right)\end{aligned} (31) Finally, we note that A_{t}^{\text{far}} , ( 15 ), satisfies \|A_{t}^{\text{far}}\|\leq\exp\left(-\frac{\mathsf{d}^{2}}{2t}\right)\,O\left(\exp\left(-\frac{c}{t}\right)\right). Putting together ( 30 ), ( 31 ), and ( 15 ) to conclude \displaystyle A_{t}(x) \displaystyle=A_{t}^{\text{near}}+A_{t}^{\text{far}} \displaystyle=Z_{Q_{n},t}\,e^{-\mathsf{d}^{2}/(2t)}\,\left(n+O(t^{1/2})\right)+2^{m/2}Z_{Q_{n},t}\,e^{-\mathsf{d}^{2}/(2t)}\,e^{-\delta r_{0}^{2}/(4t)}+e^{-\mathsf{d}^{2}/2t}\,O\left(e^{-\frac{c}{t}}\right). Again, keeping only the lower order terms we get, A_{t}(x)=e^{-\mathsf{d}^{2}/(2t)}(2\pi t)^{m/2}(\det Q_{n})^{-1/2}\left(n+O(t^{1/2})\right), (32) uniformly for x\in K as t\downarrow 0 . Ratio A_{t}(x)/D_{t}(x) and conclusion. Equations ( 28 ) and ( 32 ) hold uniformly for x\in K as t\downarrow 0 . Computing \frac{A_{t}(x)}{D_{t}(x)} yields, \frac{A_{t}(x)}{D_{t}(x)}=\frac{n+O\left(t^{1/2}\right)}{1+O\left(t^{1/2}\right)}=n+O\left(t^{1/2}\right). Recalling that n=x-y^{*}=x-P_{M}(x) , we conclude \nabla_{x}\log u_{t}(x)=-\frac{1}{t}\frac{A_{t}(x)}{D_{t}(x)}=-\frac{x-P_{M}(x)}{t}+O\left(t^{-1/2}\right), Rearranging, this is equivalent to x+t\,\nabla_{x}\log u_{t}(x)=x-\frac{A_{t}(x)}{D_{t}(x)}=x-n+O\left(t^{1/2}\right)=P_{M}(x)+O\left(t^{1/2}\right), that is, \sup_{x\in K}\left\|x+t\,\nabla_{x}\log u_{t}(x)-P_{M}(x)\right\|\;\leq\;C_{K}\,t^{1/2} for all sufficiently small t>0 . This establishes the convergence uniformly on K as t\downarrow 0 , x+t\,\nabla_{x}\log u_{t}(x)\longrightarrow P_{M}(x)\quad\text{with rate }O\left(t^{1/2}\right).\qed Appendix C Example Details C.1 Example 1: The Special Orthogonal Group SO(3) Its tangent space at the identity I is isomorphic to its Lie algebra \mathfrak{so}(\cdot) . T_{I}\mathrm{SO}(d)=\mathfrak{so}(d)=\{A\in\mathbb{R}^{d\times d}:A^{\top}=-A\}. For d=3 , a convenient basis for \mathfrak{so}(3) is: E_{1}=\begin{bmatrix}0&0&0\\ 0&0&-1\\ 0&1&0\end{bmatrix},~E_{2}=\begin{bmatrix}0&0&1\\ 0&0&0\\ -1&0&0\end{bmatrix},~E_{3}=\begin{bmatrix}0&-1&0\\ 1&0&0\\ 0&0&0\end{bmatrix}. To generate trajectories on \mathrm{SO}(3) , we integrate a matrix ODE whose velocity is a (state-dependent) scalar multiple of a left-invariant vector field. We define g(X)\;=\;\sum_{i=1}^{3}E_{i}X,\qquad s(X)\;=\;\mathrm{tr}(X^{2}+I), and evolve \frac{dX}{dt}\;=\;s(X)\,g(X)\;=\;\mathrm{tr}(X^{2}+I)\left(\sum_{i=1}^{3}E_{i}X\right). Since each E_{i} is skew-symmetric, \sum_{i}E_{i}\in\mathfrak{so}(3) and thus g(X)\in T_{X}\mathrm{SO}(3)=\{AX:A^{\top}=-A\} ; multiplying by the scalar s(X) preserves tangency. Therefore, starting from X(0)\in\mathrm{SO}(3) , the flow remains on \mathrm{SO}(3) (up to numerical error). Exponential If a network produces coefficients c=(c_{1},c_{2},c_{3})\in\mathbb{R}^{3} , define \xi=\sum_{i=1}^{3}c_{i}E_{i}\in\mathfrak{so}(3) . A group update with step \Delta t is g(n+1)=\exp(\Delta t\,\xi)\,g(n),\qquad g\in\mathrm{SO}(3). Projection Given Y\in\mathbb{R}^{d\times d} , the projection of Y in the Frobenius norm to the closest rotation matrix is the (proper) polar/SVD projection: Y=U\Sigma V^{\top},\qquad P_{\mathrm{SO}(d)}(Y)=\det(Y)\,UV^{\top}. This is not an exact projection everywhere in space, but only when close to the manifold. C.2 Example 2: The Sphere The sphere is set of all points in \mathbb{R}^{d} with unit \ell_{2} norm. We generate synthetic trajectories on the unit sphere by advecting points under a smooth, time-independent tangent vector field. Let p(t)\in\mathbb{R}^{3} with \|p(t)\|=1 and write spherical angles (\theta(p),\phi(p)) via the standard parametrization r(\theta,\phi)=(\sin\theta\cos\phi,\sin\theta\sin\phi,\cos\theta) . On a (\theta,\phi) grid we construct a smooth random field by sampling scalar coefficient fields a(\theta,\phi),b(\theta,\phi) (low-frequency trigonometric mixtures) and setting V(\theta,\phi)=a(\theta,\phi)\,\partial_{\theta}r(\theta,\phi)+b(\theta,\phi)\,\partial_{\phi}r(\theta,\phi),\qquad\widehat{V}(\theta,\phi)=\frac{V(\theta,\phi)}{\|V(\theta,\phi)\|}, then storing its Cartesian components (U,V,W) on the grid (with periodicity in \phi ). Given p\in\mathbb{S}^{2} , we define the ambient velocity by bilinear interpolation \widetilde{V}(p)=\mathrm{Interp}_{(\theta,\phi)}(U,V,W)\in\mathbb{R}^{3} and project to the tangent space \dot{p}\;=\;\Pi_{T_{p}\mathbb{S}^{2}}\,\widetilde{V}(p),\qquad\Pi_{T_{p}\mathbb{S}^{2}}(v)=v-(v^{\top}p)\,p. We integrate this ODE using an explicit midpoint (RK2) step in \mathbb{R}^{3} together with the sphere retraction R(x)=x/\|x\| at the midpoint and at the end of the step: p_{\mathrm{mid}}=R\!\left(p+\tfrac{\Delta t}{2}\,\Pi_{T_{p}\mathbb{S}^{2}}\widetilde{V}(p)\right),\qquad p^{+}=R\!\left(p+\Delta t\,\Pi_{T_{p_{\mathrm{mid}}}\mathbb{S}^{2}}\widetilde{V}(p_{\mathrm{mid}})\right). Training pairs (x_{0},x_{T}) are obtained by sampling x_{0}\sim\mathrm{Unif}(\mathbb{S}^{2}) and evolving for T steps. Exponential The tangent space at x is defined as T_{x}\mathbb{S}^{d}\;=\;\{\,v\in\mathbb{R}^{d+1}:\langle x,v\rangle=0\,\}. The orthogonal projection onto T_{x}\mathbb{S}^{d} is given by P_{x}(u)\;=\;u-\langle u,x\rangle\,x. The exponential map at x is defined as \exp_{x}(v)\;=\;\begin{cases}\cos(\|v\|_{2})\,x\;+\;\sin(\|v\|_{2})\,\dfrac{v}{\|v\|_{2}},&v\neq 0\\ x,&v=0\end{cases}, v\in T_{x}\mathbb{S}^{d} . Accordingly, the discrete-time update is \displaystyle u(n)=f_{\theta}(x(n)), \displaystyle v(n)=P_{x(n)}(u(n))=u(n)-\langle u(n),x(n)\rangle x(n), \displaystyle x(n+1)=\exp_{x(n)}(\Delta t\,v(n)). Projection Projection onto S^{d} of an unconstrained y\in\mathbb{R}^{d+1} is P_{\mathbb{S}^{d}}(y)=\frac{y}{\|y\|_{2}}\quad(y\neq 0). C.3 Example 3: Manifold with boundary, closed unit disk For x\in\mathbb{R}^{2} , we let the vector field be F(x_{1},x_{2})=\underbrace{\begin{bmatrix}0&-1\\ 1&0\end{bmatrix}}_{J}\begin{bmatrix}x_{1}\\ x_{2}\end{bmatrix}+\alpha\begin{bmatrix}x_{1}\\ x_{2}\end{bmatrix}=\begin{bmatrix}-x_{2}+\alpha x_{1}\\ x_{1}+\alpha x_{2}\end{bmatrix}, where \alpha>0 . Since the set \|x\|=1 is invariant for the dynamics when \alpha=0 , the projected dynamical system in this case can be summarized as \dot{x}=\begin{cases}Jx+\alpha x,&\|x\|<1,\\ Jx,&\|x\|=1.\end{cases} Exponential We do not have an exponential map for this manifold. Projection For points outside the disk we project onto the boundary by normalizing. C.4 Example 4: Cucker-Smale Dynamics on SO(3) The data are generated from a geometric Cucker–Smale model describing collective alignment of rigid-body orientations ( 16 ) . Let R_{i}(t)\in SO(3) denotes the orientation of agent i , and A_{i}(t)\in\mathfrak{so}(3) is its body angular velocity, identified with a vector a_{i}(t)\in\mathbb{R}^{3} via the hat map (a linear operator that maps a vector to its corresponding skew-symmetric matrix in the Lie algebra). The quantity \theta_{ki}\in[0,\pi) represents the geodesic distance between R_{k} and R_{i} on SO(3) , defined by \theta_{ki}=\|\log(R_{k}^{\top}R_{i})\|, and n_{ki}\in\mathbb{S}^{2} is the associated unit rotation axis. The communication weight \phi_{ik}=\phi(\theta_{ki}) is a nonnegative function of the geodesic distance, typically chosen to vanish at the cut locus to ensure well-posedness. The resulting dynamics are given by \displaystyle\dot{R}_{i} \displaystyle=R_{i}A_{i}, \displaystyle\dot{a}_{i} \displaystyle=\frac{\kappa}{N}\sum_{k=1}^{N}\phi_{ik}\left[\left(1-\cos\frac{\theta_{ki}}{2}\right)(n_{ki}\!\cdot\!a_{k})n_{ki}+\sin\frac{\theta_{ki}}{2}\,(a_{k}\times n_{ki})+\cos\frac{\theta_{ki}}{2}\,a_{k}-a_{i}\right]. The communication weight used in the numerical simulations is given by \phi(\theta_{ki})=\cos\!\left(\theta_{ki}/2\right) for \theta_{ki}<\pi , otherwise 0. C.5 Example 5. Protein Dataset on SE(3) We use the ProteinNet “Tertiary structure” field, which provides for each residue i (amino acid in the protein chain) the Cartesian coordinates of the backbone atoms (N_{i},\ C_{\alpha}{}_{i},\ C_{i}) (in picometers). Following AlphaFold’s construction of backbone frames (Supplementary Information, §1.8.1, Algorithm 21) ( 22 ) , we associate to each residue a rigid transform T_{i}=(R_{i},t_{i})\in SE(3) as follows. Let x_{N},x_{C_{\alpha}},x_{C}\in\mathbb{R}^{3} denote the three backbone atom positions of the residue. Define v_{1}=x_{C}-x_{C_{\alpha}},~v_{2}=x_{N}-x_{C_{\alpha}}, and perform a Gram–Schmidt orthonormalization in the N\!-\!C_{\alpha}\!-\!C triad: e_{1}=\frac{v_{1}}{\|v_{1}\|},\quad\tilde{e}_{2}=v_{2}-(e_{1}^{\top}v_{2})\,e_{1},\quad e_{2}=\frac{\tilde{e}_{2}}{\|\tilde{e}_{2}\|},\quad e_{3}=e_{1}\times e_{2}. Set the rotation R_{i}=[\,e_{1}~e_{2}~e_{3}\,]\in SO(3) (flip e_{3} if needed so that \det R_{i}=+1 as in Alg. 21), and the translation t_{i}=x_{C_{\alpha}} . Packing into homogeneous coordinates yields the per-residue frame T_{i}\;=\;\begin{bmatrix}R_{i}&t_{i}\\ 0~0~0&1\end{bmatrix}\in SE(3),\qquad x_{\mathrm{global}}\;=\;R_{i}\,x_{\mathrm{local}}+t_{i}. ProteinNet supplies a per-residue mask; when any backbone atom is missing we treat T_{i} as undefined and exclude pairs that touch masked residues. For learning, we predict T_{i+1} given T_{i} . The Special Euclidean Group \mathrm{SE}(3) . The special Euclidean group is the Lie group of rigid motions in \mathbb{R}^{3} , \mathrm{SE}(3)=\left\{\begin{bmatrix}R&t\\ 0&1\end{bmatrix}:R\in\mathrm{SO}(3),~t\in\mathbb{R}^{3}\right\}, with identity I=\begin{bmatrix}I_{3}&0\\ 0&1\end{bmatrix} . Its tangent space at the identity is isomorphic to the Lie algebra T_{I}\mathrm{SE}(3)=\mathfrak{se}(3)=\left\{\begin{bmatrix}M&v\\ 0&0\end{bmatrix}:M\in\mathfrak{so}(3),~v\in\mathbb{R}^{3}\right\}. A convenient basis is given by three rotational generators \{E_{i}\}_{i=1}^{3}\subset\mathfrak{so}(3) (as above, embedded in the top-left block) and three translational generators \{T_{i}\}_{i=1}^{3} , \widehat{E}_{i}=\begin{bmatrix}E_{i}&0\\ 0&0\end{bmatrix},\qquad T_{1}=\begin{bmatrix}0&0&0&1\\ 0&0&0&0\\ 0&0&0&0\\ 0&0&0&0\end{bmatrix},\ T_{2}=\begin{bmatrix}0&0&0&0\\ 0&0&0&1\\ 0&0&0&0\\ 0&0&0&0\end{bmatrix},\ T_{3}=\begin{bmatrix}0&0&0&0\\ 0&0&0&0\\ 0&0&0&1\\ 0&0&0&0\end{bmatrix}. Thus, if a network outputs coefficients c\in\mathbb{R}^{3} (rotation) and u\in\mathbb{R}^{3} (translation), we form \xi=\sum_{i=1}^{3}c_{i}\,\widehat{E}_{i}+\sum_{i=1}^{3}u_{i}\,T_{i}\ \in\ \mathfrak{se}(3). Exponential. A group update with step \Delta t is obtained via the matrix exponential g(n+1)=\exp(\Delta t\,\xi)\,g(n),\qquad g\in\mathrm{SE}(3). Projection. Given a near-rigid transform Y=\begin{bmatrix}A&b\\ 0&1\end{bmatrix}\in\mathbb{R}^{4\times 4} , we project its rotational block to \mathrm{SO}(3) by the proper polar/SVD map: A=U\Sigma V^{\top},\qquad P_{\mathrm{SO}(3)}(A)=\det(UV^{\top})\,UV^{\top}, and define the \mathrm{SE}(3) projection by keeping translation unchanged, P_{\mathrm{SE}(3)}(Y)=\begin{bmatrix}P_{\mathrm{SO}(3)}(A)&b\\ 0&1\end{bmatrix}. As with \mathrm{SO}(3) , this acts as a reliable correction when Y is sufficiently close to the manifold. Appendix D Flow-matching Learned Projection: Implementation and Hyperparameters This appendix records the algorithm, exact velocity-network architecture, synthetic data generation, optimization loop, sweep grid, and saved artifacts used to reproduce the learned projections. D.1 Algorithm Algorithm 1 Approximate Projection onto M via Flow Matching 1: Input: Distribution \mu supported on M\subset\mathbb{R}^{d} 2: Output: Approximate projection map P_{M}(x_{0}) 3: Generate samples X_{1},\dots,X_{N}\sim\mu 4: Sample velocities v_{n}\sim\mathcal{N}(m,\Sigma) 5: Define perturbed paths Y^{t}_{n}=X_{n}+v_{n}t , for t\in[0,T] 6: Train v_{\theta}(x,t) by minimizing \min_{\theta}\frac{1}{N}\sum_{n=1}^{N}\int_{0}^{T}\|v_{\theta}(Y^{t}_{n},t)-v_{n}\|^{2}\,dt 7: Approximate the projection P_{M}(x_{0}) by integrating the backward flow: \frac{d}{ds}x(s)=-v_{\theta}(t,x(s)),\qquad x(0)=x_{0} D.2 Model architecture The velocity model v_{\theta}:\mathbb{R}^{d}\times[0,T]\to\mathbb{R}^{d} is implemented by FlowVelocityNet . Inputs are formed by concatenation [x;t]\in\mathbb{R}^{d+1} and mapped to \mathbb{R}^{d} by an MLP with hidden width 256 and 8 hidden blocks: • Input block: \mathrm{Linear}(d{+}1\to 256)\rightarrow\mathrm{LayerNorm}(256)\rightarrow\mathrm{GELU}\rightarrow\mathrm{Dropout}(0.1) . • Hidden blocks (repeated 8 times): \mathrm{Linear}(256\to 256)\rightarrow\mathrm{LayerNorm}(256)\rightarrow\mathrm{GELU}\rightarrow\mathrm{Dropout}(0.1) . • Output layer: \mathrm{Linear}(256\to d) . LayerNorm is applied after each linear layer and before GELU. The dimension d is inferred from the dataset tensor after flattening any sequence axis (below). D.3 Synthetic flow-matching dataset generation From initial points X=\{x_{n}\}_{n=1}^{N}\subset\mathbb{R}^{d} , we create supervised pairs by velocity advection: 1. Sample i.i.d. u_{n}\sim\mathcal{N}(0,I_{d}) . 2. Normalize and scale: v_{n}\;=\;0.5\cdot\frac{u_{n}}{\|u_{n}\|_{2}+10^{-8}}. 3. Choose the time horizon T either as a fixed value or via auto : T\;=\;2\alpha\cdot\mathrm{median}_{n}\|x_{n}\|_{2},\qquad\alpha\in\{0.25,0.5,1.0\}, computed on the training split. Construct a uniform time grid of 30 points on [0,T] . 4. Form y_{n,k}=x_{n}+v_{n}t_{k} and flatten pairs to inputs z_{n,k}=[y_{n,k};t_{k}]\in\mathbb{R}^{d+1} and targets v_{n}\in\mathbb{R}^{d} (repeated across k ). This yields (30N) pairs. The validation synthetic set is constructed once from X_val and held fixed. The training synthetic set is regenerated once per epoch from X_train . D.4 Training objective and optimization We minimize mean-squared error between predicted and target velocities. Optimization uses AdamW with batch size 256 , global gradient-norm clipping at 1.0 , and seed 0 (NumPy and PyTorch). We apply a ReduceLROnPlateau schedule on validation loss with factor 0.5 and patience 100 epochs, and select the best checkpoint by lowest validation loss. Training is configured for up to 2000 epochs with early stopping patience 1000 epochs (except cs : 200 epochs, scheduler patience 10 , early stopping patience 10 ). Learning rate and weight decay are swept as described below. D.5 Sweep grid and run organization For each dataset, we train the Cartesian product \alpha\in\{0.25,0.5,1.0\},\quad\mathrm{lr}\in\{3\times 10^{-4},10^{-3},3\times 10^{-3}\},\quad\mathrm{wd}\in\{0,10^{-4}\}. D.6 Projection operator implementation Given a trained v_{\theta} , the learned projection integrates \frac{d}{dt}x(t)=-v_{\theta}(x(t),t),\qquad t\in[0,T], from t=T to t=0 and returns x(0) . A differentiable implementation uses explicit Euler with a user-specified number of steps. For evaluation-time projection, we use RK45 via SciPy solve_ivp with tolerances rtol =10^{-6} and atol =10^{-8} . D.7 Manifold distances / constraint violations We evaluate constraint satisfaction using explicit, dataset-dependent residuals. Below, all norms are computed per-sample and then aggregated by mean (and, when recorded, max) over the evaluated batch. Sphere dataset ( \Omega=S^{2}\subset\mathbb{R}^{3} ). For x\in\mathbb{R}^{3} , d_{\mathrm{sphere}}(x,\Omega)\;:=\;\bigl|\|x\|_{2}-1\bigr|. (33) Disk dataset ( \Omega=\{x\in\mathbb{R}^{2}:\|x\|_{2}\leq 1\} ). For x\in\mathbb{R}^{2} , d_{\mathrm{disk}}(x,\Omega)\;:=\;\max\{0,\ \|x\|_{2}-1\}. (34) \mathrm{SO}(3) dataset (and cs dataset; \Omega=\mathrm{SO}(3) represented in \mathbb{R}^{9} ). Each prediction is a row-major vector r\in\mathbb{R}^{9} reshaped to a matrix R\in\mathbb{R}^{3\times 3} . We record: \displaystyle d_{\mathrm{orth}}(R) \displaystyle:=\|RR^{\top}-I_{3}\|_{F}, (35) \displaystyle d_{\det}(R) \displaystyle:=\bigl|\det(R)-1\bigr|, (36) \displaystyle d_{\mathrm{SO(3)}}(R) \displaystyle:=d_{\mathrm{orth}}(R)+d_{\det}(R). (37) The “distance to manifold” curve for \mathrm{SO}(3) and cs uses the mean of d_{\mathrm{SO(3)}} . Protein dataset ( \Omega=\mathrm{SE}(3) represented in \mathbb{R}^{16} ). Each prediction is a row-major vector g\in\mathbb{R}^{16} reshaped to G\in\mathbb{R}^{4\times 4} , with R:=G_{1:3,1:3}\in\mathbb{R}^{3\times 3} and last row \ell^{\top}:=G_{4,:}\in\mathbb{R}^{4} . We record: \displaystyle d_{\mathrm{orth}}(G) \displaystyle:=\|RR^{\top}-I_{3}\|_{F}, (38) \displaystyle d_{\det}(G) \displaystyle:=\bigl|\det(R)-1\bigr|, (39) \displaystyle d_{\mathrm{row}}(G) \displaystyle:=\|\ell-[0,0,0,1]\|_{\infty}, (40) \displaystyle d_{\mathrm{SE(3)}}(G) \displaystyle:=d_{\mathrm{orth}}(G)+d_{\det}(G)+d_{\mathrm{row}}(G). (41) The “distance to manifold” curve for protein uses the mean of d_{\mathrm{SE(3)}} . Which distance is plotted. For sphere and disk we plot \mathbb{E}[d_{\mathrm{sphere}}] and \mathbb{E}[d_{\mathrm{disk}}] , respectively. For \mathrm{SO}(3) , cs, and protein we plot the mean of the corresponding summed residuals, i.e., \mathbb{E}[d_{\mathrm{SO(3)}}] for \mathrm{SO}(3) /cs and \mathbb{E}[d_{\mathrm{SE(3)}}] for protein. Appendix E Training protocol and model selection for manifold learning experiments E.1 Model architecture (Feedforward, residual) We implement the feedforward baseline using RegularFeedForward , which maps x\in\mathbb{R}^{d} (or x\in\mathbb{R}^{B\times S\times d} pointwise over the leading axes) to an output of the same shape. The network is a stack of L identical blocks with a residual update and a learnable per-layer step size. • FF block. Each block F:\mathbb{R}^{d}\to\mathbb{R}^{d} is \mathrm{Linear}(d\!\to\!h)\rightarrow\mathrm{ReLU}\rightarrow\mathrm{Dropout}(p)\rightarrow\mathrm{Linear}(h\!\to\!h), implemented as FFBlock . In our instantiations we take h=d so that the residual update is dimensionally consistent. • Stacking and residual connection. Let x_{0}=x . For layers i=1,\dots,L : y_{i}=F_{i}(x_{i-1}),\qquad x_{i}=x_{i-1}+\Delta t_{i}\,y_{i}, where \Delta t_{i} is a learnable scalar (initialized to a common value). The output is x_{L} . E.2 Model architecture (Transformer, residual) We implement the transformer baseline using RegularTransformer , a standard transformer encoder that operates on sequences x\in\mathbb{R}^{B\times S\times d} with feature dimension d . The model consists of L encoder layers (with n_{\mathrm{head}} heads and feedforward width d_{\mathrm{hid}} ) and a final linear readout: • Input formatting: x is permuted to \mathbb{R}^{S\times B\times d} before entering the encoder stack. • Encoder stack (repeated L times): each layer is a PyTorch TransformerEncoderLayer with parameters \texttt{TransformerEncoderLayer}(d,\ n_{\mathrm{head}},\ d_{\mathrm{hid}},\ \texttt{dropout}), followed by a pointwise ReLU and a residual update with learnable per-layer scale \Delta t_{i} : x\leftarrow x_{\mathrm{in}}+\Delta t_{i}\,\mathrm{ReLU}(\mathrm{EncLayer}_{i}(x_{\mathrm{in}})),\qquad i=1,\dots,L, where \{\Delta t_{i}\}_{i=1}^{L} are learned parameters initialized to a common value.:contentReference[oaicite:3]index=3 • Output layer: after permuting back to \mathbb{R}^{B\times S\times d} , a final linear map \mathrm{Linear}(d\to d) produces the output. E.3 Projected vs. Exponential models (IAA vs. FAA) We enforce manifold structure in two ways: (i) projection in the ambient representation, and (ii) exponential updates via a user-supplied \exp -map hook. Both mechanisms are implemented in Transformer and FeedForward variants and expose a switch between applying the geometric map internally (IAA) or only at the end (FAA). Projected models (ambient projection). ProjectedTransformer and ProjectedFeedForward keep the state in the ambient coordinates (same feature dimension in/out) and apply a projection hook \Pi(\cdot) supplied as proj_func . • IAA (internal projection). After each layer/block update, apply \Pi before the next layer: x_{i+1}\;=\;\Pi\!\big(x_{i}+\Delta t_{i}\,f_{i}(x_{i})\big),\qquad i=0,\dots,L-2, and then also apply a final projection to the output. This corresponds to use_internal_projection=True , which calls internal_proj_func between layers/blocks and end_proj_func / final_proj_func at the end. • FAA (final projection only). Skip the internal projections and project only once at the end: x_{i+1}\;=\;x_{i}+\Delta t_{i}\,f_{i}(x_{i}),\qquad\hat{x}\;=\;\Pi(x_{L}). This corresponds to use_internal_projection=False while keeping the mandatory end projection. Exponential models (exp-map update). ExponentialTransformer and ExponentialFeedForward implement a Lie-type update using a callable exp_func passed via internal_exp_func / end_exp_func (transformer) or exp_func (FF). In particular, the transformer produces an m -dimensional output (manifold dimension) via a linear head \mathbb{R}^{\texttt{input\_dim}}\to\mathbb{R}^{m} . • IAA (internal exp). Apply the exp-map between layers/blocks: g_{i+1}\;=\;\mathrm{Exp}\!\big(\Delta t_{i}\,\xi_{i}\big)\,\cdot\,g_{i}, where \xi_{i} is predicted by the network at layer i . In code this is use_internal_exp=True (transformer) / use_internal_exponential=True (FF), invoking internal_exp_func between layers and still applying end_exp_func at the end. • FAA (final exp only). Skip the internal exp-map and apply a single exp-map at the end using the initial state as basepoint: \hat{g}\;=\;\mathrm{Exp}\!\big(\xi\big)\,\cdot\,g_{0}, implemented by setting use_internal_exp=False / use_internal_exponential=False . In this mode, the end exp hook is applied with basepoint \texttt{g}_{0} (the original input) rather than the internally-updated state. E.4 Probabilistic model (anchor-based output) We implement a probabilistic predictor by discretizing the output space with N anchors (a.k.a. particles) \{Y^{(n)}\}_{n=1}^{N}\subset\mathbb{R}^{d} (or flattened \mathbb{R}^{9} for \mathrm{SO}(3) / \mathbb{R}^{16} for \mathrm{SE}(3) ). Anchors are chosen as a subset of the training targets Y_{\mathrm{train}} (default in our experiments) or by synthetic sampling on the manifold (sphere / \mathrm{SO}(3) / \mathrm{SE}(3) , with bounded translations for \mathrm{SE}(3) ). Voronoi labels. Given a training pair (x_{t},y_{t}) , we assign a discrete label by nearest-anchor (Voronoi partitioning) \ell_{t}\;=\;\arg\min_{n\in[N]}\|y_{t}-Y^{(n)}\|_{2}, implemented via a batched distance matrix and \arg\min over anchors. Network output. The model backbone is the same as the regular feedforward / transformer, but its final linear layer is replaced to output logits in \mathbb{R}^{N} (one score per anchor). Training loss. Let z_{\theta}(x_{t})\in\mathbb{R}^{N} be logits and p_{\theta}(x_{t})=\mathrm{softmax}(z_{\theta}(x_{t}))\in\Delta^{N-1} . We train with squared error on the simplex against the one-hot label: \mathcal{L}(\theta)\;=\;\frac{1}{B}\sum_{t=1}^{B}\bigl\|p_{\theta}(x_{t})-e_{\ell_{t}}\bigr\|_{2}^{2}, implemented as sum of squared errors over anchors (then averaged over the batch). Inference. Given p_{\theta}(x) , we produce a continuous prediction either by expectation (weighted average) \hat{y}\;=\;\sum_{n=1}^{N}p_{\theta,n}(x)\,Y^{(n)}\quad\text{(default)}, or by \arg\max (snap to the most likely anchor), \hat{y}=Y^{(\arg\max_{n}p_{\theta,n}(x))} . Implementation notes. For the probabilistic mode, the training loader yields (x,\ell) while validation compares \hat{y} to the true target via MSE. For sequential data (e.g. cs ), labels are formed after flattening the sequence axis, and transformer models use the last-timestep logits during training. E.5 Shared hyperparameter sweep For every dataset–model-family combination, we run a small hyperparameter sweep: • Depth: D\in\{4,6,8\} . • Weight decay: \lambda\in\{0,\ 1e-4\} Each sweep run is trained independently. E.6 Optimization and stopping criteria All models are trained for up to 10{,}000 epochs using AdamW with initial learning rate \eta_{0}=1e-3 and batch size B=500 . We reduce the learning rate using a validation-driven plateau schedule (ReduceLROnPlateau) with factor \gamma=0.5 and patience p=1000 epochs. We checkpoint the model with the best validation loss (lowest value) across the full training run. E.7 Validation-based model selection For each dataset and model family, we select the final reported model as follows: 1. For each hyperparameter configuration (depth, weight decay, and any model-specific knobs), train to completion and record the best validation loss and corresponding checkpoint. 2. Among configurations within a model family, pick the configuration with the lowest best-validation loss. 3. Evaluate that selected checkpoint on the held-out test set and record test metrics. This selection is performed separately for each dataset and each model family, ensuring that test results are obtained from models chosen without test-set feedback. E.8 Test metrics We report two metrics per dataset–model pairing. Prediction error (MSE). Let \hat{x}_{T}=f_{\theta}(x_{0}) denote the model prediction for the target x_{T} . We report \mathrm{MSE}\;=\;\frac{1}{N_{\mathrm{test}}}\sum_{i=1}^{N_{\mathrm{test}}}\|\hat{x}_{T}^{(i)}-x_{T}^{(i)}\|^{2}. Distance to manifold / constraint violation. For each dataset, we compute a constraint-violation diagnostic d(\hat{x}_{T},\Omega) appropriate to the geometry. We aggregate by reporting the mean over the test set: \mathrm{MeanDist}\;=\;\frac{1}{N_{\mathrm{test}}}\sum_{i=1}^{N_{\mathrm{test}}}d\!\left(\hat{x}_{T}^{(i)},M\right). The metric used can be found in Appendix D.7 .