Curvature-based machine-learning method for automated segmentation of dendritic spines Authors: Abdel Kader A Geraldo, Michael A Chirillo, Kristen M Harris, Thomas G Fai Published: 2026 ABSTRACT AbstractRecent advances in connectomics have been led by high-resolution reconstruction of large volumes of neural tissues using electron microscopy (EM), providing unprecedented insights into brain structure and function. Dendritic spines—dynamic protrusions on neuronal dendrites—play crucial roles in synaptic plasticity, influencing learning, memory, and various neurological disorders. However, current spine-analysis methods often rely on manual annotation of subcellular features, limiting their ability to handle the complexity of spines in dense dendritic networks. This paper introduces a novel automated computational framework that integrates discrete differential geometry, machine learning, and 3D image processing to analyze dendritic spines in these intricate environments. By generating distributions of spine morphology from high-resolution images including many thousands of spines, our approach captures subtle variations in spine shapes, offering a nuanced understanding of their roles in synaptic function. This framework is tested on multiple EM datasets with the aim of enhancing our understanding of synaptic plasticity and its alterations in disease states. The proposed method is poised to accelerate neuroscience research by providing a scalable, objective, and comprehensive solution for spine analysis, uncovering insights into the role of spine geometry for neural function. Significance We develop a deep-learning method to probe key aspects of the structure of neuronal dendritic spines and formulate a series of neural-network geometries that improve spine detection. The training data consist of manually segmented dendritic segments from hippocampal area CA1 of juvenile rats. We validate the method by applying the trained model to data obtained from published reconstructions of spiny dendrites and demonstrate that our method successfully learns key features of neuronal structure. By taking this step toward the automation of spine segmentation, we are able to obtain statistical information on individual synaptic structures and correlations between their geometries. Introduction Recent advances in electron microscopy (EM) and connectomics have made it possible to reconstruct astonishingly large volumes of neural tissue at nanometer resolution.1,2 These breakthroughs have opened new opportunities for studying brain microstructures, particularly dendritic spines, dynamic protrusion on dendrites where most excitatory synapses in the brain occur. The morphology of dendritic spines—encompassing shape, size, and density—is highly plastic and closely tied to fundamental neurological processes such as learning and memory.3,4,5,6,7 Quantitative studies have shown that fine-scale geometric features of spines, including head volume, neck length, and curvature, are strongly associated with synaptic strength, developmental state, and functional specialization.8,9,10 Changes in spine structure have also been linked to drug abuse, environmental influences, and a wide range of neurodevelopmental, neurodegenerative, and psychiatric disorders.11,12,13 Accurate detection and analysis of dendritic spines in three-dimensional (3D) reconstructions is therefore essential for advancing our understanding of synaptic organization and plasticity. However, spine identification remains challenging due to their heterogeneous morphologies, dense clustering, and local curvature variations. Manual annotation is highly time intensive and impractical for large-scale datasets, underscoring the need for automated and reliable computational methods. Several segmentation approaches have been proposed in recent years,14,15,16,17,18,19,20,21,22 yet challenges remain in achieving both accuracy and scalability. In particular, many existing 3D convolution neural-network (CNN)-based methods23,24,25 operate directly on voxel representations, which require dense volumetric grids and therefore scale poorly to large EM datasets. This voxel dependence leads to substantial memory overhead, slow inference, and difficulty capturing fine geometric details such as thin spine necks. More broadly, large EM volumes contain millions of surface elements, creating significant computational constraints for voxel-level processing or repeated high-resolution inference. Together, these challenges underscore the need for automated approaches that incorporate geometric cues and remain robust across large, heterogeneous datasets. To address these challenges, we introduce a method for automated spine segmentation that combines discrete differential geometry with deep learning. Our framework begins by preprocessing 3D reconstructions of dendritic segments to smooth the triangulated mesh, reduce noise, and enhance surface quality. From the resulting meshes, we extract geometric descriptors such as Gaussian and mean curvature along with additional features including distances from the dendritic shaft skeleton and clustering-based descriptors. Together, these features provide a rich representation of both local and global morphology. Building on this geometric foundation, we developed a series of deep neural-network (DNN) architectures to evaluate how feature enrichment impacts segmentation performance. The baseline model, DNN1, relies primarily on curvature-based descriptors. DNN2 extends this by incorporating distance-to-skeleton features, improving the separation of spines from shafts. Finally, DNN3 integrates a set of enriched geometric and topological descriptors, enabling the network to capture subtle morphological variations and complex spine arrangements. This progression of models demonstrates how systematically incorporating new features enhances both training convergence and segmentation accuracy. In this work, we use deep learning to generate a dataset consisting of over 500 segmented spines and analyze the distributions of spine neck diameter, spine area, and spine volume along with their correlations. The resulting distributions address a fundamental biological question of how form and function are related in dendritic spines.3,26 Materials and methods In this section, we employ differential geometry to design a DNN architecture for the segmentation of dendritic spines. As a first step, we analyze how curvature can contribute to the characterization of dendritic morphology. Building on this analysis, we then develop and explore a DNN that leverages the geometric properties of dendrites to achieve accurate segmentation. For convenience, a summary table of the symbols and variables used throughout the materials and methods section is provided in Table S1. Dendritic morphology analysis using discrete differential geometry Segmentation of dendritic shafts and spines can be guided by their differential geometric properties, particularly Gaussian and mean curvature. These curvature measures capture local shape variations that distinguish the roughly cylindrical shaft from protruding spines. However, raw curvature values obtained directly from EM reconstructions are often noisy due to the sectioning process and mesh irregularities. To address this, we first smooth the dendritic triangulated surface mesh using discrete differential-geometry techniques, then compute curvature values, and finally enhance these through image-processing filters. This three-step process—smoothing, curvature computation, and enhancement—provides robust geometric descriptors that form the basis for accurate segmentation. Mean and Gaussian curvature In this section, we motivate the use of Gaussian and mean-curvature values as key variables for segmenting the dendritic shaft. In the study of dendritic morphology, the shaft is typically modeled as an approximately cylindrical structure from which spines protrude. The discrete mean H and Gaussian curvatures K computation details are provided in Section S5. The Gaussian curvature provides insight into the surface’s shape at different points. At hyperbolic points, where the surface curves in opposite directions (saddle-like), the Gaussian curvature is negative. Conversely, at elliptical points, where the surface curves uniformly in the same direction (dome-like), the Gaussian curvature is positive. This contrasts with the cylindrical shaft, along which the Gaussian curvature is zero. Applying these principles to dendritic morphology, we observe that, along the spine neck and at the intersection between the neck of a spine and the dendritic shaft, the surface exhibits a saddle shape, leading to negative Gaussian curvature. On the other hand, the spine head, with its more spherical or dome-like structure, exhibits positive Gaussian curvature. The mean curvature further characterizes the surface. On a concave surface, the mean curvature is positive, while, on a convex surface, it is negative. This distinction helps to identify regions of significant shape change. For instance, at the transition from the dendritic shaft to the spine neck, the surface is concave, leading to relatively positive mean curvature. In contrast, the spine head, which is a convex region, has relatively negative mean curvature. These curvature measures are most useful for segmentation when the triangular mesh is sufficiently smooth, as smoothness preserves the overall geometry of the dendritic branches while reducing discretization noise. Under these conditions, the curvature fields more accurately capture the cylindrical shape of the dendritic shaft and the geometric deviations introduced by spine necks and spine heads. As a result, the smoothed curvature values provide a more reliable representation of the underlying morphology, which is valuable in distinguishing the shaft from protruding spines. Smoothing the triangulated surface mesh To analyze the triangulated surface of the dendritic mesh, we first address the inherent roughness introduced by the EM sectioning process. We apply discrete differential-geometry techniques—specifically a discrete approximation of mean-curvature (Willmore) flow—to smooth the triangular surface mesh. We define kbend as the bending coefficient of the surface. The curvature energy over the surface A is and the associated bending force is obtained as the negative gradient of this energy: The surface is then evolved by a gradient-descent step, which produces a smoothed version of the mesh. In practice, this smoothing is implemented using standard discrete differential-geometry operators. Although the mesh consists of discrete triangles, smoothing is well defined: each vertex exchanges curvature information only with the vertices in its 1-ring neighborhood (the set of triangles directly adjacent to it), as illustrated in Figure S1. Updating each vertex using curvature information from its neighbors provides a discrete analog of smoothing on a continuous surface. This suppresses high-frequency numerical noise introduced by the discretization while preserving the underlying geometric structure of the dendritic mesh. Further numerical details, following the approach of Wu et al.,27 are provided in Section S5. The results of performing this smoothing are illustrated in Figure 1, where we compute the Gaussian and mean curvature of a segment of spiny dendrite. The data were obtained from high-resolution 3D EM reconstructions of dendritic segments in the CA1 region of the hippocampus from P21 rats. Slices were subjected to in vitro theta-burst stimulation to induce long-term potentiation (LTP) or to control stimulation, as described elsewhere.28,29,30 Comparing the curvature before and after smoothing, we observe that the processed mesh exhibits a more easily interpretable profile of the dendritic surface, which can be exploited to improve the accuracy of spine-shaft segmentation. Effect of curvature smoothing on dendritic meshes. The plots labeled (init) and (smooth) show the initial dendritic curvature and the curvature after smoothing, respectively. The plots labeled mean and gauss correspond to the mean curvature and Gaussian curvature of a segment of dendrite. Smoothing enhances the curvature profile, producing a more easily interpretable pattern in the mesh that can be effectively leveraged to improve segmentation accuracy. For visualization, Gaussian curvature values are thresholded to remain within an absolute value of 45, while the mean curvature is constrained within an absolute value of 15, thereby highlighting the most relevant mesh faces. The triangular mesh is obtained from CA1 dendritic reconstructions.28,29,30 Enhancement of curvature through image processing Here, we provide an intuition for using image-processing techniques to enhance the mean and Gaussian curvature values. The goal is to emphasize regions of the surface with significant curvature changes, which are more likely to correspond to dendritic shafts or spines, thereby facilitating segmentation. This forms the foundation for the machine-learning methods that will be developed in the segmentation algorithm. First, let us define the following sigmoidal function, which is often used in image processing to normalize and enhance contrast: This transformation maps all real values into the interval (0, 1). For large positive x, ζ(x) → 1, while, for large negative x, ζ(x) → 0. Around x = 0, the function has its steepest slope, which enhances small variations near zero and makes them more distinguishable. Applying this transformation to the mean curvature H and the Gaussian curvature K, we obtain where aH, bH, aK, and bK are empirically chosen parameters used to emphasize specific geometric features of the dendritic mesh. For example, in the neck region of a spine, we expect relatively high negative Gaussian curvature. By appropriately choosing aK and bK, these negative values are pushed toward the lower end of the sigmoid, making them stand out more clearly during segmentation. In contrast, along the cylindrical shaft, the Gaussian curvature is close to zero. Proper tuning ensures that values near zero are mapped consistently with the shaft so that these regions are correctly identified. For mean curvature, which distinguishes concave and convex regions, the transformation highlights transitions: concave regions (positive mean curvature) are enhanced toward higher sigmoid values, while convex regions (negative mean curvature) are mapped toward lower values. In practice, within this paper, the quantities H˜ and K˜ are not computed explicitly. Their formulation serves only to motivate the use of a DNN as an approximation function for the segmentation task, with a sigmoidal activation used in the output layer. Figure 3 shows the spine and shaft probability outputs of the DNN that will be examined next. These results demonstrate a substantial improvement in segmentation quality compared to the manually tuned baseline. The overall processing pipeline—from mesh smoothing and geometric feature extraction to DNN-based prediction—is summarized in the flowchart shown in Figure S2. Visualization of geometric-based and learned probability representations used for dendritic spine segmentation. The part labeled “sigmoid” shows the sigmoid of a linear combination of mean intensity and Gaussian curvature, which highlights the neck region and serves as a handcrafted baseline cue for segmentation. The parts labeled “probability shaft” and “probability spine” display the class-specific probability outputs of the network DNN3 for shaft and spine, respectively. Compared with the sigmoid representation, the learned probabilities provide a clearer and more discriminative separation of spine and shaft regions. The triangular mesh is obtained from CA1 dendritic reconstructions.28,29,30 DNN approach for spine and shaft analysis In the previous section, we provided an intuitive explanation of how segmentation can be enhanced using image-processing techniques. We then introduced empirical filtering parameters that can improve segmentation quality. Instead of manually selecting these parameters and explicitly computing H˜ and K˜, we can leverage DNNs to learn an optimal segmentation approximation function based on H and K. At the same time, DNNs allow us to incorporate non-linearities that further enhance segmentation performance. We analyze three different DNN architectures. The first network is designed to support the second by assisting in the extraction of the shaft skeleton, while the third network relies solely on external Python libraries for skeletonization. Machine-learning methods have also been employed for dendrite segmentation of dendritic spines obtained from confocal reconstruction images using CNNs.20 In this work, we adopt a simple DNN architecture inspired by the physics-informed neural-network (PINN) model,31,32,33,34,35,36,37,38 which has been widely applied to approximate ordinary and partial differential equations. These models leverage their well-known ability to serve as universal function approximators.37 Given the geometrical characteristics of the dendritic triangular mesh, this approximation capability of DNNs constitutes a fundamental tool in our algorithm. Here, we present the architectures that produced the best results among the different trials. DNN (DNN1) using the Gaussian and mean curvature As for the sigmoidal function (4) introduced in the previous section, various model parameters require fine-tuning to enhance the shaft segmentation process. Rather than manually selecting these parameters and computing H˜ and K˜, we train a DNN in a two-step procedure. We first train a DNN, denoted as DNN1, whose architecture is illustrated in Figure 2. The input layer receives both the mean curvature H and the Gaussian curvature K values of the dendritic triangular mesh (after smoothing), together with their squared terms to capture higher-order variations. The input features are preprocessed such that the Gaussian curvature values and their squared terms are thresholded to remain within an absolute value of 45. Specifically, for K = (K1, K2, …, Km), we enforce Ki ∈ [−45, 45]. We then compute Ki2 and threshold its value to the range [0, 45]. Similarly, the mean-curvature values and their squared terms are constrained within an absolute value of 15, following the same procedure as in the Gaussian curvature case. DNN1 architecture used to segment the dendrite shaft. The input layer consists of the mean curvature H and the Gaussian curvature K, computed after smoothing, and their squares. The network includes four hidden layers, each with 50 neurons and ReLU activation functions. The output layer employs a sigmoidal activation function. These thresholds are applied because curvature values can sometimes become very large, and such extreme values do not significantly improve prediction accuracy but can instead cause instability during training and inference. To further reduce instability, any NaN values, if present, are replaced with the average of the corresponding curvature values. The network consists of four hidden layers, each containing 50 neurons with Rectified Linear Unit (ReLU) activation functions to introduce non-linearity.38 The output layer employs a sigmoid activation function, as described in section enhancement of curvature through image processing, to classify vertices into dendritic shafts and spines. The segmentation produced by DNN1 is not fully satisfactory, as regions within spines are sometimes misclassified as part of the shaft. This misclassification arises because certain spines regions exhibit relatively flat curvature, making them appear similar to shaft regions. Skeletonization In this section, we describe that the process used to obtain the skeletonization of dendritic meshes is an image-processing technique that reduces binary shapes to thin, single-pixel-wide lines while preserving their topological structure and connectivity.39,40 The first step in building the skeleton is to ensure that the mesh is watertight, meaning that it forms a completely closed surface with no gaps, holes, or disconnected edges. A watertight mesh guarantees a well-defined interior and exterior, which is essential for reliable geometric processing and for preserving the topological structure during skeletonization. To achieve this, we wrap the existing mesh using the algorithm described in Section S6.1. This procedure converts the surface into a uniformly sampled point cloud, estimates and orients normals, and then applies Poisson surface reconstruction to generate a closed, watertight representation. In addition to Poisson reconstruction, we also evaluated the alpha-wrap method provided by PyMeshLab,41 which is based on the classical α-shape formulation.42 Alpha wrapping produces well-behaved closures and can preserve fine geometric detail; however, in our experiments, it was substantially slower than Poisson surface reconstruction for meshes of this size and resolution. For this reason, we adopted Poisson reconstruction as our primary wrapping method. For completeness, the alpha-wrap implementation is also included in Section S6.1. Once a watertight mesh is obtained, we compute its skeleton using the scikit-image skeletonization package, which implements algorithms from Lee et al. and Zhang and Suen.39,40 Further implementation details are provided in Section S6. This simplified representation captures the essential branching geometry of dendrites for subsequent segmentation and morphological analysis. DNN (DNN2) using shaft skeleton We can enhance dendritic spine-shaft segmentation by incorporating the shaft skeleton. This involves using the distance between the shaft-skeleton vertices and the dendritic branch as an additional input to the DNN. For this purpose, we use the shaft segmented with DNN1 together with the skeletonization procedure outlined in section skeletonization. To begin, we consider the shaft-segmented meshes obtained from DNN1. Since this mesh results from the dendritic branch after spine regions have been segmented, it is no longer watertight. Therefore, we apply the procedure described earlier to make the mesh watertight. Once the shaft mesh skeletonization is completed, we compute the distance D between the shaft skeleton and the dendritic-branch mesh vertices. Here, for each vertex Xl, l ∈ {1, 2, …, n} in the dendritic triangular mesh with n vertices, and {V1, V2, …, Vv} the set of shaft-skeleton vertices, we compute the Euclidean norm: With this additional information, we develop an improved DNN, DNN2, whose architecture is shown in Figure S3. This model is similar to DNN1 except that it incorporates the new input feature. As in the earlier model, the input layer receives both the Gaussian and mean-curvature values of the dendrite, along with the squared values of these curvatures as additional features. The network consists of four hidden layers, each containing 50 neurons with ReLU activation functions to introduce non-linearity. The output layer employs a sigmoid activation function, as discussed in section enhancement of curvature through image processing, to classify vertices into dendritic shafts or spines. Dendrite spine-shaft segmentation using dendritic-branch regions (DNN3) To improve segmentation accuracy, we introduce a DNN (DNN3) that incorporates regional information from dendritic-branch segments. These segments are derived from the dendrite skeleton, which serves as a structural reference for spatial organization. For this model, once the skeleton is extracted, we compute the shortest distance from each mesh vertex to its nearest skeleton point. These distances are then clustered using the k-means algorithm to partition the dendritic mesh into multiple regions. This regional segmentation enables us to distinguish between different parts of the dendrite shaft, spine neck, and spine head—regions that are critical for accurate classification. As shown in Figure 4, to capture variations in dendritic morphology, we apply k-means clustering with multiple values of k. We denote these segmentation features by Sk, where k ∈ {2, 3, …, 10} corresponds to the number of clusters. In particular, the value of each feature Sk is given by the corresponding cluster label. Therefore, this model incorporates nine additional scalar features. In this study, we explore values of k ranging from 2 to 10 to provide a multi-scale representation of regional structure. Additional features used in the DNN training. The plot labeled “gauss−sq” and “mean−sq” show, respectively, the square of the Gaussian and mean curvature of the dendrite. The plot labeled “length sh. skl. vert.” indicates the distance from the shaft skeleton to the dendritic mesh vertices, denoted by D. The plot labeled “kmean−k” illustrates the k subdivision from k-means clustering, corresponding to the set Sk. The triangular mesh is obtained from CA1 dendritic reconstructions.28,29,30 In addition to these region-based segmentation features, we incorporate Gaussian curvature and its squared value as geometric descriptors. These measures are particularly effective for identifying neck regions, as discussed in earlier sections. We intentionally exclude mean curvature as we have observed that the mean-curvature input increases the probability of misclassifying all dome-like regions as spines, leading to false positives. By omitting mean curvature and focusing on more discriminative features, DNN3 achieves improved segmentation performance compared to the baseline models. Note that, in practice, the k-means algorithm is not deterministic because its initial values depend on a random seed. To mitigate this and obtain consistent region assignments, we perform 50 independent clustering runs and select the labeling corresponding to the run with the lowest final k-means inertia. Since inertia measures the within-cluster compactness optimized by k-means, choosing the run with minimal inertia yields the most stable and coherent clustering among all runs. This procedure effectively filters out seed-dependent fluctuations and provides reproducible region assignments without requiring post hoc consensus voting. We also considered using the silhouette score as an alternative selection criterion; however, computing silhouette values for all vertices is prohibitively expensive due to the large number of mesh vertices. For this reason, inertia-based selection offered a practical and computationally efficient solution. Loss function We use the binary cross-entropy (BCE) to compute the loss function, and its derivation is presented below. Let us consider a set of m dendritic meshes, denoted as D=(D1,D2,…,Dm), where each mesh Di consists of ni vertices. For the training set, we assume that each Di has a ground-truth classification matrix Yi∈{0,1}ni×2, which represents the classification of vertices in Di in one-hot encoding, such that We define Y = (Y1, Y2, …, Ym) as the set of all annotation matrices. Next, we define the set of mean-curvature vectors for all dendritic meshes as H = (H1, H2, …, Hm) and the set of Gaussian curvature vectors as K = (K1, K2, …, Km). For each mesh Dj, j ∈ {1, 2, …, m}, the mean-curvature vector is given by Hj=(H1,j,H2,j,…,Hnj,j), and the Gaussian curvature vector by Kj=(K1,j,K2,j,…,Knj,j), where Hi,j and Ki,j denote the mean and Gaussian curvatures of Dj at vertex i, for i ∈ {1, 2, …, nj}. Additionally, we define the distance vector D = (D1, D2, …, Dm), representing the distances between the central curve and the vertices, as well as additional structural descriptors Sk, where k ∈ {2, 3, …, 10}. For each mesh Dj, we write Sjk=(S1,jk,S2,jk,…,Snj,jk). We now define the feature vector for vertex i of mesh j as and the corresponding feature matrix for dendritic mesh Dj as Our objective is to find a function that best approximates the ground-truth classification by minimizing the following loss function: where ‖·‖ denotes an appropriate norm measuring the discrepancy between the predicted and actual classifications. In particular, we use a weighted sum of the BCE, which encourages a more balanced optimization process. The BCE formula is given by The final loss function is defined as Here the weights wj is empirically determined. Note that the mean curvature H and Gaussian curvature K used during training are computed after smoothing the curvature fields on the triangular dendritic-branch meshes. The sole purpose of this smoothing is to obtain stable and interpretable curvature estimates that enhance the segmentation performance during training. As described in section smoothing the triangulated surface mesh, smoothing the curvature improves the readability of the curvature profile and reduces noise-driven artifacts. Thus, the smoothing step in our pipeline serves only as a feature-enhancement operation, not as a geometric modification of the underlying dendrite. Importantly, all manual annotations were performed on the actual, unsmoothed dendritic branches, ensuring that the ground-truth labels correspond to the true biological geometry. During training, the network is therefore supervised to segment the real dendritic shaft and spine structures. After training, all dimensional measurements and quantitative analyses are performed on the original triangulated meshes, not on the smoothed versions. Spine and shaft detection After the training step of the algorithm, the next stage of spine-shaft segmentation involves applying the model for classification as well as performing additional post-processing steps to first isolate the shaft and then the spines. In this section, we analyze the processes required to obtain a reliable segmentation. Grouping dendritic mesh parts into connected vertices After training the DNN, the first step is to classify the vertices of a given test dendritic mesh into shaft and spine categories. The resulting classification can then be grouped into connected components of spine vertices and shaft vertices. As a consequence of the sigmoidal activation function applied to the final layer, the output of the DNN is a probability matrix Y¯=fθ(Z)∈[0,1]n×2, where each row corresponds to a vertex and contains the probabilities of that vertex being classified as a spine, X˜spine (first column greater than the second), or a shaft, X˜shaft (second column greater than or equal to the first). Formally, we define Here, a and b are user-defined weighting parameters that control the decision boundary. In our experiments, we fixed b = 0.81 and initially set a = 3. If the resulting segmentation produced too many shaft labels, we increased a; if too many spine labels were produced, we decreased a. This adjustment allowed us to fine-tune the spine-shaft separation based on the model’s output distribution. Next, we describe how to group the predicted dendritic parts—classified as either spine or shaft —into subgroups of connected components. Let each vertex Xi have an associated set of neighboring vertices denoted by Ni: We define a connected group of vertices Gi⊆D such that, for any pair Xp,Xq∈Gi, there exists a path of vertices Xi1,Xi2,…,Xim∈X˜part satisfying Here, X˜part denotes the set of vertices classified as a given part type; either X˜spine for spines or X˜shaft for shafts. The subset of part-classified vertices within this group is then Finally, the complete set of part-classified vertices Y˜part can be decomposed into disjoint set of connected vertices, where connectivity is defined by neighborhood overlap: Spine-shaft detection Using the subgroups defined above of connected components, we now define the segmentation of the dendritic mesh into spines and shaft. Let Y˜shaft denote the set of connected components classified as shaft. In practice, some of these components may actually be parts of spines, so it is crucial to separate them from the true shaft set for accurate segmentation. To achieve this, we identify the largest connected component in Y˜shaft and designate it as the entire shaft. All remaining connected components in Y˜shaft are then assumed to belong to spines. The set of vertices belonging to the shaft is defined as where · denotes the cardinality of the set. Once the entire shaft is defined, spine segmentation proceeds by removing the shaft vertices from the dendrite vertex set, reclassifying them as spine vertices, and then reapplying the connected-component grouping process described in section grouping dendritic mesh parts into connected vertices to the remaining vertices. The set of segmented spines is defined as where each individual spine component is given by Dataset description This section describes the datasets of dendritic segments used for training and testing our algorithm. To mitigate overfitting, the training and testing sets were independent and originate from different animals. This separation ensures that the model learns generalizable features of dendritic structures rather than memorizing patterns specific to a single specimen. Table S3 provides a consolidated overview of the datasets used for training and testing, including their sources, annotation strategy, and selection criteria. Training dataset The training dataset used as ground truth in this paper comprises six high-resolution 3D EM reconstructions of dendrites from the CA1 region of the hippocampus from rats at postnatal day 21 (P21), an immature developmental stage. Brain slices from these animals either underwent in vitro theta-burst stimulation to induce LTP or received control stimulation, as described elsewhere.28,29,30 For the training dataset, the meshes were manually annotated by experts, where spine and shaft segmentations were performed on all six dendritic-branch reconstructions. Each spine was stored as an individual .obj file in watertight format. The mesh corresponding to the shaft of each dendritic segment was provided separately, along with the complete mesh of the dendritic segment. Because spine and shaft meshes were stored independently, we wrapped the spines and shaft with a tight mesh from the exterior in order to generate a unified dendritic mesh using the algorithm in Section S6.1. We then used the independent spine meshes to label the constituent parts of the new wrapped dendritic mesh. To achieve this, we employed a KD-tree-based nearest-neighbor search to map spine vertices to the wrapped dendritic mesh. Specifically, for each set of spine vertices in the individual dataset vertices, we queried a KD tree built from the wrapped dendritic mesh vertices to identify all branch vertices within a radius threshold rth: Here, query_radius returns the indices of dendrite mesh vertices that lie within a distance rth of each spine vertex. By aggregating these indices, we obtain the set of dendrite vertices corresponding to the annotated spine regions on the wrapped mesh. This mapping step allows us to merge the spine and shaft meshes into a single labeled dendritic mesh, which is then used for both training and validation. Training data augmentation Only six dendritic branches are available for training, so we expanded the dataset by applying a series of geometric transformations. First, we computed the principal-component analysis (PCA) of each mesh and projected its vertices onto the corresponding PCA plane. We then applied random scaling to the vertices in two separate steps, introduced random rotations in two additional steps, and added random translations. These transformations produced four augmented versions of each original mesh, resulting in a total of 30 dendritic branches available for training and validation. Of these, 22 branches were used for training, and the remaining eight were used for validation. Testing dataset For testing, we used data from the axon-spine coupling study, which includes a complete nanoconnectomic 3D reconstruction of hippocampal neuropil obtained via serial EM.43 This dataset contains four independent annotations—two performed by each of two annotators44—providing detailed segmentation of dendritic spines. Out of 151 meshes, we selected 28 spiny dendritic branches for testing based on annotation consensus. Specifically, we included only those meshes where two or more annotators agreed on the presence of at least one dendritic spine. Additionally, for each spine mesh identified by the annotators, we retained only those meshes that were consistently labeled as spines by at least two of the selected annotations. This ensured that the testing set reflected a high-confidence subset of spiny dendritic structures. It is important to note that some structures exhibiting spine-like morphology were not annotated as spines in the test set for two reasons. First, biologically, the presence of a synaptic area in the spine head is essential for defining a spine. Thus, even if a mesh appears spiny, it is not considered a spine if no synapse is present. Second, some meshes were excluded due to incomplete reconstruction—parts of the spine may have been cut off or lost during imaging. In contrast, the training dataset does not apply as strict criteria for spine identification as the test set. As a result, the testing dataset introduces additional segmentation challenges that are not reflected in the training data. Our algorithm does not explicitly account for these differences, which may influence performance. Independent dataset In addition to the training and testing datasets, we also used dendritic segments from the large-scale nanoconnectomic EM reconstruction of mouse neocortex by Kasthuri et al.1 This dataset provides densely reconstructed neuropil at nanometer resolution but does not include manual spine or shaft annotations. We therefore use it solely for qualitative visualization (Figure 6) to demonstrate that our method generalizes to independently acquired EM volumes. These data are not used for training, validation, or quantitative evaluation. Comparison of segmentation results obtained using DNN1, DNN2, and DNN3 on a large-scale nanoconnectomic EM reconstruction of mouse neocortex. In all parts, blue indicates the dendritic shaft mesh, while red highlights the segmented spines. The top part shows the full dendritic segment with predictions from the three models. All models correctly identify the majority of spines, but notable differences emerge upon closer inspection. Three boxed regions highlight areas of disagreement, which are shown in detail in the bottom parts. (A) Spine meshes that DNN1 incorrectly labels as shaft. (B) Dendritic sub-branches that DNN2 misclassifies as spines. (C) A similar misclassification pattern for DNN3, where local curvature and branching geometry lead to spurious spine predictions. Reconstruction taken from Kasthuri et al.1 Results This section presents a comprehensive evaluation of our algorithm’s performance in segmenting dendritic shafts and spines. We first describe the identification of dendritic shafts using geometric properties and a neural-network-based approach. We then assess the ability of the algorithm to segment dendritic spines, highlighting both its strengths and limitations. Figure 6 illustrates the predicted dendritic shafts and spines from datasets from Kasthuri et al.,1 demonstrating that our method successfully identifies spines along the dendritic shaft by leveraging geometric properties such as Gaussian and mean curvature. However, misclassification can occur when the dendritic structure deviates significantly from the idealized cylindrical shape, underscoring the need for further improvements to the method. DNN prediction results analysis In this section, we present and analyze the prediction results of the three DNNs, DNN1, DNN2, and DNN3. The methodology is outlined in section dendrite spine-shaft segmentation using dendritic-branch regions (DNN3), and the evaluation here focuses on four key metrics: the training loss, the dice similarity coefficient (DICE), the Jaccard index intersection over union (IoU), and the area under the receiver operating characteristic curve (AUC). Analysis of the performance The training loss curves for the three models are shown in Figure S4. Among the three models, DNN2 achieves the most effective convergence, reaching a minimum loss of approximately 2.44, closely followed by DNN3 with a minimum loss of about 3.56. In contrast, the baseline architecture DNN1 shows the weakest convergence, with a substantially higher final loss of 6.73 throughout training. These results highlight the importance of the additional features and architectural refinements incorporated into DNN2 and DNN3 for improving optimization stability. To quantitatively assess segmentation accuracy, we computed the IoU, DICE, and AUC scores between predicted and annotated vertices for the dendritic shaft and spines on both the training and test sets. The performance curves are presented in Figure S5. For DNN2, the IoU converges to average values of approximately 0.70 for the shaft and 0.81 for the spines, with corresponding DICE scores of about 0.88 and 0.92, respectively. DNN3 achieves comparable performance across both IoU and DICE metrics, whereas DNN1 lags behind with average IoU values of 0.55 (shaft) and 0.75 (spines), along with lower DICE scores overall. The AUC curves further support these trends: both DNN2 and DNN3 maintain consistently high AUC values throughout training, while DNN1 again shows weaker discriminative performance. Importantly, the similarity between training and testing curves across all metrics indicates good generalization and minimal overfitting. Overall, these results demonstrate a clear progression in performance from DNN1 to DNN3. The incorporation of additional geometric and topological features in DNN2 and DNN3 substantially improves convergence and segmentation accuracy, underscoring the value of feature enrichment in dendritic spine detection. Dendritic spine detection analysis Following dendritic shaft segmentation, we applied the spine detection algorithm described in section spine-shaft detection to identify spines. For each detected spine, we computed the IoU to evaluate segmentation accuracy. In addition, we calculated the IoU for the union of all detected spines to assess overall performance. We then computed accuracy, precision, recall, and F1 score to further quantify classification performance. The segmentation results produced by the DNN models are shown in Figure 5. A clear qualitative progression is evident from DNN1 to the more advanced architectures DNN2 and DNN3. The DNN1 model frequently misclassifies portions of the shaft as spines, resulting in fragmented and noisy predictions. Incorporating additional geometric features—such as the distance to the shaft skeleton—allows DNN2 to reduce these errors and produce a more coherent separation between shaft and spine regions. Building on this, DNN3 integrates richer geometric and topological descriptors, enabling it to capture subtle curvature variations and resolve complex spine clusters more effectively. These improvements are visible not only in the 3D mesh renderings but also in the 2D slice, where the highlighted region clearly shows the reduction in mislabeling from DNN1 to DNN3. Overall, the predicted segmentations from DNN3 align most closely with the expert annotation. These qualitative trends are consistent with the quantitative results in Table S4, where both DNN2 and DNN3 outperform DNN1 across all evaluation metrics. Comparison of segmentation results obtained using DNN1, DNN2, and DNN3. The first three columns correspond to the predictions from DNN1, DNN2, and DNN3, respectively, and the final column shows the expert spine annotation used as ground truth. The top row displays the 3D mesh representations, while the bottom row shows 2D slices of the same regions. In the meshes, red vertices indicate correct spine predictions that match the annotation, sky-blue vertices represent shaft regions misclassified as spines, and yellow vertices represent spine regions misclassified as shaft. DNN1 produces noisier results with more misclassifications, particularly in clustered regions. DNN2 reduces shaft-to-spine errors, and DNN3 achieves the most accurate segmentation overall, with fewer misclassifications and the closest agreement with the expert annotation. The slice highlights the region where mislabeling is most apparent, especially for DNN1 and DNN2. The triangular mesh is obtained from CA1 dendritic reconstructions.28,29,30 Quantitative evaluation and analysis We evaluated segmentation performance using multiple complementary metrics, including AUC, precision, recall, and F1 score, computed under both IoU and DICE criteria, each with standard and union variants. These results, summarized in Table S4, enable a detailed comparison between our geometric-informed models (DNN1, DNN2, DNN3) and established CNN baselines (U-Net, VoxNet, VGG16-FCN). Details on the training procedures and evaluation of the three CNN models are provided in Section S9. The weighting parameters used in the geometric-informed models are specified in Equation 6, where the values of a for each model are listed and the parameter b is fixed at 0.81. Among the CNN baselines, U-Net achieves the highest AUC (0.792), indicating strong global discriminative ability. However, AUC alone does not fully capture fine-scale segmentation quality. When examining precision, recall, and F1 score under the IoU and DICE criteria, the geometric-informed model DNN3 consistently delivers the strongest performance. Under the IoU-union metric, DNN3 attains the highest recall (0.873) and F1 score (0.845), indicating that it captures spine regions more completely than both the CNN baselines and the other geometric-informed variants. Its DICE-union performance is even stronger, with an F1 score of 0.857 and a recall of 0.895. In contrast, DNN1 and DNN2 exhibit more limited recall, particularly under the standard variants of IoU and DICE, where both models tend to miss a substantial fraction of true spines despite maintaining moderate to high precision. VoxNet and VGG16-FCN perform noticeably worse across all metrics, with lower AUC values (0.682 and 0.466, respectively) and substantially reduced F1 scores, especially under the IoU criterion. These results highlight the limitations of intensity-based CNN architectures when applied to thin, high-curvature structures such as dendritic spines. The union variants of IoU and DICE consistently yield higher recall and F1 scores across all models, reflecting their ability to more fairly evaluate clustered or contiguous spine regions. This is particularly relevant in dense dendritic environments, where individual spine boundaries may be ambiguous. Under these more biologically meaningful metrics, DNN3 shows the clearest advantage, outperforming all baselines by a substantial margin. Overall, the quantitative results demonstrate a clear progression in performance from DNN1 to DNN3, with DNN3 providing the most reliable and complete segmentation of dendritic spines. The strong performance of DNN3 under both IoU and DICE—especially in the union variants—underscores the value of incorporating geometric curvature cues for accurate spine-shaft segmentation. Dendrite spine segmentation without smoothing In the segmentation of dendrite triangular meshes, curvature smoothing is typically required. However, when the number of vertices is large—more than 200,000—the process becomes computationally expensive, taking more than 20 h to produce a smooth mesh. We note that, by using DNN3, the computational time can be reduced by performing segmentation without smoothing while still achieving a level of accuracy comparable to that obtained with smoothing. We evaluated the effectiveness of curvature smoothing with DNN3 on both the training and test datasets and compared the accuracy. The computed accuracy for the test dataset is 0.658 (with and without smoothing), while, for the training dataset, it is 0.704 with smoothing and 0.780 without smoothing. These results show that smoothing the curvature of the triangular mesh when using DNN3 is unnecessary, and segmentation may even perform better without it. This phenomenon may be due to the fact that the nine region-based segmentation features Sk enable the DNN to detect the neck area without requiring an enhanced curvature profile produced by smoothing. When we tested the same process on other DNN models, the accuracy was zero in both cases. Application to dendritic surface meshes with many vertices We next use our algorithm to segment the spines of a dendritic mesh dataset containing a large number of vertices and multiple branches. In particular, the dataset we study is from the reconstruction of a sub-volume of mouse neocortex.1 We analyzed a dendritic mesh with 5,387,879 vertices and 10,777,005 faces. This dendritic mesh is about 13.42 times larger than the training dataset used in the previous sections (401,371 vertices and 802,750 faces). While most of the training and testing datasets contain only one main dendritic branch with attached spines, this dataset has nine branches with multiple spines. This highlights how much more complex these dataset is compared to the training and testing data. Because this dataset is relatively large, the segmentation process differs slightly from that used in training. Computations were performed on a MacBook Pro with an Apple M4 Max chip and 64 GB of memory. When we attempted to segment the large dendritic mesh directly, the process failed due to computational constraints. Therefore, we reduced the size of the mesh using the simplification algorithm described in Section S6.2. The mesh was simplified to nearly the size of the largest training dataset, given (about 449,109 vertices and 900,000 faces), and we then proceeded with the same testing process as before. The segmentation results are shown in Figure 6. Overall, DNN3 achieves the strongest performance, correctly labeling the majority of spines across the dendritic mesh. However, it still misclassifies a small region of the shaft as spines, as illustrated in Figure 6C. This error likely arises from local geometric similarities between the shaft surface and spine neck regions, particularly where the mesh bends or narrows. In contrast, DNN1 also performs reasonably well and produces few large false-positive spine regions. However, closer inspection of Figure 6A shows that DNN1 frequently misclassifies spine necks as shaft, and in some cases entire spines are labeled as shaft. These errors are consistent with the limited feature set used by DNN1, where Gaussian and mean curvature alone do not provide sufficient spatial context for reliable spine-shaft discrimination. Finally, DNN2 exhibits the weakest performance on this dataset. It correctly identifies spines in only a subset of regions while misclassifying large contiguous portions of the spines as shaft. This behavior suggests that the combination of shaft-skeleton distance, Gaussian curvature and mean-curvature features are insufficient to capture the complex local geometry present in this large-scale reconstruction. Dendritic spine morphologic parameters In this section, we use the segmentation results obtained from our algorithm to compute several morphological parameters of the dendritic meshes in both the training and testing datasets. These parameters include the neck diameter, head diameter, and spine length. The computation of head and neck diameters was performed using the spine skeleton. This skeleton was obtained by identifying the closest point of the triangular mesh skeleton to each vertex of the segmented spine mesh. To determine the neck and head diameters, we considered each segmented spine’s triangular mesh. For each vertex, we computed the shortest distance to its nearest skeleton point, as described in Equation 5, while computing D. This procedure yielded a thickness profile along the spine. The neck radius was then defined as the minimum of these computed averages, starting from the skeleton vertices closest to the dendritic shaft. Conversely, the head diameter was defined as the maximum computed distance, measured from the skeleton vertices at the farthest point from the shaft. Additionally, to compute the length of the spine, we used the spine skeletonization and interpolated additional points along the skeleton using a spline technique. This interpolation increases the accuracy of the computed distances by smoothing and interpolating along the dendritic spine skeleton.45,46,47 Additional details are provided in Section S7. The spine length was then computed as the sum of Euclidean distances between consecutive interpolated points: where Vi denotes the i-th interpolated vertex. The distribution of morphological parameters in the training and large dendritic mesh datasets is shown in Figure 7. In the first column, the head diameter is plotted against the neck diameter of dendritic spines, while in the second column the spine volume is plotted against the spine area, with a colormap encoding spine length. For clearer visualization, some outliers were removed. Marginal histograms of head and neck diameters are also included to facilitate interpretation. Distributions of morphological properties. Properties obtained using DNN3 from the training dataset44 (top row) and the large dendritic mesh dataset1 (bottom row). In the first column, spine head diameter is plotted against spine neck diameter, while, in the second column, spine volume is plotted against spine area. The colormap encodes spine length. Quantitatively, the segmented training dataset contains 210 spines, with an average neck diameter of 0.166 ± 0.0969 μm, an average head diameter of 0.512 ± 0.101 μm, and an average spine length of 1.35 ± 0.694 μm. The average spine volume is 0.122 ± 0.121 μm3 and the average spine area is 2.30 ± 1.83 μm2, yielding an average area-to-volume ratio of 12.9. Importantly, the relationship between spine volume and area is very strong, as reflected by a high coefficient of determination (R2 = 0.916). In contrast, the segmented large dendritic mesh dataset contains 562 spines, with an average neck diameter of 0.148 ± 0.0787 μm, an average head diameter of 0.661 ± 0.164 μm, and an average spine length of 5.57 ± 5.34 μm. The average spine volume is 0.090 ± 0.085 μm3 and the average spine area is 1.53 ± 1.15 μm2, yielding an average area-to-volume ratio of 13.9. Here too, the relationship between spine volume and area remains strong, with a coefficient of determination of R2 = 0.847. Discussion In this work, we developed and evaluated three geometric-informed DNN architectures for dendritic shaft and spine segmentation. Our results demonstrate a clear progression in performance from the baseline DNN1 to the enhanced models DNN2 and DNN3, with DNN3 achieving the strongest overall segmentation accuracy. By incorporating curvature-based and topology-aware features, these models substantially improve both training stability and predictive performance, particularly in regions with dense spine clusters or complex branching patterns. A key advantage of the geometric-informed approach is its computational efficiency relative to volumetric 3D CNNs. CNN-based models require dense voxel representations that scale cubically with resolution, making them memory intensive and slow when applied to high-resolution EM reconstructions. In contrast, our geometric models operate directly on mesh-derived features, avoiding voxelization and enabling faster inference on large or irregular geometries. As a result, DNN3 not only surpasses all CNN baselines in accuracy but also scales more effectively to high-resolution meshes, making it the preferred choice for detailed EM datasets where preserving fine structural detail is essential. The resulting distributions of spine size and shape connect directly to the fundamental biological question of how form determines function in dendritic spines.3,26 In previous work,48,49 we identified the neck radius as a key parameter controlling endocytic membrane trafficking and dendritic spine maintenance. The distributions generated here suggest that there are roughly bimodal distributions of spine neck diameters, which we conjecture may correspond to distinct physiological states in which vesicle entry is up- or downregulated. Although spine area and volume are observed to be positively correlated, as are the spine length and head diameter, we found only weak correlations between neck diameter and the other morphological properties of the spine (Figure 7). This suggests the possibility that the spine neck diameter functions as an independent geometrical switch to support physiological states of individual spines, such as growth, stability, or atrophy. Limitations in spine group segmentation Despite its overall promise, our algorithm encounters challenges in segmenting spines with complex morphologies and spatial arrangements. Below, we discuss three key failure cases, illustrated in Figure 8. The consistently high precision suggests that, when a spine is detected, it is almost always correct (few false positives). However, the moderate recall for IoU alone indicates that some spines are missed (false negatives), potentially affecting downstream morphological analyses. The substantial performance improvement with union-based IoU suggests that a more inclusive segmentation strategy enhances detection reliability. Examples of misclassification encountered during spine segmentation. Blue indicates the shaft mesh, while other colors indicate spine meshes that were misclassified. (A–D) Cases where multiple spines were incorrectly merged and classified as a single spine. (E) A region of the dendritic shaft that was mistakenly labeled as a spine. (F) A fraction of a spine that was only partially labeled. These examples highlight the challenges of accurately separating spines in dense clusters. The triangular mesh is obtained from CA1 dendritic reconstructions.28,29,30 Dendritic spines frequently appear in dense clusters, complicating their segmentation. These spine groups are two or more spines whose vertices are directly connected, and, after the shaft is removed, they still group together. Figure 8 shows a region containing tightly packed spines where the algorithm struggles to distinguish individual structures due to overlapping curvature features. This limitation highlights the need for improved clustering-based segmentation techniques or the integration of additional geometric descriptors capable of differentiating individual spines in high-density regions. Misclassification of dendritic shaft regions as spines Another failure mode involves the erroneous classification of dendritic shaft regions as spines, as depicted in Figure 8F. This misclassification likely stems from local curvature variations that resemble spine-like features. To mitigate this issue, we propose the incorporation of spatial continuity constraints, contextual neighborhood information, or multi-scale curvature analysis to better distinguish dendritic shafts from true spine structures. Scalability and large-dataset limitations A further limitation arises from the scale of the datasets used in training and evaluation. High-resolution 3D EM reconstructions of dendritic segments generate extremely large meshes, often containing millions of vertices. Processing such datasets requires substantial computational resources, both in terms of memory and runtime. This constraint limits the feasibility of applying our method to very large-scale reconstructions or to entire brain regions without significant preprocessing or downsampling. Moreover, the need to balance mesh resolution with computational efficiency may lead to the loss of fine structural details, particularly in thin spine necks or small protrusions. Addressing this limitation will require the development of more efficient algorithms, parallelized implementations, or hierarchical multi-resolution approaches that can scale to increasingly large datasets while preserving biologically relevant detail. Outlook Overall, our findings highlight the potential of feature-enriched geometric learning for robust dendritic spine detection. By improving segmentation accuracy, computational efficiency, and scalability, these models provide a strong foundation for downstream morphological analyses and quantitative studies of synaptic connectivity. Continued refinement of geometric descriptors, multi-resolution strategies, and post-processing methods will further enhance the reliability and applicability of this framework to increasingly large and complex neural reconstructions. By automating the segmentation of dendritic spines using a deep-learning approach specialized to the distinctive geometries of dendritic spines, this work makes it possible to analyze data at greater scale, setting the stage for future research on the relationship between dendritic spine form and function. In future work, we plan to look at spatial correlations between dendritic spine position along the dendritic shaft and to explore changes in spine shape that occur throughout development, including during long-term potentiation50 and in Parkinson’s disease51 and other disorders. Data availability To make these methods widely accessible, we have posted an open-source code repository on GitHub (GitHub:curvature-based-dendrite-segmentation), with the trained model, training data, and validation data made available in a linked repository (Zenodo: https://doi.org/10.5281/zenodo.19593999). Acknowledgments We acknowledge Myles Joyce for help with spine segmentation. This work was supported by NSF Technology Hub #1707356, NSF NeuroNex2 #2014862, NSF award 2219894, and NIH grants R01MH095980 and R56MH139176 to K.M.H. and NIH grant T32 NS007292 to A.K.A.G. We acknowledge use of the Brandeis High Performance Computing Cluster (HPCC), which is partially supported by the NSF through DMR-MRSEC 2011846 and OAC-1920147. Author contributions A.K.A.G. designed research, developed code, performed analysis, made figures, and wrote the manuscript. M.A.C. designed research and wrote the manuscript. K.M.H. designed research and wrote the manuscript. T.G.F. designed research, performed analysis, and wrote the manuscript. Declaration of interests The authors declare no competing interests. Editor: Guoyou Huang. Footnotes Supplemental information can be found online at https://doi.org/10.1016/j.bpj.2026.06.005. Supplemental citation References 52,53,54,55,56,57,58,59,60,61,62,63,64,65,66 appear in the supplemental information. Supplemental information Document S1. Figures S1–S6, Tables S1–S4, and supplemental materials and methods Document S2. Article plus supplemental information References Associated Data Supplementary Materials Document S1. Figures S1–S6, Tables S1–S4, and supplemental materials and methods Document S2. Article plus supplemental information Data Availability Statement To make these methods widely accessible, we have posted an open-source code repository on GitHub (GitHub:curvature-based-dendrite-segmentation), with the trained model, training data, and validation data made available in a linked repository (Zenodo: https://doi.org/10.5281/zenodo.19593999).