Characterization of Residual Morphological Substructure Using Supervised and Unsupervised Deep Learning 2021 Characterization of Residual Morphological Substructure Using Supervised and Unsupervised Deep Learning – References Kameswara Bharadwaj Mantha, Daniel H. McIntosh, Cody Ciaschi, Rubyet Evan, Luther Landry, Henry C. Ferguson, Camilla Pacifici, Joel Primack, Nimish Hathi, Anton Koekemoer, Yicheng Guo, + + The CANDELS Collaboration Abstract Automated identification and characterization of galactic substructure is an essential step in understanding the transformative physical processes driving the galaxy evolution. In this study, we investigate the application of deep learning (DL) frameworks to characterize different kinds of galactic substructures hosted within parametric light-profile subtracted “residual” images of a large sample galaxies from the CANDELS survey. We develop a supervised Convolutional Neural Network (CNN) and unsupervised Convolutional Variational Autoencoder (CvAE) and train it on the single-Sérsic profile fitting based residual images of 10,046 10,046 bright and massive galaxies ( H < 24.5 ​ mag H<24.5\,{\rm mag} and M stellar ≥ 10 9.5 ​ M ⊙ M_{\rm stellar}\geq 10^{9.5}\,M_{\odot} ) spanning 1 < z < 3 110^{11}M_{\odot} ), enhancing global galactic star-formation, and facilitating central super-massive black hole growth and triggering. Observationally identifying major mergers is a key methodological step in empirically verifying the “merging – galaxy evolution” connection. Commonly, close-pair and morphological methods are used to identify major mergers, among which quantitative metrics (e.g., CAS , G-M_{20} parameters) and qualitative visual classification of galaxy images or light-profile subtracted residual images (e.g., from GALFIT ) are frequently-employed approaches to assess the presence of disturbed morphological substructure. While these methods yield broadly-consistent, rising merger frequency evolution measurements out to z\sim 2 , each approach selects from a distribution of merger stages (e.g., pre-coalescence vs. late-stage), and suffers from systematic selection biases caused by cosmological surface-brightness dimming, different observability timescales, and subjective interpretation of low surface-brightness signatures. Advancements in machine learning has enabled recent efforts to overcome some of these biases through supervised frameworks and unsupervised deep learning (DL) networks to better identify mergers out to z\sim 3 and accelerate visual classifications. In this study, we develop two new public unsupervised and supervised DL frameworks and demonstrate their applicability on GALFIT -based residual images of \sim 10,000 massive galaxies spanning 11.5 vary from moderately increasing to flat, or diminishing trends ( Williams et al. 2011 ; Man et al. 2016 ; Mundy et al. 2017 ; Ventou et al. 2017 ; Mantha et al. 2018 ; Duncan et al. 2019 ) . This may be due to important study-to-study variant observational effects such as photometric redshifts and their errors, stellar-mass uncertainties, sample incompleteness, and intrinsic field-to-field pair fraction variations. These aspects are being investigated in an ongoing study and will be discussed in an upcoming paper (Mantha et al., in prep ). Simultaneously, several studies have also used morphological methods to identify merging systems, motivated by the theoretical merger simulation predictions that merging galaxies exhibit morphological disturbances induced by gravitational tidal forces during the interaction. Most morphology-based studies often adopt subjective visual identification of disturbed morphology ( Wolf et al. 2005 ; Bell et al. 2006a ; Jogee 2009 ) or quantitative metrics of large-scale galaxy morphology such as Gini-M 20 ( Abraham et al. 2003 ; Lotz et al. 2008 ; Conselice et al. 2009 ) , CAS ( Abraham et al. 1996b ; Abraham et al. 1996a ; Conselice 2003 ; Conselice et al. 2008 ; López-Sanjuan et al. 2009 ) parameters. These morphology based studies find a wide range of (often strongly evolving) merger frequency evolutionary trends \propto(1+z)^{2-5} at z\lesssim 1.5 ( Lotz et al. 2011 ) , albeit with some finding no redshift dependence ( Cassata et al. 2005 ; Lotz et al. 2008 ) , and some noting \sim 25-50\% merger fractions at z\sim 2-3 ( Conselice et al. 2003 ) . Such variations are due to morphological-study specific systematic effects such as small number statistics owing to narrow pencil-beam surveys, redshift dependent cosmological surface brightness dimming, morphological k -corrections when using bluer rest-frame wavelengths, that can impact the completeness and purity of the selected merger samples. Some recent studies have used latest Hubble Space Telescope ( HST ) based data ( Koekemoer et al. 2011 ; Grogin et al. 2011 , e.g., CANDELS survey) in conjunction with more focused indicators of mergers (e.g., multiple nuclei in close proximity, faint and transient tidal features; ( Lackner et al. 2014 ; Mantha et al. 2019 ) ) as a means to evade from some of such systematics. Tidal features are theoretically predicted to be prevalent among merging galaxies ( Toomre & Toomre 1972 ; Eneev et al. 1973 ; Barnes & Hernquist 1996 ; Duc & Renaud 2013 ) , and previous studies have identified galaxies hosting merging activity by detecting tidal features using qualitative visual inspections ( Bell et al. 2006a ; McIntosh et al. 2008 ) and quantitative residual metrics ( Tal et al. 2014 ; Hoyos et al. 2012 ) of the residual images produced by parametric light-profile fitting software ( Peng et al. 2002 , e.g., GALFIT;) . However, these studies often rely on the subjective visual interpretations of human classifiers that can be time intensive and non-repeatable (especially for large-scale surveys), or use quantitative metrics that only indicate the plausibility of tidal signatures and do not quantitatively capture their key properties (strength, shape, color, etc). In an attempt to overcome some of these pitfalls, ( Mantha et al. 2019 ) , we developed a more direct quantitative way to extract and quantify the strengths of residual substructure, including tidal features hosted by CANDELS galaxies. However, the sample used to demonstrate our feature extraction tool is still based on select galaxies identified through visual inspection. This motivates the strong need to develop automated methods for characterizing residual substructures and providing an accelerated path towards performing a more focused visual/quantitative assessment of galaxies hosting highly interpretive signatures (e.g., tidal features). One methodological avenue that has seen recent booming advancements in the field of image-based automated characterization is Deep Learning . Deep learning (DL) conceptually allows the user to automatically extract low-level pixel features (often referred to as latent space) by iteratively adjusting a series of tuneable parameters within the “deep” layers such that the DL network attempts to achieve an overall global (user-defined) goal. This automated extraction of latent space can be broadly categorized into these approaches – user-guided (i.e., supervised) learning, unguided (unsupervised) learning, and sparsely-guided (semi-supervised) learning. The extracted latent space can be used for further regression/classification tasks or for dimensionality reduction purposes. Many recent studies have adopted DL frameworks for various astronomical applications such as galaxy morphology classifications ( Huertas-Company et al. 2015 ; Dieleman et al. 2015 ; Domínguez Sánchez et al. 2018 ; Huertas-Company et al. 2018 ) , gravitational lens finding ( Hocking et al. 2018 ; Metcalf et al. 2019 ; Cheng et al. 2020 ) , galaxy merger identifications ( Ferreira et al. 2020 ; Tohill et al. 2021 ) . Several of these morphological or merger related DL-based studies often train Convolutional Nueral Networks ( Fukushima & Miyake 1982 ; Lecun et al. 1998 , CNNs; for supervised learning;) or Convolutional Autoencoders ( Masci et al. 2011 , CAE; for unsupervised learning;) on the single or multi-wavelength band images of galaxies from the telescopic imaging. However, such an approach may not be ideal in the case of galaxy merger identifications where their signatures are faint and are sub-dominant when compared to the global galactic light profile. As such, devising DL networks to train on the residual images (after subtracting the galactic light profile) may be much more fruitful, yet this approach is relatively unexplored. Only recently, Storey-Fisher et al. 2021 developed Generative Adversarial Networks to model multi-band galaxy images and used the resultant model-subtracted images to identify “anomalous” galaxies. Moreover, it is a common practice among large astronomy survey science communities to run light-profile fitting software ( Peng et al. 2002 , e.g., GALFIT;) and generate residual images for a large galaxy samples. Therefore, analyzing residual images using DL frameworks remains a worthwhile exploratory path, which can provide provide insightful information and automated assessment of the morphological substructure hosted by large galaxy samples. In this DL-based exploratory study, we develop supervised CNN and unsupervised variational CAE (CvAE) networks to characterize different kinds of residual substructures hosted by a large sample of \sim 10,000 massive and bright galaxies from the HST CANDELS survey spanning 13\sigma significant above the background as: SPF=\sum_{x,y\in GOI}|~f_{\rm pix>3\sigma}(x,y)~|, (1) where f_{x,y} is the flux of the pixel corresponding to pixel coordinate (x,y) , and the GOI region is the product of the binary GOI mask (top-right panel in Figure 3 ) and the residual data. In Figure 6 , we show an example visual illustration of the SPF calculation. For each galaxy, we identify sources in the original image using a python-based implementation SEP ( Barbary et al. 2016 ) of a standard source-extraction software SExtractor ( Bertin & Arnouts 1996 ) . We generate a binary sky mask such that all the source segmentation regions are set to zero and the background sky regions are set to unity (top-right panel in Figure 6 ). We then generate a binary GOI mask as described in § 3.2 and produce a GOI residual region by multiplying the GOI mask with the residual data (bottom-left panel in Figure 6 ). We extract the distribution of background sky pixels from the background sky image (sky mask \times residual image), and fit it with a Gaussian to derive its upper and lower 3\sigma bounds. We select the pixels in GOI residual region that have f(x,y)>3\sigma and f(x,y)<3\sigma , and derive its SPF value as the cumulative absolute flux sum of these pixels. For the example shown in Figure 6 , the total positive flux of >3\sigma pixels is 7.56\,{\rm e/s} (native, exposure time normalized image units) and the total negative flux is -3.73\,{\rm e/s} , which corresponds to a SPF=11.28\,{\rm e/s} . We also quantify the Bumpiness ( Blakeslee et al. 2006 ) parameter, which conceptually quantifies the strength of second-order light-profile moments and deviations of galaxy’s structure from a smooth parametric model. It is defined as the ratio of the residual rms to the best-fit sérsic mean model: B=\frac{\sqrt{(1/N)~\sum_{(x,y)\in GOI}~[R_{s}^{2}(x,y)-\sigma_{\rm sky}^{2}]}}{(1/N)~\sum_{(x,y)\in GOI}~[S(x,y)]}, (2) where R_{s} is the smoothed residual image after convolution with a 2D BoxCar filter of size 3\times 3\,{\rm pix} , \sigma_{\rm sky}^{2} is the variance of background sky computed using the best-fitted Gaussian to the background-sky distribution in the above SPF calculation steps, S is the best-fit sérsic model generated by van der Wel et al. 2012 , and N is the number of pixels corresponding to the GOI segmentation region (i.e., white regions in GOI-mask panel of Figure 3 ). Finally, we also compute the Residual Flux Fraction ( Hoyos et al. 2012 , RFF ;) , which conceptually quantifies the additional residual light that cannot be accounted by background noise fluctuations. It is defined as: RFF=\frac{\sum_{x,y\in GOI}R(x,y)-0.8\times\sum_{x,y\in GOI}\sigma_{\rm sky}}{\sum_{x,y\in GOI}S(x,y)}, (3) where R is the residual image, 0.8\times\sum\sigma_{\rm sky} is the expectation of residual light from Gaussian noise fluctuations, and S is the best-fit light profile model. In § 5 ), we assess the deep latent space features of our trained DL models in conjunction with SPF , B , and RFF metrics to interpret their learning. Figure 6: Visual illustration of our Significant Pixel Flux ( SPF ) calculation using an example galaxy image (top-left) from our sample. We show the source-masked image (top-right) and the Region of Interest ( ROI ; bottom-left) highlighting the Galaxy of Interest ( GOI ). We show the histograms of pixel fluxes (bottom-right) in the ROI (green), the sky background distribution (dashed black) fit with a Gaussian (solid black) and their lower and upper 3\sigma bounds with blue and red vertical lines, respectively. We highlight those pixels that are above and below the 3\sigma bounds in bottom-left panel with their respective colors. For this example, SPF =11.28 . Figure 7: Overview of our supervised CNN (top) and unsupervised Convolutional Variational Autoencoder (bottom) frameworks used to analyze the galactic residual substructures (see § 4.1 & § 4.2 ). For both networks, we also visually illustrate our analysis of the latent space features using Principle Component Analysis and unsupervised clustering (see § 5 ). 4 Proposed Deep Learning Models To better automate the characterization of residual galactic substructures and improve the identification of plausible merging signature hosting galaxies, we develop two frameworks – i) a supervised Convolutional Neural Network (CNN) model; ii) an unsupervised Convolutional variational Autoencoder (CvAE) model – to learn the latent space features describing the variety of residual substructures hosted by our sample of \sim 10,000 massive galaxies from the HST CANDELS survey. In this section, we describe the architecture of our CNN and CvAE frameworks and discuss their training and optimization strategy. 4.1 Supervised Framework The architecture of our supervised framework is inspired from the CNN-based classifier developed by Huertas-Company et al. 2015 . In Figure 7 , we show the visual overview of our supervised CNN-based framework. Briefly, our network takes the “object-only” residual image stamps as single channel inputs (i.e., one object-only image per sample), followed by three sets of Convolutional and MaxPooling layers for feature extraction, and two Dense Fully-Connected layers. Given that our input data is scaled and spans [-1,1], we choose a “ Tanh ” activation function for all the convolutional layers, whose resultant activations also span [-1,1]. For our deep latent space feature layer, we choose to extract 512 vectors, motivated by a separate dimensionality reduction exercise, where we apply a direct PCA decomposition of the object-only training images and find that 512 eigen vectors hold \gtrsim 90\% of the explained sample variance (i.e., >90\% of information is captured by reduced, 512 dimensions). We choose Tanh activation for this Dense layer and a softmax activation for the final classification layer (of size 5 ) such that the output is probabilistic prediction per target class (here visual residual class labels) that sums up to unity. Following the standard practices to avoid over-fitting scenarios, we introduce a 50% dropout layer after each Convolutional + MaxPooling block, which conceptually omits (randomly) half of the layer features from propagating further. We experimented with different dropout percentage values and found that using lower values (e.g., \sim 5-10\% ) resulted in worsened model performance with reduced testing set accuracy due to overfitting. Furthermore, we also introduce a Gaussian Noise layer with a standard deviation \sigma=0.02 at the entrance of the network to facilitate learning of latent space that is less susceptible to image noise patterns, and ensures that the network never sees the exact same image during model training. Using a higher value for the entrance random noise layer (e.g., \sigma\sim 0.1 ) induces too much stochasticity into the network and leads to an oscillating model convergence (i.e., cyclic learning-unlearning patterns). To optimize our supervised CNN model during the training, we use categorical cross-entropy loss function with Adam optimizer ( Kingma & Ba 2014 ) with starting learning rate of 10^{-4} . We train our network with the augmented training sample of 25000 images for a maximum of 100 epochs in batch sizes for weight updation of 32 images, and decay the learning rate by 2\times for every 1/3^{\rm rd} of the total steps to aid the model convergence. Our model achieved an accuracy of \sim 95\% after 75 training epochs and it yielded a testing accuracy of \sim 75\% . We note that the model achieves an accuracy of \sim 91\% on the Clean class, but the overall (lower) testing accuracy is due to considerable cross-class confusion between the Peculiar, General, and the Asymmetric classes. We acknowledge that this stage needs further improvement and will one of the main items of future work. In the later section, we assess the latent space features learned by our CNN model. 4.2 Unsupervised Framework There are several approaches to learning the deep latent space in an unsupervised fashion, amongst which Autoencoders, specifically Convolutional Autoencoders ( Masci et al. 2011 , CAE;) are a popular option. Briefly, they comprise of an Encoder-Decoder framework, where the objective is to learn the dimensionality-reduced latent space vectors via iterative adjusting of layer weights such that the output reconstruction (by a Decoder network) matches closely to the input image. Motivated by this ideology, we implement an unsupervised Convolutional Autoencoder based framework to learn the deep latent space that characterizes different galactic residual substructures. Specifically, we adopt a Convolutional Variational Autoencoder (CvAE), which is an improvement over the simple CAE, where the network learns a distribution of latent space vectors described by a unit Gaussian distribution instead of simple latent space point vector, which makes the learned latent space less sensitive to unknown nuisance in the input images. In practice, we noticed that the output reconstruction from the CvAE is qualitatively (i.e., by visually by eye) closer to the input images than CAE counterparts (which are often more “blurry” in appearance). We model our CvAE framework based on the previously mentioned supervised CNN architecture (see § 4.1 ). In Figure 7 (bottom panel), we show our CvAE architecture, where our Encoder network involves three 2D Convolutional layers, whose end features are flattened and passed to a fully-connected, latent space feature layer of size 512 (following our aforementioned PCA-based suggestions). We then connect this latent feature layer to two more fully-connected layers (each of size 512 ) to describe the mean ( \mu_{l} ) and variance ( \sigma^{2}_{l} ) of the latent space features ( l ), which are then passed to a custom unit Gaussian Sampling layer. The output of this layer (and the encoder network) is a latent space feature vector of size 512 , where each feature is sampled from the Gaussian( \mu_{l} , \sigma^{2}_{l} ). Our decoder network starts with an input layer of size 512 (i.e., matching the Encoder output size), followed by three 2D transpose convolutional layers that mirror the sizes of our encoder networks’ convolutional layers. We use ReLu activation function for all the layers in the Encoder-Decoder framework, except for the output layer of the decoder network where we use the Tanh activation. Finally, similar to our CNN framework, we also introduce a Gaussian Noise layer ( \sigma=0.02 ) at the start of the encoder framework to enable latent-space features that are less susceptible to image noise patters. It is worth noting that we do not use feature dropout in between the convolutional layers (unlike in our CNN framework), as in doing so, the output reconstructions qualitatively deviated considerably from their input counterparts, where dropping features resulted in blurrier reconstructions. To optimize our network during the training, we jointly minimize the input image reconstruction loss and the Kullback-Leibler ( KL ) divergence loss of the latent space feature distribution. Conceptually, our optimization process corresponds to requiring the reconstructed outputs to be closer to the input images while simultaneously ensuring that their underlying latent space features (describing the input-output images) are also closely matched. Our total loss function, which is the summation of the image reconstruction loss and the KL -divergence loss is as follows: \mathcal{L}_{\rm total}=\mathcal{L}_{\rm recon}+\mathcal{L}_{\rm KL}, (4) where we choose \mathcal{L}_{\rm recon} as the mean-squared error ( MSE ) between the input ( I ) and the reconstructed image ( \hat{I} ) as: \mathcal{L}_{\rm recon}=\frac{1}{N}\sum_{i,j}(I-\hat{I})^{2}. (5) The KL divergence loss of the latent space vectors ( l ) with mean ( \mu_{l} ) and variance ( \sigma_{l}^{2} ) is given by: -0.5\times[1+\log_{10}(\sigma_{l}^{2})-\mu^{2}_{l}-\exp\{\log_{10}(\sigma_{l}^{2})\}]~. (6) To optimize our CvAE network during the training, we adopt the same strategy as in our supervised approach (see § 4.1 ) with the Adam optimizer and an initial learning rate of 10^{-4} with a decay factor of two every 1/3^{\rm rd} of the steps over a maximum of 100 training epochs. In Figure 8 , we show the input-reconstructed image pairs of example galaxy residuals from the testing data. We notice that the overall visual structure of reconstructed residuals is preserved, albeit with a globally “smoothed” appearance. Figure 8: Example visualization of our unsupervised Convolutional Variational Autoencoder network’s input (top) and reconstruction (bottom) images. 5 Discussion: Assessment of Supervised and Unsupervised Latent Space using PCA and Unsupervised Clustering Thus far, we have developed two DL frameworks, a supervised CNN and an unsupervised CvAE network, and trained them on our augmented training image sample of residual images to learn the latent space representations of different residual substructure characteristics spanning in our data set. In this section, we assess the supervised and unsupervised latent space features with Principle Component Analysis ( Jolliffe 1986 ; de la Calleja & Fuentes 2004 , PCA;) along with supplemental, independently derived quantitative metrics ( SPF , B , and RFF ). We also explore unsupervised clustering algorithms and Support-Vector Machines ( Hearst 1998 ; Huertas-Company et al. 2011 , SVM; ) to identify classification boundaries in the PCA space that can distinguish different residual structures. Figure 9: Visualization of principle component axes of the supervised latent space color coded with five residual class labels (left panel). In the right panel, we show the PC1 vs. PC2 only for the Clean, Peculiar, and General Classes. Figure 10: The supervised CNN based latent space visualized in the PCA eigen axes for a subset of classes (Clean, Peculiar, and General; shown in Figure 9 ) and color coded by different residual quantitative metrics – \log_{10}(SPF) (left panel), Bumpiness ( B ; middle panel), and Residual Flux Fraction ( RFF ; right panel). 5.1 PCA Analysis of Supervised CNN Latent Space Using our trained CNN network, we extract the latent space features representing the unaugmented training image set by running them through the network (with frozen, trained weights) and extracting information from the fully-connected layer penultimate to the classification output (see Figure 7 ). We then run PCA on these latent features using the python-based implementation scikit-learn ( Pedregosa et al. 2011 ) and extract three top-most information bearing eigen components capturing \gtrsim 90\% of the explained variance. In Figure 9 (left panel), we show the training sample projected in the first two principle component axes (PC1 vs. PC2) and color code their corresponding visual residual class labels. We notice that the Clean vs. Peculiar and General classes appear distinctly separate (by eye) in the PCA space, where the bulk of Clean class objects are clustered alongside the “left-arm” and the Peculiar and General classes are along the “right-arm” in the apparent “U” shape of the PC1 vs. PC2 distribution. This distinction can be clearly seen in the right panel of Figure 9 , where we show the PC1 vs. PC2 distribution for peculiar and general classes (i.e., galaxies hosting strong residuals) as these objects are of particular scientific interest, in the context of merging induced tidal features. On the other hand, we notice that the core and asymmetric classes lie in the “saddle” region of the PC1 vs. PC2 distribution ( PC1\in\,\sim[0,1] ), and are considerably overlapping with the peculiar and general classes (at PC1\gtrsim 1 ). Our results based on Figure 9 conceptually illustrate that the CNN learns a higher-order measure of residual strength (especially along PC1) and uses it as a primary leverage to characterize residual substructures into strong, intermediate, and clean groupings. This is intuitive as the CNN’s learning mimics that of a human’s interpretation of the residual strengths when characterizing them. Figure 11: Visualization of the unsupervised CvAE latent space in the PCA eigen axes (PC1 vs. PC2), where each data point is color coded by its visual residual class label (left panel). In the right panel, we only show the Clean, Peculiar, and General classes. Figure 12: Visualization of the unsupervised CvAE latent space in PCA eigen axes along with residual quantitative metrics SPF , B , and RFF following a similar layout as in Figure 10 . To test our above interpretation that the PC1 vs. PC2 may be correlated with a measure of quantitative residual strength, we assess the learned latent space with independently quantified measures of residual strength – SPF , B , and RFF . In Figure 10 , we show the PC1 vs. PC2 distribution for the Clean, Peculiar, and General residual classes along with their quantitative metrics SPF , B , and RFF . We find that all the three quantities correlate distinctly with the with the PC1 and weakly with the PC2 axis. More notably, we find a strong bimodal correlation of PC1 and PC2 with the SPF , where the objects with PC\gtrsim 0.5 have higher SPF with \sim 10-100\times than the objects at PC1\lesssim 0 ( \log_{10}(SPF)\sim-0.5\,{\rm to}\,-1.5 ). We also note a relatively weaker, but noticeable correlation with the Bumpiness and Residual Flux Fractions, where the PC\gtrsim 0.5 have higher B and RFF values than the objects spanning PC\lesssim 0 . Our observations based on Figure 10 conceptually suggest that the latent space learned by our CNN network physically corresponds to a combination of different residual strength metrics. It is worth noting that the training sample distribution along the third principle component (PC3) is nearly identical to the PC2 distribution. As a result, our analysis of the above trends in PC1 vs. PC3 space are very similar to the PC1 vs. PC2 based conclusions. Therefore, we just focus on the PC1 vs. PC2 space to make our main points. Figure 13: Left panel : The sum of squared distances between the data points using k -means clustering technique as a function of number of permissible clusters, indicating a transition from a sharp to gradual fall off at an optimal number of 6 clusters. Right panel : We show unsupervised Gaussian Mixture Modeling based clustering of the PCA embedding of the CNN latent features, where the GMM identified clusters are labelled with unique color coding (see color bar). Figure 14: Visualization of the residual images sampled randomly from the GMM-based clusters identified in the supervised CNN-based PCA space (in Figure 13 ). We outline the row pairs with borders corresponding to the color coding used in Figure 13 , except for the second and third rows, which are random samplings from multiple clusters (GMM labels =0,1,5 ). On each residual image, we indicate the GMM cluster label. 5.2 PCA Analysis of Unsupervised CvAE Latent Space Using our encoder frontend of the trained unsupervised CvAE network, we extract the latent space features corresponding to our unaugmented training image sample. Analogous to our supervised case, we then perform PCA analysis on these latent vectors and analyze trends in the first two principle components PC1 and PC2. In Figure 11 (left panel), we show the PC1 vs. PC2 distributions for all five visual residual classes, and only focus on the Clean, Peculiar, and General classes in its right panel. Immediately, we notice that the overall structure of the PC1 vs. PC2 distribution is less distinctive and more contiguous than our supervised case, where the Clean, Core, and Asymmetric classes follow a continuum with no distinguishing correlation across PC1 and PC2 axes. These results conceptually suggest that although our unsupervised CvAE framework learns a residual strength equivalent measure, it lacks the additional discriminatory information from the visual-based residual characterizations. Nevertheless, it is worth noting that the distribution of General and Peculiar classes along the PC1 axis fall towards the lower tail-end ( PC\lesssim-2 ) of the gaussian-like distribution of the Clean class objects. Similar to our supervised latent space exercise discussed in § 5.1 , we also assess the unsupervised latent features in PC1 vs. PC2 with the residual quantitative metrics ( SPF , B , and RFF ). In Figure 12 , we show the PC1 vs. PC2 for our Clean, Peculiar, and General Classes and color code each data point based on their SPF , B , and RFF values. We find a bimodality in the SPF values along the PC1 axis, where PC1\gtrsim 2 ( PC1\lesssim-2 ) have smaller (larger) SPF values, where there is continuum of objects with intermediate SPF value during -2\lesssim PC1\lesssim 2 . Similarly, we notice a weak correlation between the B and RFF quantities and PC1 , where smaller PC1 values correspond to higher B and RFF measures. It is worth mentioning that there is no distinct correlation between the three metrics and the PC2 axis. Our exercise illustrates that the unsupervised CvAE latent space is only somewhat informative (relative to our supervised case) in terms of distinguishing different residual characteristics, where only one informative eigen axis ( PC1 ) correlates with (independent) physical measures of residual strength. 5.3 Unsupervised Clustering of the CNN and CvAE Latent Features in PCA Space To assess the discriminatory power of our supervised and unsupervised latent space features to naturally distinguish between different residual characteristics, we investigate unsupervised clustering in their corresponding PCA space along with Support Vector Classifiers ( Boser et al. , SVC;) . There are various popular unsupervised clustering techniques to identify data-driven natural clusters in N-dimensional parameter space such as k -means clustering ( Lloyd 1957 ; MacQueen et al. 1967 ) , Spectral Clustering ( Alpert et al. 1999 ) , Density based spatial clustering ( Ester et al. 1996 , DBscan; ) , and Gaussian Mixture Modeling ( McLachlan & Basford 1988 , GMM;) . For our exercise, we choose the posterior probability based cluster assignment approach – GMM, informed by a preliminary k -means clustering assessment of the data distribution in PC1 vs. PC2 space. Figure 15: Same as Figure 13 , but for our unsupervised CvAE latent features in PCA space. First, we repeatedly run k -means clustering on the principle components of the supervised CNN latent features with different number of permissible clusters ( k\in[1,15] ) and compute the “inertia” as the sum of squared distances of the data points to their nearest cluster centers. In Figure 13 (left panel), we show the k -means inertia for different number of clusters, which falls sharply up to an optimal value of k=6 clusters and shifts to a more gradual fall off. This test to find data-driven optimal number of clusters is dubbed the “Elbow” method. Using this information, we run the unsupervised GMM clustering on our CNN latent features in PC1 vs. PC2 space and identify 6 clusters. In Figure 13 (right panel), we show the PC1 vs. PC2 space with data points color coded based on their GMM identified cluster labels ( n_{\rm gmm,cnn} ). We notice that the clusters n_{\rm gmm,cnn}=2,4 have been naturally identified, which quantitatively correspond to residuals with high and low SPF values, respectively. We also find that the clusters n_{\rm gmm,cnn}=0,1,5 span the intermediate region between the n_{\rm gmm,cnn}=2,4 clusters, which also quantitatively correlate to intermediate SPF values. Finally, our method identified the n_{\rm gmm,cnn}=3 cluster data as “outliers” to the remaining clusters. To visually assess the GMM-based clusters, in Figure 14 , we show the residual images of the randomly selected GMM-based cluster members. Interestingly, we notice that n_{\rm gmm,cnn}=2 indeed hosts galaxies with strong and often interesting residual characteristics, and the residuals of the n_{\rm gmm,cnn}=4 objects are nearly “clean” in appearance. On the other hand, we notice that the residuals of objects spanning the clusters n_{\rm gmm,cnn}=0,1,5 host weaker (compared to n_{\rm gmm,cnn}=2 ), diffuse asymmetric signatures. Finally, we find that a majority of the n_{\rm gmm,cnn}=3 residuals host distinct (yet visually-similar) central residual patterns, with no obvious signatures outside the central region. Our exercise suggests that the supervised CNN model learns latent space which can be segregated into qualitatively and quantitatively similar residual characteristics. To automatically identify quantitative boundaries amongst the identified GMM clusters, we use the SVC with a radial-basis function kernel ( \gamma=0.7 and C=1 ). In Figure 13 , we show the SVM-based classification boundaries of the GMM clusters. Our GMM-based cluster identities and SVM-based classification boundaries can be further used to refine/validate our visual residual class characteristics used for training purposes. Figure 16: Following the similar layout as in Figure 14 , we show the residual images sampled randomly from the GMM-based clusters identified in the unsupervised CvAE based PCA space. Following a similar approach to our supervised case, we run an initial k -means inertia test for our unsupervised CvAE latent features in PCA space, and find that it allows an optimal k=2 clusters (see Figure 15 . Using this information, we run GMM-based clustering to identify 2 clusters in the CvAE-based PCA space (see right panel in Figure 15 ). After running an SVC classifier with a “Linear” kernel (owing to the simplistic nature of the PC1 vs PC2 distribution), we find that these two GMM-based clusters ( n_{\rm gmm,vae} ) are split roughly at PC1\sim 0 , where n_{\rm gmm,vae}=1 ( =0 ) cluster quantitatively corresponds to objects with smaller (larger) SPF values, respectively (see Figure 12 ). In Figure 16 , we visually show example residual images randomly sampled per GMM-identified cluster. We find that the n_{\rm gmm,vae}=0 residuals host a mixture of visually strong and intermediate strength residuals, and the n_{\rm gmm,vae}=1 residuals are mostly “clean” with some hosting weak diffuse signatures. This exercise reaffirms our previous discussion that the unsupervised latent features are less informative and lack clear discriminatory power in terms of characterizing residual substructure. 6 Conclusions In this paper, we carryout a Deep Learning (DL) based exploratory study of the residual substructures hosted by a large sample of \sim 10,000 massive and bright galaxies from the Hubble Space Telescope ( HST ) CANDELS survey spanning a redshift range 1