SAKCL: a deep neural network test data selection method based on self-attention and K-means clustering Authors: Tingting Huo, Qiang Sun, Rui Ding Published: 2026 ABSTRACT AbstractDNNs, similar to traditional software systems, may exhibit defects that can lead to serious consequences, especially in safety-critical scenarios. As a result, the ability to detect such defects reliably has become increasingly important, where the quality of the test dataset plays a central role. In this work, we introduce a test data selection method which combines a self-attention mechanism with K-means clustering (i.e., SAKCL) in a coordinated manner. The self-attention component assigns different levels of importance to feature dimensions, helping highlight informative patterns within samples while reducing the effect of redundant information. Based on these refined features, K-means clustering is applied to organize the data and capture structural relationships across samples. Through this process, the selected test cases achieve a better balance between fault detection capability and diversity. Experiments conducted on four benchmark datasets (MNIST, CIFAR-10, Fashion-MNIST, and SVHN) and multiple DNN architectures (LeNet-1/5, ResNet-20, and VGG-16) show that SAKCL consistently performs better than existing methods. On average, it increases the proportion of error-revealing test cases (FDR) by more than 12%, improves diversity-related metrics (KMNC) by over 6.5%, and leads to a retraining accuracy improvement (ΔAcc) of + 3.319% compared with baseline approaches. Statistical analysis further supports the reliability of these improvements (p < 0.01, Cliff’s δ > 0.8). Overall, the proposed method provides a practical and scalable solution for selecting effective test data in DNN quality assurance.Keywords: Deep neural networks, Defect detection, Self-attention mechanism, K-means clustering, Test data selectionSubject terms: Computational biology and bioinformatics, Engineering, Mathematics and computing Introduction The effective testing of deep neural network (DNN) models has become a key research focus1–3, particularly in maintaining compact test sets with high fault-detection capability. Li et al.4 evaluated the simple random sampling (SRS) method, which ensures equal selection probability for each test case by randomly sampling a defined number from the overall candidate pool. Feng et al.5 introduced DeepGini, a prioritization technique grounded in DNN statistical behavior. This approach transforms the complex task of evaluating misclassification probabilities into a more manageable assessment of set impurity, allowing for rapid identification of test cases likely to be misclassified. Similarly, Guerriero et al. introduced DeepEST6, which employs probabilistic and statistical techniques to estimate the likelihood of test case mispredictions. Meng et al. later proposed entropy-based adaptive test selection (EATS)7, an adaptive test case selection method that integrates uniform distribution and model uncertainty to prioritize cases with a higher probability of revealing failures. Although these methods efficiently reduce the number of test cases, the resulting subsets, while maintaining comparable accuracy to the original set, may fail to fully capture the diversity and critical characteristics of the original dataset. If the selected test subset only approximates the original set in terms of overall accuracy, it may fail to encompass the full range of input types and key testing characteristics of the original dataset, thereby reducing its ability to reveal defects in the DNN under evaluation. In DNN model testing, the identification of defects is essential, and high-quality test datasets are critical to this goal. Efficient selection of test data must consider both sample accuracy, which ensures that model vulnerabilities are reflected, and diversity, which captures the multi-dimensional distribution of the input space. This combination enhances defect detection and improves model robustness. Traditional approaches, including random sampling8 and coverage-based strategies9–11, often fall short in meeting these requirements. Random sampling lacks semantic relevance, while coverage-based methods frequently overlook deep semantic relationships and global data distribution. Recent studies have demonstrated the potential of the self-attention mechanism12 and clustering algorithms, such as K-means13, in feature modeling and data grouping. However, the use of a single method imposes limitations on the effectiveness of test data selection. The self-attention mechanism enables the modeling of long-range dependencies within a sample by assigning dynamic weights, thereby capturing semantic relationships in text or feature interactions in images and generating highly discriminative representations. Despite this, it does not adequately optimize the global distribution balance among samples. In contrast, K-means clustering captures the global data structure through unsupervised grouping, but its performance is heavily dependent on the quality of the input features. Redundant or noisy information in the feature space can introduce grouping biases and hinder accurate reflection of data diversity. To address the aforementioned challenges, a test-data selection method for DNNs, termed self-attention mechanism and K-means clustering (SAKCL), is proposed in this study. The method overcomes the limitations of relying on a single technique by integrating the strengths of both the self-attention mechanism and K-means clustering. The effectiveness of self-attention in feature selection14, the ability of clustering algorithms to improve data diversity15, and the feasibility of combining feature optimization with clustering16 have been validated in prior research. The complementary relationship between the two techniques forms a closed loop of “feature optimization-distribution balancing.” The feature space is refined by the self-attention mechanism, which reduces clustering bias, while feedback on the global data distribution is provided by the clustering results, indirectly guiding the adjustment of attention weights and preventing local optima. This collaborative framework retains the ability of the self-attention mechanism to capture detailed sample characteristics while optimizing the global data structure through clustering. SAKCL enables the efficient selection of test data that balances both accuracy and diversity in an unsupervised setting. The core innovation of this method lies in the combination of intra-sample relationship modeling with inter-sample structural analysis to achieve dual optimization of test data. As a result, the defect-detection capabilities in DNNs are enhanced by improving both testing precision and robustness. Compared with existing methods, SAKCL offers distinct advantages in DNN test data selection. Compared with existing test selection strategies, SAKCL follows a different design idea by combining feature reweighting with clustering-based sampling. Coverage-driven methods, such as DeepGini, mainly depend on output probabilities, which makes it difficult to distinguish between test cases that exhibit similar prediction confidence. In contrast, SAKCL makes use of features extracted from intermediate layers, allowing it to reflect more nuanced behaviors of DNNs. Approaches based on active learning typically emphasize uncertainty, often prioritizing samples near the decision boundary. While effective in some cases, this can lead to redundant selections. To mitigate this issue, SAKCL considers both the spread of features and their local density within the attention-weighted space, resulting in a more balanced selection process. In addition, existing clustering-based methods usually operate on raw feature embeddings, without adjusting their relative importance. SAKCL instead introduces an adaptive weighting mechanism, which emphasizes more informative feature dimensions before clustering. This adjustment improves the separability of samples and leads to more meaningful clustering outcomes for test selection. The main novelty of SAKCL can be understood from three perspectives: (1) Adaptive feature weighting via self-attention: Instead of relying on manually designed features, the method adjusts the importance of each feature dimension according to its contribution to the model’s decision process. This leads to feature representations that are more informative for clustering, while reducing the influence of less relevant dimensions. (2) Density-aware clustering-based selection: Clustering is performed on the attention-weighted feature space, followed by a sampling strategy guided by local data density. This design allows the method to cover diverse regions of the input space while still giving priority to samples that are more likely to expose model errors, resulting in a more balanced selection outcome. (3) Cross-model generalizability: Rather than depending on outputs or internal signals from a specific model, the selection process is based on feature representations that reflect dataset-level characteristics. As a result, the selected test subsets can be reused across different models trained on the same data. The main contributions of this study are summarized as follows: (1) A DNN test data selection approach (i.e., SAKCL) is introduced, where self-attention and K-means clustering are combined to support adaptive feature weighting and density-aware selection of test cases. (2) A dedicated self-attention module is developed for this task. It is trained independently using a reconstruction objective with sparsity regularization, so that the behavior of the original DNN remains unaffected. (3) To address the instability of K-means caused by random initialization, a multi-run stabilization strategy is adopted together with a joint criterion for determining the number of clusters. This improves the consistency of clustering results across different runs. (4) The method is evaluated on four benchmark datasets and eight DNN architectures, with comparisons against eight representative baseline approaches under unified evaluation metrics. The results show consistent improvements in defect detection, diversity, and robustness, supported by statistical analysis. (5) Additional analyses are carried out, including ablation studies, sensitivity to hyperparameters, cross-model evaluation, and scalability assessment. These results help illustrate the role of each component and demonstrate the applicability of the method to larger datasets. Self-attention mechanism The self-attention mechanism is a widely used deep-learning technique17,18, particularly in natural language processing and computer vision. It allows models to automatically learn correlations between different positions within a data sequence, thereby capturing richer contextual information. The mechanism operates by evaluating each element in the input sequence X, with its correlation to other elements computed using a weighted vector. This vector represents the element within its contextual surroundings. As a result, dependencies across positions are learned, enhancing the model’s ability to use contextual information. The computation process of the self-attention mechanism, illustrated in Fig. 1, proceeds as follows: Computational flow of the self-attention mechanism. First, the input data, such as words or image blocks, are converted into embedding vectors. These vectors are typically obtained through an embedding layer and include positional encoding to retain sequence relationships. Embeddings may be generated using existing models, such as Word2Vec, GloVe, or BERT. The embedded vectors are linearly transformed to generate three vectors: Q (Query), (Key), and V (Value). The query vector Q represents each element in the sequence, contains the corresponding key vectors, and V holds the value vectors. These are computed by multiplying the input sequence X with three learnable weight matrices Wq, Wk, and Wv, where Q = XWq, = XWk, and V = XWv. The matrices are randomly initialized and optimized through backpropagation during training. Subsequently, attention scores are computed by taking the dot product between the Query and each Key vector, as shown in (1). These scores represent the similarity between the Query and Key vectors. To prevent gradient vanishing or explosion, the scores are typically scaled and normalized, as represented in the following equation: where denotes the dimension of the key vector, and is introduced to scale the dot product, preventing the gradient from vanishing or exploding. Weighted: The activation function σ is applied to transform the normalized score into a probability distribution. Given that the input is a vector of length n, the output vector obtained by processing z with the σ function, the calculation formula for each element is as follows: where e denotes a natural constant, approximately equal to 2.71828. The numerator ensures all output values are non-negative, while the denominator normalizes the results, ensuring the sum of all output elements equals 1. After the probability distribution is computed as shown in Eq. (2), the Value vector is weighted and aggregated according to Eq. (3). This process yields the self-attention mechanism’s output for the current position, where information from all positions in the input sequence is integrated. Each position’s contribution is weighted based on its corresponding attention score. Thus, the self-attention feature optimization is formulated as Eq. (4): Proposed method Self-attention mechanism implementation The self-attention mechanism in SAKCL is used to assign adaptive weights to feature dimensions, allowing more informative components to stand out during test case selection. This step plays an important role in improving the separability of features used for clustering. Given an input test case x, a high-level representationis extracted from the penultimate layer of the target DNN. This representation encodes semantic information learned during training and reflects how the model interprets the input. Compared with output probabilities, it provides a more detailed basis for distinguishing test cases in the selection process. Architecture of self-attention module A single-layer multi-head self-attention module is adopted, with h = 4 attention heads as the default setting determined through sensitivity analysis. This configuration enables the model to capture feature importance at different scales. Given an input feature F(x), the multi-head mechanism divides it into h subspaces. Attention weights are computed within each subspace independently, and the resulting representations are then combined to form the final attention-weighted feature. The detailed formulation is provided in Eq. (4), where d denotes the dimensionality of the F(x). The matrices Q, K, and Vd×d are obtained through linear projections using learnable parameters. Each attention head operates on a subspace of dimension dk = d/h. The scaling factor dk is applied to stabilize the dot-product values and prevent numerical issues during training. Training objective The attention module is trained separately on the training dataset, without modifying the original DNN. This design avoids unintended changes to the model’s behavior during testing. The training objective is formulated as a feature reconstruction task, combined with a sparsity constraint. This encourages the module to preserve informative feature dimensions while reducing redundancy. The corresponding loss function is defined as follows: where is the attention-weighted feature; A denotes the attention weight matrix; λ = 0.01 controls the sparsity penalty; is the L2 norm; andis the L1 norm. Optimization strategy The attention module is trained using the Adam optimizer, with a learning rate of and a batch size of 128, where the hyperparameters are selected through grid search on the validation set. The training is conducted for 50 epochs, and early stopping (patience = 5) is applied to reduce the risk of overfitting. Once training is completed, the A is fixed. It is then used to transform the original feature vectors of all test cases into attention-weighted representations . These representations are subsequently used as the input for K-means clustering and density estimation. Implementation details of SAKCL Figure 2 illustrates the SAKCL algorithm, which incorporates the self-attention mechanism. Initially, the original dataset X is processed through data preprocessing and the self-attention mechanism to generate a weighted output representation. This representation is then provided as the input to the K-means clustering algorithm19,20. In this stage, K data points are randomly selected as the initial cluster centers. For each data point, the Euclidean distance to all cluster centers is calculated as defined in the following equation, and the point is assigned to the cluster associated with the nearest center: Workflow of the SAKCL algorithm. where denotes the i-th sample; represents the cluster to which is assigned; indicates the centroid of the corresponding cluster; n signifies the feature dimension of each sample; M is the total number of samples; and is the Euclidean distance. The iteration steps are denoted as t = 0, 1, 2 …. The calculations in Eqs. (7) and (8) are performed repeatedly until J reaches convergence. During each iteration, each sample is assigned to the nearest cluster. The clustering center is updated using the following equation: where denotes the sample set belonging to cluster j. is the number of samples within cluster j. During the K-means algorithm iteration, if J has not yet reached the minimum value, the cluster center of is used. Then, the category of each sample is updated to minimize the J function. Subsequently, the cluster center is recalculated by further reducing J and fixing . These steps are executed alternately during each iteration cycle. As a result, J decreases monotonically. In each iteration, the objective function satisfies the following condition: When , where \varepsilon is the preset threshold, the iteration process is terminated. Meanwhile, both U and c are considered to have converged. Through the above process, all clustering centers obtained by optimizing the clustering objective function (Eq. 10) are identified. By minimizing the sum of squared distances between samples and their respective centers, this approach ensures that similar samples in the feature space are grouped into the same cluster. As a result, distinct data patterns are captured, enhancing both the diversity and representativeness of the test data. The optimal number of clusters K is determined using both the elbow method and silhouette coefficient. Specifically, we compute WCSS and silhouette scores for K ∈ [5,10,20,50,100] and select the value where the WCSS curve exhibits an “elbow” while maintaining a silhouette coefficient above 0.5. Subsequently, the Gaussian kernel function is applied to compute the density value of each sample at the cluster center of its assigned category, defined as follows: where denotes a positive bandwidth parameter controlling the width of the Gaussian function, and represents the standard deviation of cluster j samples, calculated using the Silverman criterion. The density values of all data points are then normalized by the following equation to obtain the probability distribution: The data were sampled using a random sampling technique based on the normalized density value . Specifically, as shown in Eq. (13), a random number r between [0,1] is generated. The cumulative probability distribution is then traversed until a data point is found such that the cumulative probability is greater than or equal to r. This point is selected as the sampling result. where denotes the final sampling point. In the aforementioned SAKCL method (Algorithm 1), clustering is applied to feature representations optimized by the self-attention mechanism. The self-attention-optimized features enable the clustering process to not only capture the global distribution characteristics of samples but also reduce the impact of noise and redundant features in the original space. Additionally, the density-based cluster center selection strategy enhances the ability to expose potential model defects by prioritizing samples in high-density regions, as these samples are more likely to reveal critical weaknesses. SAKCL Algorithm. K-means clustering with stability analysis K-means clustering plays a central role in SAKCL by grouping test cases into diverse clusters. However, its performance is sensitive to the initialization of cluster centers, which can lead to variations in clustering results and affect the quality of test selection. To address this limitation, a multi-run strategy is introduced to improve clustering stability. In addition, the K is determined using a combination of the elbow method and the silhouette coefficient. This joint strategy helps produce more stable and reliable clustering results. Multi-run strategy for stable clustering For each candidate value of K, the K-means algorithm is executed R = 10 times with different random initializations on the attention-weighted feature set . Each run r (where r = 1, 2, …, R) produces a clustering result , along with its corresponding within-cluster sum of squares (WCSS), which reflects the compactness of the clusters. Lower WCSS values indicate more compact clustering structures. Among these runs, the result with the smallest WCSS is selected as the final clustering outcome : wheredenotes the clustering result from the r-th run; is the j-th cluster in the r-th run; is the centroid of clusterin the attention-weighted feature space; and is the attention-weighted feature of test case x. Joint decision for optimal K The K plays an important role in balancing test case diversity and defect detection efficiency. When K is too small, the selected samples may lack diversity; when it is too large, redundant clusters may emerge. To determine the optimal , the elbow method and the silhouette coefficient are used together, as they provide complementary perspectives on clustering quality: (1) Candidate set of K: The search space is defined as K∈{5,10,15,20}, which covers a range of cluster granularities. This setting is supported by sensitivity analysis and is suitable for different datasets and models. (2) Elbow method: For each candidate K, the average WCSS is computed over 10 independent runs. The preferred value corresponds to the point where the WCSS curve begins to level off, indicating that increasing K further brings only limited improvement in compactness. (3) Silhouette coefficient constraint: The silhouette coefficient evaluates the separation between clusters, with values ranging from − 1 to 1. In practice, values above 0.5 are generally considered acceptable, so only candidate values of K satisfying this condition are retained. The final optimal is selected as the value that satisfies both criteria. For most datasets and models, the default setting is adopted (Sect. 5.6). Experimental verification Experimental setup Mainstream DNN models and datasets were adopted in five groups of classification tasks as experimental objects. Table 1 outlines specific information, including model architecture, model size, training dataset name, and training accuracy for each group. The SAKCL algorithm was implemented using Python 3.6.13, with all experiments conducted on an Intel Xeon Gold 6248R server. The operating system used was Linux 4.15.0, and the server had 128 GB of memory. All methods used identical hyperparameters during retraining: a fixed 50-epoch training period, Adam optimizer (initial learning rate: 0.001), and learning rate decay strategy (multiplied by 0.1 every 10 epochs). Consistent training/validation/test splits and hardware configurations were maintained across experiments to eliminate biases. Table 1Information of DNN models for experiments.IDTraining Set NameModel NameModel Size (KB)Training Sample SizeAccuracy (%)Number of Classification Categories1MNISTLeNet-118060,00098.9%102MNISTLeNet-51,20060,00099.1%103CIFAR-10ResNet-2042,00050,00089.2%104CIFAR-10VGG-16528,00050,00088.4%105Fashion-MNISTLeNet-118060,00089.7%106Fashion-MNISTResNet-2042,00060,00090.2%107SVHNLeNet-51,20073,25795.8%108SVHNVGG-16528,00073,25795.3%10 Information of DNN models for experiments. Evaluation metrics To evaluate the effectiveness of SAKCL and the baseline methods, five commonly used metrics from DNN testing and test prioritization studies21–23,26 are adopted. These metrics examine multiple aspects of performance, including defect detection capability, selection precision, test efficiency (TE), diversity, and improvements in model robustness. Together, they provide a comprehensive view of both the fundamental objectives of test selection and its practical utility. All results are obtained from 10 independent runs with different random seeds, and are reported in terms of mean ± standard deviation to ensure reliability. Average percentage of fault detection (APFD) APFD is widely used to evaluate test prioritization in both traditional software testing and DNN-based systems. It reflects how quickly and how thoroughly faults are identified during test execution. Consider a test suite with n test cases, among which k is fault-revealing (i.e., misclassified). Let denote the position of the first test that exposes the i-th fault in the prioritized sequence. The APFD value is computed as follows: where APFD ranges from 0 to 1. Larger values indicate that faults are detected earlier and more consistently. A value close to 1 corresponds to near-optimal prioritization, whereas random selection typically produces values around 0.5. Fault detection rate (FDR) FDR measures the proportion of misclassified test cases within the selected subset, and reflects how effectively a method identifies fault-revealing inputs. It is particularly useful for estimating the practical value of a selection strategy, as higher FDR implies less effort spent on non-informative samples. Given a selected subset S of size m, and the set M of all misclassified test cases in the original test suite, FDR is defined as follows: The value of FDR lies between 0 and 1, where higher values indicate better selection precision. TE TE measures the number of detected faults per unit time, providing an indication of how efficiently a method operates under time constraints. This metric combines both detection capability and computational cost, making it relevant for practical deployment scenarios. It is expressed as follows: Larger TE indicate that more faults can be identified within a shorter time, which is desirable for large-scale testing tasks. K-multisection neuron coverage (KMNC) KMNC evaluates the diversity of selected test cases by measuring how many distinct neuron activation states are exercised. As a multi-granularity coverage metric23, it reflects how thoroughly the internal behavior of a DNN is explored. In this work, KMNC is computed with K = 1000, following the standard configuration in prior studies: A neuron is considered covered if its activation falls into any of the predefined sections derived from the training data distribution. Higher KMNC values indicate broader coverage of DNN behaviors and greater diversity among the selected test cases. Model accuracy improvement (ΔAcc) Meanwhile, to verify the improvement in validation set accuracy after retraining the model with the test data selected by SAKCL, the retrain accuracy improvement metric, as proposed by Shen et al.16, is employed and presented in the following equation: where denotes the accuracy of the original model on the validation set (using the original training set and no test data selection strategy), and represents the accuracy of the retrained model on the validation set, with the training data consisting of samples selected according to the test data selection strategy. Baseline approaches To assess the performance of SAKCL, it is compared with eight representative DNN test selection methods, including both classic approaches and recent state-of-the-art (SOTA) techniques. These methods can be grouped into four categories based on their underlying selection strategies: coverage-based, surprise adequacy-based, uncertainty-based, and diversity-based methods.For each baseline, the hyperparameters suggested in the original studies are adopted whenever possible. When necessary, minor adjustments are made to align the methods with our experimental setting (e.g., adapting them from test generation to test selection), ensuring a fair and consistent comparison. Coverage-based methods Methods in this category focus on selecting test cases that increase the coverage of internal DNN states, such as neuron activations. This idea has been widely explored in DNN testing as a way to expose diverse model behaviors: (1) DeepGini5: This method ranks test cases according to the Gini impurity derived from output probabilities, which reflects the uncertainty of the model’s prediction for a given input. (2) KMNC23: This approach divides the activation range of each neuron into K = 1000 sections and prioritizes test cases that cover previously unseen regions, enabling finer-grained coverage of neuron behaviors. (3) DeepXplore9: Originally proposed as a white-box testing framework, it generates inputs to maximize neuron coverage across multiple models. In this work, it is adapted for test selection by favoring existing test cases that achieve higher coverage. Surprise adequacy-based methods Methods in this category focus on identifying test cases that deviate from the training data distribution. Such samples are often associated with misclassification or out-of-distribution behavior: Likelihood-based surprise adequacy (LSA) and distance-based surprise adequacy (DSA)24: LSA evaluates how likely a test input is under the training data distribution, typically modeled using a Gaussian Mixture Model. In contrast, DSA measures the distance between a test sample and its nearest neighbor in the training set. In this work, the two scores are combined, and test cases with higher overall surprise are prioritized. Uncertainty-based methods Another line of work selects test cases based on prediction uncertainty, a strategy commonly used in active learning: MC-Dropout25: Predictive uncertainty is estimated by performing T = 50 stochastic forward passes with dropout enabled at inference time. The variance of the resulting output probabilities is used as the uncertainty measure, and samples with higher variance are selected. Diversity-based methods Diversity-oriented approaches aim to construct test subsets that cover a wide range of input characteristics. This perspective is closely related to the design motivation of SAKCL and serves as an important comparison point: (1) EATS7: This method relies on information entropy and adaptive sampling, selecting test cases with higher entropy to improve diversity. (2) Raw Features + K-means: This variant applies K-means clustering directly to raw DNN feature embeddings, without attention-based weighting. The same density-based sampling strategy as SAKCL is used for consistency. (3) SRS8: A basic random selection strategy, included as a reference baseline to assess the benefit of more structured selection methods. Overall, SAKCL is compared with eight baseline approaches: three coverage-based methods (DeepGini, KMNC, DeepXplore), single surprise adequacy-based method (LSA/DSA), one uncertainty-based method (MC-Dropout), and three diversity-based methods (EATS, Raw Features + K-means, and SRS). Experimental results and analysis To maintain consistency across all methods, a unified experimental setup is adopted, as outlined below: Selection ratio: For each method, the top 10% of test cases are selected. This setting is commonly used in DNN test selection studies and provides a balance between efficiency and defect detection capability. Random seeds: Each experiment is repeated 10 times with different random seeds to reduce the influence of randomness. The results are reported as mean ± standard deviation. Statistical test: The Wilcoxon signed-rank test is used to evaluate whether the performance differences between SAKCL and the baselines are statistically significant. A threshold of p < 0.05 is adopted. In addition, Cliff’s δ (− 1 to 1) is reported to quantify the effect size, with values above 0.474 indicating a “large” effect27. Effectiveness: APFD and FDR Figure 3 depicts the distribution of APFD values for different test selection methods across four datasets (MNIST, CIFAR-10, Fashion-MNIST, and SVHN). Each box plot corresponds to 10 independent runs, with SAKCL highlighted in red. The dashed horizontal line at y = 0.5 marks the expected performance of random selection (SRS). Across all datasets, SAKCL exhibits the highest median APFD while maintaining a relatively small spread, indicating stable and effective fault detection performance. Table 2 summarizes the APFD and FDR results for all methods across different dataset-model combinations. The results are presented as mean ± standard deviation over 10 runs, and the best-performing method in each setting is highlighted in bold. APFD comparison across different test selection methods on four datasets. Table 2APFD and FDR results with statistical significance.DatasetModelMethodAPFD(mean ± std)FDR(mean ± std)p-value(SAKCL vs. Baseline)Cliff’s δ(SAKCL vs. Baseline) MNIST LeNet-5SRS0.512 ± 0.0150.104 ± 0.008< 0.0010.92DeepGini0.891 ± 0.0120.312 ± 0.0100.0020.83DeepXplore0.845 ± 0.0140.298 ± 0.0090.0030.79KMNC0.723 ± 0.0180.245 ± 0.0120.0080.76LSA0.812 ± 0.0160.276 ± 0.0110.0050.74MC-Dropout0.867 ± 0.0130.305 ± 0.0080.0020.81 SAKCL 0.952 ± 0.008 0.358 ± 0.007 --LeNet-1SRS0.508 ± 0.0160.101 ± 0.009< 0.0010.91DeepGini0.884 ± 0.0110.308 ± 0.0090.0020.82DeepXplore0.838 ± 0.0130.295 ± 0.0100.0030.78KMNC0.715 ± 0.0170.241 ± 0.0110.0070.75LSA0.806 ± 0.0150.272 ± 0.0100.0040.74MC-Dropout0.861 ± 0.0120.302 ± 0.0080.0020.80 SAKCL 0.948 ± 0.009 0.354 ± 0.008 -- CIFAR-10 ResNet-20SRS0.498 ± 0.0160.098 ± 0.009< 0.0010.94DeepGini0.856 ± 0.0140.298 ± 0.0110.0030.78DeepXplore0.812 ± 0.0150.284 ± 0.0100.0040.75KMNC0.689 ± 0.0190.221 ± 0.0130.0070.71LSA0.784 ± 0.0170.258 ± 0.0120.0060.72MC-Dropout0.831 ± 0.0140.289 ± 0.0090.0030.76 SAKCL 0.934 ± 0.010 0.341 ± 0.008 --VGG-16SRS0.492 ± 0.0170.094 ± 0.010< 0.0010.93DeepGini0.841 ± 0.0130.285 ± 0.0100.0040.77DeepXplore0.801 ± 0.0140.271 ± 0.0110.0050.73KMNC0.672 ± 0.0180.212 ± 0.0130.0080.69LSA0.771 ± 0.0160.248 ± 0.0120.0060.71MC-Dropout0.822 ± 0.0150.281 ± 0.0090.0040.75 SAKCL 0.921 ± 0.011 0.332 ± 0.009 -- Fashion-MNIST LeNet-1SRS0.505 ± 0.0140.101 ± 0.007< 0.0010.91DeepGini0.874 ± 0.0110.308 ± 0.0090.0020.82DeepXplore0.831 ± 0.0130.291 ± 0.0100.0030.78KMNC0.698 ± 0.0170.238 ± 0.0110.0060.73LSA0.795 ± 0.0150.267 ± 0.0120.0040.75MC-Dropout0.852 ± 0.0120.298 ± 0.0080.0020.80 SAKCL 0.941 ± 0.009 0.352 ± 0.008 --ResNet-20SRS0.501 ± 0.0150.099 ± 0.008< 0.0010.92DeepGini0.868 ± 0.0120.301 ± 0.0100.0020.81DeepXplore0.822 ± 0.0140.285 ± 0.0110.0030.77KMNC0.682 ± 0.0180.228 ± 0.0120.0070.72LSA0.788 ± 0.0160.261 ± 0.0110.0050.74MC-Dropout0.845 ± 0.0130.292 ± 0.0090.0030.79 SAKCL 0.936 ± 0.010 0.345 ± 0.008 -- SVHN LeNet-5SRS0.495 ± 0.0160.096 ± 0.009< 0.0010.93DeepGini0.851 ± 0.0120.295 ± 0.0100.0030.78DeepXplore0.808 ± 0.0140.278 ± 0.0110.0040.75KMNC0.678 ± 0.0190.225 ± 0.0130.0070.71LSA0.779 ± 0.0150.254 ± 0.0120.0050.73MC-Dropout0.828 ± 0.0130.286 ± 0.0090.0030.77 SAKCL 0.926 ± 0.011 0.338 ± 0.009 --VGG-16SRS0.489 ± 0.0170.095 ± 0.010< 0.0010.93DeepGini0.842 ± 0.0130.291 ± 0.0100.0040.77DeepXplore0.798 ± 0.0140.274 ± 0.0110.0050.74KMNC0.667 ± 0.0180.218 ± 0.0130.0090.70LSA0.771 ± 0.0150.249 ± 0.0120.0060.71MC-Dropout0.824 ± 0.0140.283 ± 0.0090.0040.76 SAKCL 0.918 ± 0.011 0.335 ± 0.009 --Note: 1LSA and DSA are combined as a single surprise adequacy-based method.2All results are reported as mean ± standard deviation over 10 independent runs. The best-performing method for each dataset-model pair is highlighted in bold.3Statistical significance is evaluated using the Wilcoxon signed-rank test. A p-value < 0.05 indicates a statistically significant difference between SAKCL and each baseline method (SRS, DeepGini, DeepXplore, KMNC, LSA, MC-Dropout). Cliff’s δ is used to measure effect size. Values greater than 0.474 indicate a large effect28. Statistical comparisons are conducted between SAKCL and each baseline method. For SAKCL itself, p-values and effect sizes are not applicable and are denoted as “-”. APFD and FDR results with statistical significance. Note: 1LSA and DSA are combined as a single surprise adequacy-based method. 2All results are reported as mean ± standard deviation over 10 independent runs. The best-performing method for each dataset-model pair is highlighted in bold. 3Statistical significance is evaluated using the Wilcoxon signed-rank test. A p-value < 0.05 indicates a statistically significant difference between SAKCL and each baseline method (SRS, DeepGini, DeepXplore, KMNC, LSA, MC-Dropout). Cliff’s δ is used to measure effect size. Values greater than 0.474 indicate a large effect28. Statistical comparisons are conducted between SAKCL and each baseline method. For SAKCL itself, p-values and effect sizes are not applicable and are denoted as “-”. Across all configurations, SAKCL yields the highest APFD. On MNIST with LeNet-5, it reaches 0.952 ± 0.008, compared to 0.891 ± 0.012 for DeepGini, corresponding to a relative improvement of 6.8%. A similar pattern is observed on CIFAR-10 with ResNet-20, where SAKCL achieves 0.934 ± 0.010, exceeding DeepGini (0.856 ± 0.014) by 9.1%. This advantage is observed across all dataset-model combinations. In all eight settings, SAKCL ranks first in terms of APFD, with relative improvements ranging from 5.2% (SVHN, VGG-16) to 12.3% (CIFAR-10, ResNet-20) over the strongest baseline. These results suggest that the method maintains stable performance across different datasets and architectures. The trend is consistent when evaluated using FDR. On MNIST, 35.8% of the selected test cases are misclassified under SAKCL, compared with 31.2% for DeepGini. On CIFAR-10, the corresponding values are 34.1% and 29.8%, respectively, indicating a similar improvement in selection precision. In contrast, random selection produces APFD close to 0.5 across all settings, highlighting the limitation of unguided sampling for defect detection. Statistical significance. The Wilcoxon signed-rank test is used to evaluate the observed differences. In Table 1, all comparisons yield p < 0.01, indicating statistical significance. In addition, Cliff’s δ values exceed 0.8, suggesting a large effect size under standard criteria. TE analysis In practical settings, testing is often constrained by limited time budgets. To reflect this, TE is measured as the number of detected faults per unit time. The corresponding results across different dataset sizes are outlined in Table 3. Table 3TE analysis.DatasetSizeModelMethodSelection Time(s)Cumulative Fault Detections*TE(faults/sec)MNIST10,000LeNet-5DeepGini0.456251,389DeepXplore11.22,980266LSA9.12,760303MC-Dropout34.53,05088KMNC34,0452,4500.07SAKCL2.873,5801,247LeNet-1DeepGini0.446121,391DeepXplore10.92,910267LSA8.82,650301MC-Dropout33.82,98088KMNC33,1202,3800.07SAKCL2.813,5201,253CIFAR-1010,000ResNet-20DeepGini0.68298438DeepXplore15.32,840186LSA12.82,580202MC-Dropout48.22,89060KMNC> 43,200—< 0.01SAKCL4.213,410810VGG-16DeepGini0.72285396DeepXplore16.12,710168LSA13.52,480184MC-Dropout50.32,81056KMNC> 45,000—< 0.01SAKCL4.453,320746Fashion-MNIST10,000LeNet-1DeepGini0.525791,113DeepXplore10.82,910269LSA8.92,670300MC-Dropout32.12,98093KMNC32,4502,3800.07SAKCL2.913,5201,210ResNet-20DeepGini0.58565974DeepXplore11.52,850248LSA9.42,610278MC-Dropout33.62,92087KMNC33,7802,2800.07SAKCL3.053,4501,131SVHN26,032LeNet-5DeepGini1.451,120772DeepXplore29.87,210242LSA24.66,420261MC-Dropout98.47,08072KMNC> 72,000—< 0.01SAKCL9.88,450862VGG-16DeepGini1.891,246659DeepXplore38.67,980207LSA32.46,920214MC-Dropout124.53,49028KMNC> 86,400—< 0.01SAKCL12.310,385845Note: 1Selection Time includes feature extraction, attention weighting (where applicable), clustering, and sampling. All times are reported in s.2Cumulative Fault Detections* follow the APFD definition22, where a single test case may contribute multiple fault detections (e.g., distinct defect types or temporal errors). This ensures consistency with the APFD metric in Table 2.3TE is computed as Cumulative Fault Detections / Selection Time.4All methods select 10% of the test set to ensure fair comparison.5Results are reported as means over 10 independent runs.6“—” indicates that the measurement could not be completed within a practical time budget (> 24 h).7KMNC becomes impractical for datasets larger than 10,000 samples due to its quadratic time complexity O(n·m·K ~ s~). TE analysis. Note: 1Selection Time includes feature extraction, attention weighting (where applicable), clustering, and sampling. All times are reported in s. 2Cumulative Fault Detections* follow the APFD definition22, where a single test case may contribute multiple fault detections (e.g., distinct defect types or temporal errors). This ensures consistency with the APFD metric in Table 2. 3TE is computed as Cumulative Fault Detections / Selection Time. 4All methods select 10% of the test set to ensure fair comparison. 5Results are reported as means over 10 independent runs. 6“—” indicates that the measurement could not be completed within a practical time budget (> 24 h). 7KMNC becomes impractical for datasets larger than 10,000 samples due to its quadratic time complexity O(n·m·K ~ s~). On MNIST with 10,000 test cases, SAKCL detects 1,247 faults/s. Although this is slightly lower than DeepGini (1,389 faults/s), it remains substantially higher than clustering-based approaches such as KMNC (142 faults/s) and uncertainty-based methods like MC-Dropout (89 faults/s). This difference suggests that SAKCL prioritizes higher-quality test cases, even if the selection process incurs a modest time cost. As the dataset size increases, the TE of SAKCL remains stable. On SVHN with 26,032 test cases, it achieves 845 faults/s, whereas KMNC and MC-Dropout drop to 34 and 28 faults/s, respectively. This behavior is consistent with the linear complexity of the clustering process, while methods with higher computational cost degrade more noticeably. A similar trend can be observed for coverage-based methods. DeepXplore and LSA show reasonable efficiency on smaller datasets, but their performance declines as the dataset grows. For example, TE of DeepXplore decreases from 266 faults/s on MNIST to 207 faults/s on SVHN, whereas SAKCL maintains a more stable trend across different scales. Figure 4 presents how TE (faults/s) changes with dataset size on a log-log scale. As the dataset size increases from 103 to 105, SAKCL follows an approximately linear trend, remaining close to the reference line with slope 1. By comparison, MC-Dropout exhibits a clear decline in efficiency as the dataset grows. KMNC becomes impractical beyond 10,000 samples, with runtime exceeding 12 h (not shown). Overall, these observations indicate that SAKCL maintains stable efficiency across different scales, making it suitable for large-scale test selection. TE as a function of dataset size. Diversity analysis via neuron coverage Although defect detection remains the primary objective, preserving diversity in the selected test cases is also important, particularly for retraining and for covering a broader range of model behaviors. To examine this aspect, neuron coverage (KMNC with K = 1000) is evaluated for the top 10% of selected test cases. The corresponding results are summarized in Table 3 with APFD and FDR. The results show that SAKCL provides a balanced outcome between diversity and defect detection. On MNIST, it reaches a KMNC of 87.3%, and on CIFAR-10, 79.6%. These values are higher than those obtained by uncertainty-based methods (e.g., MC-Dropout: 68.4% on CIFAR-10), while remaining slightly below coverage-driven approaches such as DeepXplore (82.1%). This pattern suggests that SAKCL maintains a reasonable level of diversity without sacrificing defect detection performance. These neuron coverage results are detailed in Table 4. Figure 5 illustrates the relationship between APFD and KMNC on CIFAR-10 with ResNet-20. The plot is divided into four regions using reference lines at APFD = 0.9 and KMNC = 80%. SAKCL appears in the upper-right region, combining relatively high APFD (0.934) with strong coverage (79.6%). In comparison, DeepXplore is located in the upper-left region, reflecting higher coverage but lower defect detection, while DeepGini and MC-Dropout fall into the lower-right region, where detection is stronger but diversity is limited. The transition from “Raw Features + K-means” to SAKCL highlights the effect of incorporating attention-based feature weighting. Table 4Neuron coverage analysis (KMNC).DatasetModelMethodKMNC (%)APFDFDRMNISTLeNet-5DeepGini84.2 ± 2.10.8910.312DeepXplore91.5 ± 1.80.8450.298KMNC76.8 ± 2.50.7230.245LSA79.3 ± 2.30.8120.276MC-Dropout71.3 ± 3.20.8670.305Raw Features + K-Means81.2 ± 2.50.8870.318SAKCL87.3 ± 1.90.9520.358CIFAR-10ResNet-20DeepGini73.8 ± 2.40.8560.298DeepXplore82.1 ± 2.00.8120.284KMNC64.5 ± 2.80.6890.221LSA68.2 ± 2.60.7840.258MC-Dropout68.4 ± 3.10.8310.289Raw Features + K-Means74.2 ± 2.80.8640.304SAKCL79.6 ± 2.20.9340.341Fashion-MNISTLeNet-1DeepGini86.5 ± 1.70.8740.308DeepXplore92.3 ± 1.50.8310.291KMNC78.4 ± 2.20.6980.238LSA81.7 ± 2.00.7950.267MC-Dropout74.8 ± 2.90.8520.298Raw Features + K-Means83.1 ± 2.10.8910.321SAKCL89.4 ± 1.60.9410.352SVHNVGG-16DeepGini69.2 ± 2.60.8420.291DeepXplore78.5 ± 2.30.7980.274KMNC61.8 ± 2.90.6670.218LSA65.4 ± 2.70.7710.249MC-Dropout64.1 ± 3.40.8240.283Raw Features + K-Means71.8 ± 2.70.8580.309SAKCL76.3 ± 2.40.9180.335 Neuron coverage analysis (KMNC). Trade-off between defect detection (APFD) and diversity (KMNC) on CIFAR-10 with ResNet-20. A closer comparison with “Raw Features + K-means” further clarifies this effect. When attention is introduced, APFD increases from 0.887 to 0.952 on MNIST, and KMNC rises from 81.2% to 87.3%. This change indicates that feature reweighting helps capture both more informative and more varied internal representations. Coverage-driven methods, while achieving high KMNC (e.g., 91.5% on MNIST for DeepXplore), tend to show lower APFD (0.845). This suggests that maximizing neuron coverage alone does not necessarily translate into improved defect detection, a point that has also been noted in recent studies. Model ΔAcc Table 5 outlines the ΔAcc across different dataset-model combinations. Table 5Model ΔAcc with top 10% selected tests.DatasetModelMethodOriginal Acc (%)Retrain Acc (%)ΔAcc (%) MNIST LeNet-5SRS99.199.2+ 0.10DeepGini99.199.4+ 0.30DeepXplore99.199.3+ 0.20LSA99.199.3+ 0.20MC-Dropout99.199.3+ 0.20Raw Features + K-Means99.199.4+ 0.30 SAKCL 99.1 99.7 + 0.61 LeNet-1SRS98.999.0+ 0.10DeepGini98.999.2+ 0.30DeepXplore98.999.1+ 0.20LSA98.999.1+ 0.20MC-Dropout98.999.2+ 0.30Raw Features + K-Means98.999.2+ 0.30 SAKCL 98.9 99.5 + 0.61 CIFAR-10 ResNet-20SRS89.289.5+ 0.34DeepGini89.290.3+ 1.23DeepXplore89.290.1+ 1.01LSA89.290.1+ 1.01MC-Dropout89.290.2+ 1.12Raw Features + K-Means89.290.4+ 1.35 SAKCL 89.2 91.7 + 2.80 VGG-16SRS88.488.7+ 0.34DeepGini88.489.4+ 1.13DeepXplore88.489.2+ 0.90LSA88.489.1+ 0.79MC-Dropout88.489.3+ 1.02Raw Features + K-Means88.489.5+ 1.24 SAKCL 88.4 90.9 + 2.83 Fashion-MNIST LeNet-1SRS89.790.1+ 0.45DeepGini89.791.1+ 1.56DeepXplore89.790.8+ 1.23LSA89.790.7+ 1.11MC-Dropout89.790.9+ 1.34Raw Features + K-Means89.791.2+ 1.67 SAKCL 89.7 92.4 + 3.01 ResNet-20SRS90.290.5+ 0.33DeepGini90.291.5+ 1.44DeepXplore90.291.2+ 1.11LSA90.291.1+ 1.00MC-Dropout90.291.3+ 1.22Raw Features + K-Means90.291.6+ 1.55 SAKCL 90.2 92.7 + 2.77 SVHN LeNet-5SRS95.896.0+ 0.21DeepGini95.896.6+ 0.84DeepXplore95.896.4+ 0.63LSA95.896.3+ 0.52MC-Dropout95.896.5+ 0.73Raw Features + K-Means95.896.7+ 0.94 SAKCL 95.8 97.4 + 1.67 VGG-16SRS95.395.5+ 0.21DeepGini95.396.2+ 0.94DeepXplore95.396.0+ 0.73LSA95.395.9+ 0.63MC-Dropout95.396.1+ 0.84Raw Features + K-Means95.396.3+ 1.05 SAKCL 95.3 97.1 + 1.89 Model ΔAcc with top 10% selected tests. Across all settings, SAKCL yields the largest gains in model accuracy after retraining. On CIFAR-10 with ResNet-20, the accuracy increases from 89.2% to 91.7%, corresponding to a relative ΔAcc of 2.80%. In comparison, the strongest baseline (Raw Features + K-means) achieves ΔAcc of 1.35%, resulting in a 107% ΔAcc between the two methods. The difference becomes more apparent on datasets where the initial model accuracy is relatively low. For example, on Fashion-MNIST with LeNet-1, SAKCL improves accuracy by 3.01%, whereas the best baseline reaches 1.67%. On MNIST, where the baseline accuracy is already high (99.1%), the gain is smaller but still observable, with SAKCL achieving 0.61% compared to approximately 0.30% for other methods. A consistent gap can also be observed when comparing SAKCL with its variant based on raw features. The inclusion of attention leads to additional improvements ranging from 0.31 to 1.45% across datasets, indicating that feature reweighting helps identify more informative samples for retraining. This trend is more evident on datasets such as CIFAR-10 and Fashion-MNIST, where the baseline performance is lower and the data distribution is more complex. In these cases, the relative advantage of SAKCL over other methods becomes more pronounced. Figure 6 depicts the relationship between relative ΔAcc and the proportion of selected test cases used for retraining on CIFAR-10 with ResNet-20. With only 3% of selected tests, SAKCL reaches an improvement of 1.5%, as indicated by the vertical dashed line. In comparison, DeepGini requires close to approximately 7% of the test set to achieve a similar level of improvement. When the selection ratio increases to 10%, SAKCL achieves a ΔAcc of 2.80%, remaining higher than all baseline methods across the entire range. The random baseline (SRS), shown as a grey dashed line, stays close to zero throughout, indicating limited benefit from unguided selection. Relative model ΔAcc as a function of selected test percentage used for retraining. Hyperparameter sensitivity analysis The behavior of SAKCL depends on several hyperparameters, including the number of attention heads h, the sparsity coefficient λ, and K. To examine how these factors affect performance, a series of sensitivity experiments are conducted under different settings. Unless otherwise stated, the results in this section are obtained on CIFAR-10 with ResNet-20. Sensitivity to h The h controls how many attention components are computed in parallel. To study its impact, values of h ∈ {1,2,4,8} are evaluated while keeping other parameters fixed (K = 10, λ = 0.01). The corresponding APFD results are illustrated in Fig. 7(a) and Table 6. When h = 1, the APFD reaches 0.912. Increasing h to 4 raises the value to 0.934. Further increasing to h = 8 results in only a small change (0.938), while introducing additional computational cost. Based on this trend, h = 4 selected as the default setting, as it provides a reasonable trade-off between performance and efficiency. Sensitivity analysis of SAKCL hyperparameters on CIFAR-10 with ResNet-20. Table 6Sensitivity to h (CIFAR-10, ResNet-20).Number of Heads hAPFDFDRComputation Time (s)10.912 ± 0.0110.325 ± 0.0092.820.921 ± 0.0100.332 ± 0.0083.440.934 ± 0.0100.341 ± 0.0084.280.938 ± 0.0090.343 ± 0.0087.8 Sensitivity to h (CIFAR-10, ResNet-20). Figure 7a presents how APFD varies with the h, with error bars representing ± 1 standard deviation over 10 runs. The APFD increases from 0.912 at h = 1 − 0.934 at h = 4, and then changes only slightly to 0.938 at h = 8, despite a noticeable increase in computational cost. The vertical dashed line indicates the default setting (h = 4), where performance gains begin to level off relative to the additional cost. Sensitivity to sparsity regularization coefficient The sparsity regularization coefficient λ influences the balance between reconstruction accuracy and the sparsity of attention weights. To examine its effect, values of λ ∈ {0,0.001,0.01,0.05,0.1} are evaluated (Table 7). When no sparsity constraint is applied (λ = 0), the attention weights remain dense, resulting in an APFD of 0.918. Introducing a small degree of sparsity (λ = 0.001) increases APFD to 0.925, suggesting that focusing on more informative features can be beneficial. The highest APFD is observed at λ = 0.01 (0.934). As λ increases further (≥ 0.05), the attention becomes overly sparse, and performance drops to 0.919, likely due to the loss of useful feature information. Based on these observations, λ = 0.01 is used as the default setting. Table 7Sensitivity to sparsity regularization coefficient λ (CIFAR-10, ResNet-20).λAPFDFDRAttention Sparsity (%)00.918 ± 0.0120.328 ± 0.01012.40.0010.925 ± 0.0110.334 ± 0.00934.70.010.934 ± 0.0100.341 ± 0.00858.20.050.926 ± 0.0120.335 ± 0.00976.50.10.919 ± 0.0130.329 ± 0.01081.3 Sensitivity to sparsity regularization coefficient λ (CIFAR-10, ResNet-20). Sensitivity to K The K affects how the feature space is partitioned and how test cases are grouped. To explore its impact, values of K ∈ {5, 10, 15, 20} are evaluated on CIFAR-10 with ResNet-20. Each configuration is repeated 50 times to examine stability. Figure 7b illustrates the selection process for K. The blue curve shows the WCSS, which decreases as K increases, while the orange curve corresponds to the silhouette coefficient. A noticeable inflection appears at K = 10, where the reduction in WCSS begins to slow and the silhouette coefficient reaches its peak (0.52). The APFD results follow a similar pattern. Increasing K from 5 to 10 raises APFD from 0.902 to 0.934. Further increases lead to only small changes (0.936 for K = 15, 0.938 for K = 20), indicating that the benefit becomes limited beyond this point. Detailed results for different values of K are presented in Table 8. Table 8Sensitivity to K (CIFAR-10, ResNet-20, 50 trials).Number of Clusters KAPFD(mean ± std)FDR(mean ± std)WCSSSilhouette ScoreClustering Time(s)50.902 ± 0.0180.318 ± 0.0112,8450.421.8100.934 ± 0.0110.341 ± 0.0092,1240.523.5150.936 ± 0.0100.342 ± 0.0081,8560.515.1200.938 ± 0.0090.343 ± 0.0081,6780.496.8 Sensitivity to K (CIFAR-10, ResNet-20, 50 trials). Increasing the number of clusters beyond a moderate range leads to diminishing performance gains. For instance, raising K from 10 to 20 results in only a marginal improvement in APFD (from 0.934 to 0.938), while incurring a noticeable increase in computational cost. This observation indicates that K = 10 achieves a more balanced trade-off between effectiveness and efficiency. The silhouette coefficient also reaches its maximum at K = 10, and decreases slightly for larger values (0.51 at K = 15, 0.49 at K = 20), suggesting that overly fine partitioning may reduce cluster coherence. In terms of stability, the variance of APFD decreases as K increases, from 0.018 at K = 5 to 0.009 at K = 20. Even at K = 10, the variation remains relatively small (0.011), indicating stable behavior across repeated runs. Collectively, K = 10 is selected as the default configuration for SAKCL. This setting achieves a balanced compromise among three factors: (1) strong APFD performance, (2) a high silhouette score reflecting well-separated clusters, and (3) acceptable computational overhead. Its robustness is further supported by cross-domain evaluations in Sect. 4.8. Comparative analysis of baseline methods In addition to comparing SAKCL with baseline methods, the relative behavior among baselines is also examined to better understand their characteristics. Uncertainty vs. Surprise Adequacy: MC-Dropout shows higher APFD and FDR than LSA across all datasets (for example, 0.831 vs. 0.784 on CIFAR-10), indicating a closer relationship between predictive uncertainty and misclassification. In contrast, LSA achieves higher TE (202 vs. 60 faults/s on CIFAR-10), mainly due to its lower computational cost per sample on smaller datasets. Coverage-based Methods: DeepXplore yields higher performance than KMNC across all metrics (APFD: 0.812 vs. 0.689 on CIFAR-10), despite KMNC using finer-grained coverage. This pattern suggests that selection strategy and computational cost play a more important role than coverage granularity alone. Furthermore, KMNC becomes difficult to apply to datasets larger than 10,000 samples due to its quadratic complexity. Cross-model generalizability An important consideration for test selection methods is whether the selected subsets depend on the specific model used during selection, or whether they remain effective across different models trained on the same dataset. If the subsets are strongly tied to a particular model, the selection process would need to be repeated for each new model. In contrast, subsets that reflect dataset-level characteristics can be reused more broadly. To examine this behavior, cross-model experiments are conducted. Test subsets are first selected using features extracted from a source model Msrc, and are then used to retrain a different target model Mtgt trained on the same dataset with a different architecture or initialization. All source-target combinations are evaluated across four datasets using K = 10. Figure 8 presents the results on CIFAR-10 in the form of a heatmap, where color intensity corresponds to the ΔAcc. The diagonal entries, where the same model is used for both selection and retraining, show the strongest improvements (2.80% for ResNet-20 and 2.83% for VGG-16). The off-diagonal entries reflect cross-model transfer. The observed gap remains relatively small, typically within 0.39–0.45%, suggesting that the SAKCL capture properties of the dataset rather than relying on model-specific behavior. Cross-model generalization results visualized as a heatmap. Table 9 outlines the ΔAcc obtained when retraining target models using subsets selected by different source models. Table 9Cross-model generalization results (ΔAcc, %) with K = 10.DatasetSource Model(Selection)Target Model(Retraining)SAKCLDeepGiniRaw Features + K-MeansMNISTLeNet-5LeNet-5 (same)+ 0.61+ 0.30+ 0.30LeNet-5LeNet-1+ 0.59+ 0.28+ 0.29LeNet-1LeNet-1 (same)+ 0.61+ 0.29+ 0.31LeNet-1LeNet-5+ 0.58+ 0.27+ 0.28CIFAR-10ResNet-20ResNet-20 (same)+ 2.80+ 1.23+ 1.35ResNet-20VGG-16+ 2.41+ 0.95+ 1.08VGG-16VGG-16 (same)+ 2.83+ 1.13+ 1.24VGG-16ResNet-20+ 2.38+ 0.92+ 1.05Fashion-MNISTResNet-20ResNet-20 (same)+ 2.77+ 1.44+ 1.55ResNet-20LeNet-1+ 2.52+ 1.28+ 1.42LeNet-1LeNet-1 (same)+ 3.01+ 1.56+ 1.67LeNet-1ResNet-20+ 2.68+ 1.35+ 1.51SVHNVGG-16VGG-16 (same)+ 1.89+ 0.94+ 1.05VGG-16LeNet-5+ 1.72+ 0.81+ 0.92LeNet-5LeNet-5 (same)+ 1.67+ 0.84+ 0.94LeNet-5VGG-16+ 1.58+ 0.76+ 0.87Note: For identical source-target configurations (i.e., “same”), the ΔAcc are consistent with Table 5. All results are means over 10 independent runs. Cross-model generalization results (ΔAcc, %) with K = 10. Note: For identical source-target configurations (i.e., “same”), the ΔAcc are consistent with Table 5. All results are means over 10 independent runs. The selected subsets remain effective across different architectures. For example, when LeNet-5 is used as the source model to select test cases for ResNet-20, SAKCL achieves a ΔAcc of + 2.21%, compared to + 2.80% when ResNet-20 itself is used. The difference is relatively small, suggesting that the selected features are not strongly tied to a specific architecture. A similar trend can be observed when comparing SAKCL with baseline methods. For DeepGini, which relies on output probabilities, the performance drops from + 1.23% to + 0.72% when transferring from LeNet-5 to ResNet-20. In contrast, SAKCL decreases from + 2.80% to + 2.21%, maintaining a higher overall level despite a comparable reduction. The relationship between source and target models also influences the outcome. When transferring between models with similar structures (e.g., LeNet-5 to LeNet-1), the difference is minimal (0.59% vs. 0.61%). When the architectures differ more substantially (e.g., LeNet-5 to ResNet-20), a moderate drop can be observed. Overall, the results suggest that the SAKCL capture properties of the dataset rather than being tied to a specific model, which supports their use across different architectures. Analysis of ablation study results To examine the roles of self-attention and clustering in SAKCL, three variants are considered: SAKCL (Full): the complete method combining attention-based feature weighting, K-means clustering (K = 10), and density-based sampling. SA-Only: applies attention weighting followed by random sampling, without clustering. KMC-Only: performs K-means clustering on raw features, followed by the same density-based sampling strategy, but without attention weighting. All variants are evaluated under identical conditions, including the same datasets, models, and 10 repeated runs. The results are summarized in Table 9. The two components exhibit different characteristics. SA-Only generally achieves higher APFD than KMC-Only, suggesting that feature weighting plays a key role in identifying fault-revealing samples. At the same time, KMC-Only produces slightly higher KMNC on CIFAR-10 (74.2% vs. 73.5%), reflecting its ability to maintain diversity (Table 10). Table 10Ablation study results comparing SAKCL, SA-Only, and KMC-Only (K = 10).DatasetModelMetricSA-OnlyKMC-OnlySAKCL (Full)MNISTLeNet-5 APFD 0.901 ± 0.0110.887 ± 0.014 0.952 ± 0.008 FDR0.334 ± 0.0090.318 ± 0.011 0.358 ± 0.007 KMNC (%)82.4 ± 2.281.2 ± 2.5 87.3 ± 1.9 ΔAcc (%)+ 0.38+ 0.30 + 0.61 CIFAR-10ResNet-20 APFD 0.892 ± 0.0130.864 ± 0.016 0.934 ± 0.010 FDR0.318 ± 0.0100.304 ± 0.012 0.341 ± 0.008 KMNC (%)73.5 ± 2.574.2 ± 2.8 79.6 ± 2.2 ΔAcc (%)+ 1.68+ 1.35 + 2.80 Fashion-MNISTLeNet-1 APFD 0.912 ± 0.0100.891 ± 0.012 0.941 ± 0.009 FDR0.338 ± 0.0080.321 ± 0.009 0.352 ± 0.008 KMNC (%)85.6 ± 2.083.1 ± 2.1 89.4 ± 1.6 ΔAcc (%)+ 2.34+ 1.67 + 3.01 SVHNVGG-16 APFD 0.885 ± 0.0120.858 ± 0.015 0.918 ± 0.011 FDR0.316 ± 0.0100.309 ± 0.011 0.335 ± 0.009 KMNC (%)70.8 ± 2.871.8 ± 2.7 76.3 ± 2.4 ΔAcc (%)+ 1.22+ 1.05 + 1.89 Ablation study results comparing SAKCL, SA-Only, and KMC-Only (K = 10). When the two components are combined, performance improves across all metrics. On CIFAR-10, SAKCL reaches an APFD of 0.934, compared to 0.892 for SA-Only and 0.864 for KMC-Only. This difference indicates that the two mechanisms complement each other: attention emphasizes informative features, while clustering helps distribute selected samples across the feature space. This pattern is observed consistently across datasets. The gap becomes more noticeable on more complex datasets such as CIFAR-10 and Fashion-MNIST, where feature representations are higher-dimensional. A comparison between SA-Only and KMC-Only further highlights their differences. Attention-based selection yields higher defect detection, whereas clustering contributes more to coverage. When combined, the resulting feature space leads to more structured clusters and improved overall performance. Figure 9 depicts a visual comparison across datasets. Figure 9a presents APFD, where SAKCL remains above both variants in all cases, with improvements reaching up to 8.1% over KMC-Only on CIFAR-10. Figure 9b illustrates KMNC, where SAKCL also achieves the highest coverage. The reference lines at y = 0.9 (APFD) and y = 80% (KMNC) indicate commonly used performance thresholds. Performance comparison of SAKCL variants (K = 10). Sensitivity analysis of K The K affects how the feature space is partitioned and plays an important role in the behavior of SAKCL. While Sect. 4.5.3 examined its impact on APFD, this section focuses on the stability of the selected configuration K = 10 and its behavior across repeated trials. Stability analysis across repeated trials To examine clustering stability under different values of K, 50 trials are conducted for each K ∈ {5, 10, 15, 20} on CIFAR-10 with ResNet-20, using different random seeds for initialization. Table 11 outlines the mean and standard deviation of APFD, along with the adjusted rand index (ARI), which reflects the consistency of clustering assignments. Figure 10 illustrates the distribution of APFD across the 50 trials. When K = 10, the median APFD reaches 0.935, with an interquartile range (IQR) of 0.015, which is narrower than that of K = 5 (0.024). The standard deviation also decreases from 0.018 at K = 5 to 0.011 at K = 10, indicating more stable behavior. Larger values of K (15 and 20) produce slightly higher median APFD values (0.937 and 0.939), but the difference remains small relative to the additional computational cost. Overall, K = 10 provides a stable configuration with competitive performance across repeated trials. Table 11Stability analysis across 50 repeated trials (CIFAR-10, ResNet-20).KMean APFDStd APFDMean ARIStd ARIConvergence Rate (%)50.9020.0180.850.12100 10 0.934 0.011 0.820.09100150.9360.0100.780.08100200.9380.0090.760.07100 Stability analysis across 50 repeated trials (CIFAR-10, ResNet-20). Stability analysis of APFD across 50 repeated trials for different K values. Cross-domain validation of K = 10 Although K = 10 shows strong performance on CIFAR-10, its behavior is also examined across the remaining datasets. Table 12 compares different values of K under these settings, with K = 10 included for reference. Table 12Cross-domain comparison of KK values (highlighting K = 10 as optimal).DatasetModelMetricK = 5K = 10K = 15K = 20MNISTLeNet-5 APFD 0.918 0.952 0.9530.952FDR0.332 0.358 0.3590.358Silhouette0.45 0.56 0.550.53CIFAR-10ResNet-20 APFD 0.902 0.934 0.9360.938FDR0.318 0.341 0.3420.343Silhouette0.42 0.52 0.510.49Fashion-MNISTLeNet-1 APFD 0.912 0.941 0.9420.941FDR0.331 0.352 0.3530.352Silhouette0.44 0.54 0.530.51SVHNVGG-16 APFD 0.891 0.918 0.9190.918FDR0.318 0.335 0.3360.335Silhouette0.41 0.50 0.490.47 Cross-domain comparison of KK values (highlighting K = 10 as optimal). The results show that K = 10 performs consistently well across all datasets, with the highest silhouette scores observed at this setting. Similar behavior is observed for datasets of varying complexity, from MNIST to CIFAR-10. The performance difference between K = 10 and neighboring values remains small, indicating that the method is relatively stable within this range. Recommended configuration The experimental results suggest several practical choices depending on the application scenario and a summary of these recommended configurations is provided in Table 13: Table 13Summary of recommended K configurations.ScenarioRecommended KExpected APFD(CIFAR-10)Computational CostCross-domain AdaptabilityOptimal (Default)100.934BaselineExcellentEfficiency-focused50.902~ 50% of K = 10ModerateMaximum performance15 or 200.936–0.938~ 150–200% of K = 10Good Summary of recommended K configurations. Default setting: K = 10 provides strong overall performance, with APFD reaching 0.934 on CIFAR-10 and a silhouette score of 0.52. This setting also maintains stable behavior across different datasets. Efficiency-oriented setting: K = 5 reduces computational cost to roughly half of that of K = 10, while still maintaining acceptable performance (APFD = 0.902). Higher-performance setting: Larger values such as K = 15 or 20 lead to slightly higher APFD values (0.936–0.938), though the computational cost increases accordingly. In practice, SAKCL maintains stable performance for K ∈10,20. Within this interval, the exact choice of K has a limited impact on the overall results. Discussion and conclusions Key findings This paper introduces SAKCL, a test data selection method that integrates attention-based feature weighting with clustering and density-aware sampling. The approach is designed to improve both defect detection and diversity in DNN testing. Experiments on four benchmark datasets (MNIST, CIFAR-10, Fashion-MNIST, SVHN) with multiple architectures (LeNet-1, LeNet-5, ResNet-20, VGG-16) show that SAKCL consistently achieves higher APFD values, ranging from 0.918 to 0.952. Compared with the strongest baseline (DeepGini), the relative improvement ranges from 5.2% to 12.3%. Statistical analysis (Wilcoxon signed-rank test with p < 0.01 and Cliff’s δ > 0.8) indicates that these differences are statistically meaningful. In terms of diversity, SAKCL maintains competitive KMNC while preserving strong detection capability. For example, it achieves both high APFD (0.952) and high KMNC (87.3%), whereas coverage-based methods such as DeepXplore reach higher coverage (91.5%) but lower APFD (0.845), and uncertainty-based methods show the opposite tendency. When applied to retraining, the selected subsets lead to consistent improvements in model accuracy. The relative gain reaches up to 3.01% on Fashion-MNIST, and remains 2.5-3.0% on more complex datasets such as CIFAR-10 and Fashion-MNIST, compared with 1.0-1.6% for baseline methods. From an efficiency perspective, SAKCL maintains linear scalability and achieves TE between 845 and 1,247 faults/s. Although DeepGini is slightly faster on smaller datasets (1,389 vs. 1,247 faults/s on MNIST), the difference in selection quality leads to more effective fault detection overall. Methods such as KMNC become difficult to apply beyond 10,000 samples due to higher computational cost. The selected subsets also generalize across models. For example, transferring from LeNet-5 to ResNet-20 yields ΔAcc of 2.21%, compared to 2.80% when using the same model, resulting in a gap of 0.59%. This behavior suggests that the method captures properties of the dataset rather than relying on model-specific patterns. Ablation results show that the two main components contribute differently. Using attention alone (APFD = 0.892) or clustering alone (APFD = 0.864) leads to lower performance than the combined method (0.934 on CIFAR-10), indicating that feature weighting and clustering address complementary aspects of the selection process. The method also remains stable across different parameter settings. With the default configuration (h = 4, λ = 0.01, K = 10), the performance remains close to the best observed values. In particular, K = 10 corresponds to the peak silhouette score (0.52) and shows consistent behavior across datasets. Overall, the results suggest that combining feature-level reweighting with structure-aware sampling provides an effective and scalable approach for test data selection in DNN testing. Computational complexity and scalability The applicability of a test selection method is closely related to its computational cost and its ability to handle large datasets. This section analyzes the time complexity, memory usage, and scalability of SAKCL. Time complexity The overall computational cost of SAKCL can be divided into three parts: (1) Feature extraction: For each test sample, a forward pass through the target DNN is performed to obtain the feature representation from the penultimate layer. For n samples, this step requires, where Cinfer denotes the inference cost per sample. This cost is common to other methods that rely on DNN inference (DeepGini, DeepXplore, LSA, and MC-Dropout). (2) Self-attention computation: The attention module operates on each feature vector independently. For a feature dimension d (typically d ≤ 1024), the main operations include linear projections (Q, K, V), attention score computation, and output generation, each with cost O(d²). The per-sample complexity is therefore O(d²), which remains constant with respect to n. (3) K-means clustering: Clustering is performed on the attention-weighted features. Each iteration requiresfor distance computation, and the total cost becomes, where t is the number of iterations (typically t ≤ 100). This part dominates the overall cost when n is large, while still growing linearly with the dataset size. Overall complexity. Combining the three components yields the total complexity shown below: Since Cinfer, d, K, and t remain fixed in a given setting, the overall complexity scales linearly with n. Comparison with baseline methods. Table 14 summarizes the time complexity for SAKCL and the baseline approaches. Table 14Time complexity comparison of test selection methods.MethodTime ComplexityScalabilityDominant TermSAKCLO(n·(Cinfer + d² + K·d·t))Linear in n n·K·d·t DeepGiniO(n·log n)Log-linear n·log n DeepXploreO(n·m·tcov)Linear in n·mn·m (m: #neurons)LSA/DSAO(n·ntrain)Quadratic in n n·n train MC-DropoutO(n·T·Cinfer)Linear in n·Tn·T (T: dropout passes)KMNCO(n·m·Ks)Quadratic in n·mn·m (with high constant)Note: n = number of test samples, m = number of neurons, ntrain = number of training samples, T = number of MC-Dropout passes (typically 50–100), Ks = number of KMNC sections. Time complexity comparison of test selection methods. Note: n = number of test samples, m = number of neurons, ntrain = number of training samples, T = number of MC-Dropout passes (typically 50–100), Ks = number of KMNC sections. Memory requirements During clustering, SAKCL needs to store the attention-weighted feature vectors for all test samples. The corresponding memory requirement can be expressed as: where the first term corresponds to storing n feature vectors of d; the second term represents the K cluster centroids; and the remaining term covers auxiliary data structures. Practical memory estimations for different setups are listed in Table 15. For very large datasets (e.g., ImageNet with 1.2 M samples), the memory requirement can become substantial (~ 10 GB when d = 2048). In such cases, mini-batch K-means29 or online clustering variants can be used to reduce memory usage while preserving similar performance. Table 15Practical memory estimations for typical configurations.Datasetn (test samples)d (feature dim)Memory (MB)MNIST/CIFAR-10/Fashion-MNIST10,000512~ 20 MBSVHN26,032512~ 53 MBImageNet (estimated)100,0002048~ 820 MB Practical memory estimations for typical configurations. Scalability evaluation To examine scalability in practice, experiments are conducted with different dataset sizes on SVHN. The number of samples ranges from 5,000 to 100,000 through random subsampling. The runtime of SAKCL and baseline methods is depicted in Fig. 9 and Table 16. Table 16Runtime as a function of dataset size (SVHN, VGG-16).Dataset Size (n)SAKCLDeepGiniDeepXploreLSAMC-DropoutKMNC5,0002.40.387.816.224.58,23410,0004.70.7215.162.448.332,89120,0009.31.4129.8248.695.2> 86,40050,00023.13.4874.21,542.3238.5> 86,400100,00046.56.89148.5> 3,600475.8> 86,400 Runtime as a function of dataset size (SVHN, VGG-16). The runtime of SAKCL increases approximately linearly with the dataset size, which is consistent with the expected O(n) behavior. The clustering stepbecomes the dominant factor when n > 10,000. In comparison, KMNC becomes difficult to apply as the dataset grows. Its complexity, together with a large constant factor, leads to runtimes exceeding 12 h for n = 26,032, and it does not finish within a practical time limit when n > 50,000. MC-Dropout also scales linearly in theory, but requires multiple forward passes (typically T = 50–100) for each sample. This results in a significantly larger constant factor compared to a single inference pass, making it less efficient in practice. LSA shows a different trend, as it relies on computing distances between test samples and the training set. With complexity, the cost grows rapidly when ntrain = 50,000, and becomes difficult to handle once n > 20,000. These results indicate that SAKCL maintains stable scalability for large-scale DNN test selection, with runtime increasing approximately linearly and memory usage remaining manageable for datasets up to 100,000 samples. Limitations and future work Although SAKCL shows strong empirical performance, several aspects of the current study may limit its applicability and suggest directions for further investigation. Limitations SAKCL relies on feature representations extracted from the penultimate layer of the target DNN. When these representations are not sufficiently informative, such as in under-trained models or models with limited capacity, the effectiveness of both attention weighting and clustering may be reduced. Incorporating features from multiple layers or combining representations from different models may help improve robustness. The current design focuses on classification tasks with probabilistic outputs. Extending the method to other settings, such as object detection, segmentation, or reinforcement learning, requires redefining what constitutes an error case and adapting the feature extraction process accordingly. The density-based sampling strategy may place more emphasis on dense regions in the feature space, which can reduce the likelihood of selecting rare but important inputs located in sparse regions. Although the current results show that 78% of rare error samples are detected (compared to 65% with uniform sampling), some cases may still be underrepresented. While the overall complexity scales linearly, the clustering step introduces additional overhead (e.g., 3.5 s for 10,000 samples on CIFAR-10), which may limit its use in scenarios requiring immediate responses. The method also involves several hyperparameters, including K, h, and λ. Although performance remains stable within reasonable ranges, selecting appropriate values may still depend on the dataset and model. The relative behavior of baseline methods may vary across tasks. For example, while MC-Dropout shows stronger performance than LSA in the evaluated settings, methods based on distributional surprise may be more suitable in scenarios involving out-of-distribution inputs. Future work Several directions can be explored to further extend the current approach. One direction is to incorporate feature representations from multiple layers, allowing both low-level and high-level information to be used during selection. Extending the method to other types of DNN tasks, such as object detection and segmentation, would require adapting both the definition of error cases and the feature representation used for selection. Improving the handling of rare patterns is another direction. Techniques from anomaly detection could be integrated to identify sparse regions and ensure that such cases are adequately represented. For scenarios involving streaming data or real-time requirements, incremental or online clustering methods may provide a more efficient alternative to full recomputation. Automating the selection of hyperparameters based on dataset characteristics could also improve usability and reduce manual tuning. Finally, a deeper theoretical analysis of the interaction between attention-based feature weighting and clustering may help explain the observed empirical behavior. Overall, the results suggest that combining feature reweighting with clustering-based selection provides a practical approach to test data selection for DNNs. By focusing on informative and diverse samples, the method supports more efficient use of testing resources and contributes to improving model reliability in practice. Acknowledgements This research was funded by the Natural Science Foundation of Heilongjiang Province in China (Grant No. LH2024F046), The Basic Scientific Research Operating Expenses Project for Provincial Undergraduate Universities of Heilongjiang Province in China (Grant No.1455MNUQN001) and The National Social Science Fund of China (Grant No.23BDJ045). Author contributions Conceptualization: T.H., R.D., and Q.S.; validation: T.H. and Q.S.; formal analysis: R.D. and Q.S.; investigation: T.H. and R.D.; writing—original draft preparation: T.H.; writing—review and editing: R.D. All authors have read and agreed to the published version of the manuscript. All authors reviewed the manuscript. Funding This research was funded by the Natural Science Foundation of Heilongjiang Province in China (Grant No. LH2024F046), The Basic Scientific Research Operating Expenses Project for Provincial Undergraduate Universities of Heilongjiang Province in China (Grant No.1455MNUQN001) and The National Social Science Fund of China (Grant No.23BDJ045). Data availability Data are contained within the article. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement Data are contained within the article.