Lightweight CNN SE transformer for robust weed classification with optimizer aware performance Authors: Priyadharshini G, Vetriselvi T Published: 2026 ABSTRACT AbstractDuring the early growth stages of crops, the weeds present can severely affect crop yields. Accurate and fast identification of the types of weeds present using robots can help mitigate this problem, allowing agronomists and farmers to employ more precise methods to control weed growth. In this research, we designed the custom Lightweight CNN-SE-Transformer architecture with smaller computational cost while still being able to capture a wide range of important features. This research evaluates the various convolutional network depth by increasing and decreasing the CNN layers. The model was evaluated with residuals and without residuals, skip connections to learn the ability of light weight architecture. Squeeze-and-Excitation (SE) modules in the convolutional backbone to increase the networks discriminative ability. The sequence of multiple convolutional layers with residuals helps to extract the deep feature extraction which is necessary dealing with small plants with more similarities. The main difference from previous deep learning methods is that the model is able to shrink the sequence of tokens and utilize the subsequent features to more efficiently analyze the data while still maintaining the overall performance of the model. To explicitly prove the proposed model ability, model was evaluated with two strong datasets with three class and 16 class weeds. Proposed architecture performed well on both datasets and achieves best test accuracy. For the three optimizers, Adam, RMSProp, and adam+label smoothing, the performance of each optimizer is evaluated to gain insight on the convergence and stability of the model under a wide variety of training conditions. The results across the three models have shown that Adam optimizer achieves 97% of test accuracy was the best optimizer while Adam label smoothing achieves similar to 96.5% and RMSProp achieves 95% of test accuracy shows less performance. Explainability of model in gradcam shows adam with proposed custom lightweight model understand the exact features of weed.Keywords: Weed classification, Hybrid CNN-Transformer, Squeeze-and-excitation blocks, Precision agriculture, Grad-CAM + + explainability, Agricultural computer visionSubject terms: Computational biology and bioinformatics, Mathematics and computing, Plant sciences Introduction Productivity of crops can be impacted by the quality of weeds removed from fields before crop emergence when plants compete for resources (e.g. nutrient uptake, light, water). Manual removal of weeds can be labor intensive and is not practical for widespread use. As such, researchers and scientists are interested in using computerized or automated visual inspection methods to identify weeds growing among crops1. Recently, CNN’s have been shown to perform well in classifying plants/weeds using images collected of crops2. They excel in recognizing hierarchically organized spatial patterns. However, architectural designs used for training CNNs have an important impact on their performance2,3. One aspect of CNN architecture that has a major effect on performance is the depth of networks. Deeper networks allow the model to recognize much finer grain features and therefore, can distinguish very closely related weed species growing together during the early growth stages4. A disadvantage of using deeper architectures is the potential for greater computational costs, vanishing gradients and overfitting, which may make them impractical to use for real-time processing in low power devices typically used for precision agriculture5. Lightweight CNNs have become popular due to their ability to provide high levels of accuracy while also being computationally inexpensive. However, several studies have found that some lightweight models sacrifice their ability to represent the features of interest when trying to reduce their size and/or complexity. Many conventional CNN based methods tend to only learn local features from the data and do not possess the ability to capture the longer-range dependencies or the global relationships between objects in the image. These types of relationships are critical for developing robust methods for classifying images captured in real world field settings. Hybrid architectures3,4 that combine CNNs with attention mechanisms, such as transformers, are emerging as one method to overcome these limitations. Transformers enable CNNs to process and recognize global interactions through self-attention, which complements CNN’s capabilities for processing and recognizing local features. Channel attention mechanisms, such as squeeze-excitation (SE) blocks5, further improve the discriminatory abilities of features learned by CNNs by dynamically adjusting channel-wise response values. While these recent developments6 show great promise, there remains a need to develop architectures that effectively integrate the strengths of CNNs for learning depth-wise features, attention mechanisms for capturing global contextual relationships and computational efficiency into a single lightweight architecture. This research introduces a new lightweight CNN-SE-Transformer architecture designed to systematically investigate how CNN depths affect computation efficiency. The proposed architecture utilizes residual connections to facilitate stable feature learning at multiple scales and to mitigate the effects of vanishing gradients, allowing for the effective investigation of larger CNN depths without sacrificing too much performance. The SE module allows the model to emphasize channel-wise feature representations that are useful for discriminating weeds. Meanwhile, the transformer encoder enables the model to process and extract global contextual relationships through its token representation mechanism. Unlike traditional methods that create redundant tokens, our architecture minimizes token redundancy, thus allowing efficient modeling of global relationships with minimal computation expense. Experiments were conducted to fully evaluate our model on two benchmark weed classification datasets consisting of 3-class and 16-class categorizations. We also analyzed three different optimization methods (Adam, RMSProp and Adam with label smoothing) to determine how they affected convergence behavior and overall generalization of our model. Finally, we utilized Grad-CAM-based explanation to demonstrate whether our model was able to focus on areas of an image relevant to identifying weeds. The primary contributions of this research include: Developing a new lightweight CNN-SE-Transformer architecture that maintains both the advantages of CNN’s depth-wise feature learning and the efficiency required for real-world applications in precision agriculture. Developing a new lightweight CNN-SE-Transformer architecture that maintains both the advantages of CNN’s depth-wise feature learning and the efficiency required for real-world applications in precision agriculture. Providing a comprehensive analysis of how varying CNN depth affects feature learning and how residual connections improve feature learning. Providing a comprehensive analysis of how varying CNN depth affects feature learning and how residual connections improve feature learning. Introducing channel attention (SE) and transformer-based global context modeling within a single framework. Conducting extensive evaluations on multi-class weed datasets along with explanations of why certain optimizers are better than others and providing an analysis comparing optimization level performance and explanations. Introducing channel attention (SE) and transformer-based global context modeling within a single framework. Conducting extensive evaluations on multi-class weed datasets along with explanations of why certain optimizers are better than others and providing an analysis comparing optimization level performance and explanations. Proposed results demonstrate that by combining increased CNN depth with efficient architectural design and attention mechanisms, we can greatly increase weed classification performance without increasing computational requirements, and as such will likely support the development of additional real-world precision agriculture applications. Literature review Recent years have seen substantial progress in deep learning for weed detection in sorghum and other crops. Rahman et al.7 developed an automated weed and crop recognition system using deep learning optimized with Lemrus Optimization, achieving high performance across diverse field conditions. Sharma et al.8 introduced a multi-scale feature-enhanced DenseNet for sorghum weed classification, improving resolution-aware detection but facing challenges in cluttered backgrounds and visually similar weeds. Semantic segmentation approaches have also shown effectiveness. Lee et al.9 designed a two-stage segmentation pipeline to separate tobacco and weed pixels in UAV imagery, achieving improved localization accuracy. Zhang et al.10 implemented a multiclass semantic segmentation framework using encoder–decoder networks to distinguish multiple weed species. While these methods perform well, they require extensive pixel-level annotation, which is resource-intensive. To enhance efficiency and adaptability, Li et al.11 proposed RLCFE-Net, a re-parameterized large convolutional kernel architecture capable of performing robustly across varied weed datasets. Nguyen et al.12 developed lightweight CNNs for real-time operation under challenging lighting, and Ahmed et al.13 leveraged attention-augmented Vision Transformers to capture both local and global features effectively. Hybrid CNN-Transformer models have also been employed in broader agricultural applications. Liang et al.14 applied such a network for plant disease and weed classification, achieving high accuracy by fusing local and global feature learning. Wang et al.15 provided an extensive survey on Vision Transformer applications in agriculture, detailing performance benchmarks and deployment strategies. Finally, explainability is crucial for the practical adoption of AI in agriculture. Ghosh et al.16 used Grad-CAM + + to interpret CNN-Transformer outputs18–20, highlighting biologically relevant patterns such as leaf edges and venation, thus enhancing model transparency and agronomic trust. Our proposed CNN–Transformer–SE model builds upon these insights by combining multi-scale feature extraction, Transformer-based global context, SE channel recalibration, and Grad-CAM + + interpretability to deliver accurate and explainable sorghum weed classification. Table 1 shows the survey of existing models. Table 1Shows the references details.Ref.Authors & yearMethodologyDatasetKey featuresPerformance 7 Rahman et al., 2025Deep learning + Lemrus OptimizationField imagesOptimized for robustness across varying conditionsHigh accuracy (~ 95%) 8 Sharma et al., 2023Multi-scale DenseNetSorghum imagesResolution-aware feature extractionImproved detection (~ 93%) 9 Lee et al., 2021Two-stage semantic segmentationUAV images (tobacco fields)Pixel-level localizationImproved localization accuracy (~ 91%) 10 Zhang et al., 2022Encoder–decoder segmentationMulti-weed datasetMulticlass semantic segmentationAccurate segmentation (~ 92%) 11 Li et al., 2024RLCFE-NetMultiple weed scenariosLarge convolutional kernels; efficient feature extractionRobust across diverse conditions (~ 94%) 12 Nguyen et al., 2024Lightweight CNNField imagesReal-time operation under variable lightingFast inference (~ 88%) 13 Ahmed et al., 2024Attention-augmented Vision TransformerField imagesCaptures local and global contextHigh accuracy (~ 96%) 14 Liang et al., 2023CNN–Transformer hybridMixed plant/weed datasetsCombines local–global featuresHigh accuracy (~ 96%) 15 Wang et al., 2024Vision Transformer surveyMultiple datasetsBenchmarks, challenges, deploymentComprehensive review (F1 95–99%) 16 Ghosh et al., 2024CNN–Transformer + Grad-CAM++Weed datasetsExplainability via Grad-CAM++Transparent model decisions; accuracy ~ 94% Shows the references details. Methodology Figure 1 illustrates how the structure of the proposed weed classification system will be organized. First, raw weed image(s) are acquired. Next, the images undergo an ordered, pre-processing method that improves image quality and provides standardized input. All images undergo resizing and normalization to assure consistent spatial dimensions and intensity values. To provide improved visibility of features, CLAHE contrast improvement and unsharp masking to refine edges are implemented to enable better distinction of subtle morphological characteristics across each weed class. Following this step, several data augmentation techniques (rotation, flipping, scaling, adjusting brightness) are performed on the processed dataset to increase the amount of diverse data available for use during model training, thereby enhancing generalization capabilities in various environmental conditions (i.e., field). The proposed architecture of the model in detail flow. Next, the processed dataset is randomly separated into two distinct groups; namely, a group intended for use during model parameter learning (training), and a second group intended for use to evaluate the model’s performance without bias (testing). At the heart of the proposed weed classification system is a novel, lightweight CNN-SE-Transformer architecture. A CNN backbone enhanced with residual connections is developed to extract hierarchical spatial features focusing specifically upon assessing the influence of increased CNN depth on feature representation. SE blocks are embedded within CNN layers to recalculate channel-wise feature response values, therefore allowing for the improved ability of the model to discriminate. Lastly, the extracted spatial feature maps are transformed into token representations and subsequently passed through a Transformer encoder to assess global contextual relationships that are often overlooked by traditional CNNs. Thus, the proposed hybrid design offers the model the opportunity to incorporate both localized feature extraction methods combined with global attention mechanisms while maintaining computationally efficient processes. The model is trained using three different optimization techniques (Adam, RMSProp, Adam w/Label Smoothing) to evaluate both convergence behavior and stability when trained under multiple conditions. After the model has been successfully trained, it is evaluated based upon standard performance metrics and the decision-making process of the model is visually interpreted using Grad-CAM. The combination of preprocessing, augmentation, lightweight feature extraction and attention-based modeling ultimately provides a robust weed classification framework that can accurately classify weeds in real-world agricultural environments. Dataset description Dataset 1 This study employs three publicly available datasets to develop and evaluate the WeedVis hybrid deep learning model for weed classification in sorghum fields, ensuring robust assessment across diverse conditions from https://data.mendeley.com/datasets. The Sorghum WeedDataset Classification serves as the primary dataset, consisting of 4,312 high-resolution RGB images collected from sorghum fields under varying lighting and environmental conditions in link https://data.mendeley.com/datasets/4gkcyxjyss/1. Each image is labeled at the image level as one of three classes: sorghum, grass weed, or broadleaf weed, capturing different growth stages and weed infestation levels to provide realistic field scenarios for practical model training and evaluation. To further assess the model’s generalization and scalability, the MH-Weed16 dataset is utilized from https://data.mendeley.com/datasets/d3n3mgjjbv/1, containing 19,141 images representing 16 distinct weed species commonly observed in agricultural settings. The images exhibit substantial variability in plant appearance, background complexity, and illumination, challenging the model to distinguish closely related and visually similar species. Dataset 2 The classification of the crops and weeds image dataset (SorghumWeedDataset_Classification) is being created for the area of precision agriculture and automated weed classification for research purposes. The dataset contains 4,312 RGB images of three classes: Sorghum seedlings (Class 0), Grasses (Class 1), and Broadleaf weeds (Class 2). The dataset is balanced and the distribution of samples is as follows: 1,404 sorghum samples, 1,467 grass samples, and 1,441 broadleaf weed samples, which allows for minimal class imbalance. The data is publicly available on https://data.mendeley.com/datasets/4gkcyxjyss/1. The same camera (Canon EOS 80D DSLR) with the same 22.3 mm × 14.9 mm CMOS sensor was used to capture all images and to attain fine detailed features, crop and weed species would be distinguished. The images are originally saved in the JPEG format and each image is approximately 13 MB and has a resolution of 6000 × 4000. Each image has been scaled down to 224 × 224 pixels, which makes the dataset appropriate for deep learning-based classification frameworks. The images will contain all of the necessary details, but all images will be optimized to take up as little computer storage space as possible. This dataset relates to the first stages of growth of the crops and weeds and to the problems of the critical period of weed competition, as timely recognition is essential. A variety of agricultural settings were used for the acquisition of the data. These settings included a variety of crop spacing, as well as varying light, weather, and distances between the camera and crop. All these factors contribute to the dataset being more applicable to real-world scenarios. For the sake of consistency and reproducibility across experiments, a standard Train–Validation–Test (TVT) split ratio of 70:20:10 was adopted in dataset 1 and 70:15:15 ratio as train/validation/test for dataset 2. Each image is labeled with the appropriate class for supervised learning tasks. From the SRM Care Farm, Chengalpattu district, Tamil Nadu, India, and during the months of April and May in the year 2023, data was obtained. To the best of our knowledge, this dataset is one of the first open-access datasets of the classification of crops and weeds in Indian agricultural fields. The dataset captures both the structured and unstructured layouts of planting.Table 2 shows the dataset details. Figures 2 and 3 show sample figures from the dataset. Table 2Dataset details.S.noDatasetUsageTotal imagesClasses1.SorghumWeedDataset classificationTrain+Validation+testing4,3123 (Sorghum, Grass weed, Broadleaf weed)3.MH-Weed16Train+validation+Testing19,14116 weed species Dataset details. Dataset 1 sample images (a) Sorghum (b) Grass weed (c) Broadleaf weed. Dataset 2 Sample images among 16 classes of weed species Cassia and Dudia. The dataset 1 and data set 2 is shown in Tables 3, 4 and 5 below with split count. Table 3Dataset 1 split (70:20:10).Weed/plantTrainValidationTestSorghum983281140Grass1027293147Broadleaf1009288144Total3019862431 Dataset 1 split (70:20:10). Table 4Dataset 2 split (70:15:15).Class IDWeed class nameScientific nameNumber of images0Kena Commelina benghalensis 17041Lavhala Cyperus rotundus 11772Lamber Quarter plantChenopodium spp.8363Little Mallow Malva parviflora 21004Moti dudhi Euphorbia geniculata 30025Obscure morning glory Ipomoea obscura 16376Asian Pigeonwings Clitoria ternatea 6787Bilayat Argeone mexicana 8598Choti dudhi Euphoria hirta 5809Digitaria SP Digitaria nguinalis 156110Gajar gavat Partheniumhysterophorus 154311Graceful Sandmat Euphorbia hypericifolia 26212Sicklepod Senna obtusifolia 64313Harali Cynodon dactylon 70914Dwarf cassia Chamaecrista pumila 130615Punarnava Boerhaavia diffusa 544Total19,141splitTrain:13,392 Val: 2864 Test: 2885 Dataset 2 split (70:15:15). Table 5Dataset 2 split class wise.ClassTrainValidationTestKena (Commelina benghalensis)1192255257Lavhala (Cyperus rotundus)823176178Gajar gavat (Parthenium hysterophorus)1080231232Graceful Sandmat (Euphorbia hypericifolia)1833940Sicklepod (Senna obtusifolia)4509697Harali (Cynodon dactylon)496106107Dwarf cassia (Chamaecrista pumila)914195197Punarnava (Boerhaavia diffusa)3808183Lamber Quarter plant (Chenopodium spp.)585125126Little Mallow (Malva parviflora)1470315315Moti dudhi (Euphorbia geniculata)15004501052Obscure morning glory (Ipomoea obscura)1145245247Asian Pigeonwings (Clitoria ternatea)474101103Bilayat (Argemone mexicana)601128130Choti dudhi (Euphorbia hirta)4068787Digitaria SP (Digitaria sanguinalis)1092234235 Dataset 2 split class wise. Data pre-processing The input images are represented as . All images were resized to 224 × 224 pixels to create consistency within the input dimensions and to be compatible with the proposed CNN-SE-Transformer network structure. Resizing reduces the computational complexity required to process images, yet still maintains enough spatial information needed to extract discriminable features from the images. The resized images will be in Contrast Enhancement Using Local Histogram Processing is second step. Field images often contain considerable amounts of light variation which creates contrast issues throughout the image. These types of issues can easily cause a machine learning-based system to fail when attempting to identify specific objects or features contained in the images. Therefore, we have implemented a method called Contrast-Limited Adaptive Histogram Equalization (CLAHE), which provides localized adjustments to the contrast within each region of an image. Unlike traditional histogram equalization techniques that make global changes to an image’s contrast, CLAHE applies these same adjustments locally rather than globally. CLAHE also prevents excessive amplification of noise. Where I(x,y) is the original pixel intensity, is the clipped cumulative distribution function of the tile’s histogram, and L is the number of possible intensity levels (256 for 8-bit images). The enhanced tiles are then bilinearly interpolated to avoid visible seams. Specifically, our implementation converts each input RGB image to the LAB color space. The L-channel represents the luminance portion of the image while the AB channels represent the colors red-green and blue-yellow respectively. We apply CLAHE only to the L-channels and utilize a 2.0 clip limit and an 8 × 8 tile grid. Once applying CLAHE to the L-channels, we recombine them with their corresponding A/B-channels, convert them back to RGB-space and send the resulting image through the rest of our pipeline. Our use of CLAHE significantly improves the visibility of finer details in plant morphology such as veins and edges that help distinguish visually similar weed species. Unsharp Masking is used next to provide additional refinement of the fine details and higher frequency components found in an image. Unsharp masking works by creating an image whose fine details will be amplified. This is accomplished by subtracting a blurred version of the image from its original version. After subtraction, we amplify the difference to accentuate those areas containing higher frequency information (i.e., finer texture). Since this type of information is very important in identifying weeds at their early stages where morphological differences may be almost non-existent, this step is important to emphasize this type of information in preprocessing. Where represents the gaussian blurring, represents standard deviation and sharpening factor as . Finally, this perform pixel intensity normalization on each of the preprocessed images so that the pixel values are scaled to a consistent range. While we normalize each image to the [0,1] range during intermediate processing steps, we need to convert the normalized images back to their original 8-bit unsigned integer format before saving them. This is done primarily for two reasons; (1) Storage Efficiency: Saving 8-bit unsigned integers requires less memory space compared to saving floating-point numbers, and (2) Compatibility with Standard Image Formats: Most standard image file formats require 8-bit unsigned integers. The Figs. 4 and 5 describes the visualization of the weed in different preprocessing stages. Pre-processed images of dataset 1 in different levels are shown. Final CLAHE+unsharp with pixel level normalization. Pre-processed images of dataset 2 in different levels are shown. Final CLAHE+unsharp with pixel level normalization Data augmentation In order to obtain greater robustness and better generality of the proposed model, a wide-ranging data augmentation strategy has been developed for use on the training dataset after pre-processing. Because there exists great variability in agricultural field images caused by illumination, camera angle, plant orientation, scale and environment, augmentation is necessary in order to simulate realistic variations and allow the model to learn feature that are both invariant and discriminatory. For this research, a large number of augmentations have been made via an online augmentation system. The specific geometric and photometric transformation used include rotation (± 45 degrees), to create invariance to plant orientation, since plants can be located anywhere in a field in reality. A 30% spatial shift of width and/or height was also added to create positional variability, so that the model would not depend on the location of features. Additionally, shears of 0.25 were created to represent slight geometric distortion or perspective variations that could exist when acquiring images. A zoom operation of 25%, representing varying scales of growth in weed development as well as distance from the camera was also included. Horizontal and vertical flip operations have also been added to create even greater variability in the data and provide symmetry-based invariance; useful for plant structures that contain similar patterns under reflection. Photometric augmentation was then added to simulate the variation in illumination present in many outdoor agricultural environments through application of random brightness adjustment in the range of 0.7–1.2. It should be noted that the “fill mode” of the new pixel regions generated by all geometric transforms will always be set to “reflect”, thus avoiding the creation of visual inconsistencies or artificial artifacts in these areas. The generation of new training samples for each input image occurs dynamically during each iteration of training. As a result, the amount of new data created per iteration results in the total effective size of the training dataset being much larger than actual size and minimizes the likelihood of over fitting and maximizes the model’s generality. Figures 4 and 5 provides several example original images along with their respective augmented versions from various classes. Table 6 shows the settings of augmentation used ion this research. Table 6Augmentation techniques and values.Step no.TechniqueParameters1Random rotationRange = ± 45°2Width shiftRange = 0.33Height shiftRange = 0.34Shear transformationFactor = 0.255Zoom augmentationRange = 0.256Horizontal flipEnabled7Vertical flipEnabled8Brightness adjustmentRange = (0.7, 1.2)9Fill modeReflect Augmentation techniques and values. Proposed WeedVis architecture The proposed model employs a light and efficient hybrid CNN-Transformer architecture for the task of accurate image classification. The network features a multi-stage convolutional backbone with three hierarchical blocks. Each of these three blocks has ascending channel dimensions for feature maps in the range of 32, 64, and 128. Each of these three blocks has convolutional layers that include batch normalization and a ReLU activation function in order to get the best feature extraction at a local level. In order to enhance the flow of gradients and help with the stabilization of training, the use of residual connections has been incorporated within the three hierarchical blocks, one per block (Fig. 6). Weed images with augmentation process. Also, the use of Squeeze-and-Excitation (SE) Attention is incorporated after the completion of each one of the three convolutional blocks in order to address the selective focusing of one or a few features, and improve the discriminative quality of the features. The task of level of the network is to extract and identify high-level semantic features, and at the same time, maximize an increase in the network’s performance by means of computational minimalism.Before the use of the final layers of the network is implemented, the task of transforming the resulting feature maps is done. This has a major impact on the performance of the network. Once this is accomplished, the learned features can be pooled into a global average that has been further modified by the use of a dropout layer in order to minimize overfitting. This is done in order to achieve an Objective Function. All in all, the suggested architecture is able to use a minimal number of resources while still being able to localize and classify features in a highly efficient manner. Architecture Figure 7 shows the proposed model for the classification of sorghum weeds based on the hybrid CNN-SE-Transformer architecture that attempts to minimize the number of computations required while still capturing the essential local spatial features (in this case, the local weeds), as well as the more distant global context (in this case, the distant weeds). The only input to the network is a color image of dimensions 224 × 224 × 3. Proposed Hybrid deep learning CNN-SE-TRANSFORMER architecture. The architecture consists of a convolutional backbone with three stages. Each of the three stages consists of a number of convolutional layers, and at each new stage the number of channels in the layers is increased to 32, 64, and finally 128. In the initial stage, for example, a 3 × 3 convolution is applied with 32 filters, and then the result is batch normalized and ReLU activated in the following way where I refers to the input image, and * denotes the convolutional operation. In contrast to the baseline model, the proposed architecture has many more convolutional layers and residual connections (which are particularly helpful for constructing very deep networks) within each stage, and also more max layers, and more max pooling layers, for spatial resolution reduction. In order to improve feature representation, we apply an SE block after each stage for recalibrating channel-wise responses: where is obtained through two fully connected layers with a reduction ratio (r = 8) and sigmoid activation. This mechanism amplifies the informative features and diminishes the less informative features. The second and third stages have a similar structure with 64 and 128 filters respectively, enhancing feature richness while diminishing spatial dimensions. This design, when compared to the prior model, introduces more complex deep hierarchical feature learning with residual connections that greatly enhances representation ability. The processing of the feature maps beyond the CNN-SE backbone includes a convolutional projection where we reshape the maps into a sequence of tokens. To maintain spatial information, positional encoding is incorporated prior to sending the sequence to the Transformer encoder. To enhance computational efficiency, the sequence of tokens is reduced to N = 196 (14 × 14) compared to the baseline. The Transformer module has two encoder blocks that use multi-head self-attention with 4 heads. The attention mechanism is described by: where Q, K, and V represent the matrices for query, key, and value. Each block has a feed-forward network (FFN) with 128 hidden units: To improve training and convergence, the training process is made stable with the use of residual connections and layer normalization after each attention and FFN layer. The sequence is first processed by the Transformer encoder, and then the entire image is represented compactly as a feature vector derived from the Global Average Pooling. Finally, a dropout layer with a rate of 0.3 is applied for regularization. The last classification layer contains three output units for the softmax dense layer and the classes sorghum, grass, and broadleaf weed classes: The model uses categorical-cross entropy loss and three famous optimizers like adam, RMSProp, adam+labelsmoothing. To retain optimal weights and avoid overfitting early stopping and model checkpointing are utilized. The model was evaluated on three optimizer and its performance was noted in result section. The lightweight architecture clearly demonstrates the ability to merge efficient Transformer-based global modeling, deep convolutional feature extraction, and channel-wise attention, all of which enables optimal classification performance to be achieved alongside minimized operational overhead. For additional studies on multi-class weed classification, the last layer is adjusted to have 16 classes, which is represented as follows: Architecture was shown in Fig. 7; Table 7 shows the architecture summary in detail. Table 7architecture summary in detail.StageLayer/blockOutput sizeParametersInputInput layer224 × 224 × 30Stage 1Conv2D (32) + BN + ReLU224 × 224 × 321,024Conv2D (32) + BN224 × 224 × 329,376Residual connection224 × 224 × 32128MaxPooling112 × 112 × 320SE attention block112 × 112 × 32292Stage 2Conv2D (64) + BN + ReLU112 × 112 × 6418,752Conv2D (64) + BN112 × 112 × 6437,184Residual connection112 × 112 × 642,112MaxPooling56 × 56 × 640SE attention block56 × 56 × 641,096Stage 3Conv2D (128) + BN + ReLU56 × 56 × 12874,368Conv2D (128) + BN56 × 56 × 128148,096Residual connection56 × 56 × 1288,320MaxPooling28 × 28 × 1280SE attention block28 × 28 × 1284,240Stage 4Conv2D projection14 × 14 × 128147,584Reshape196 × 1280Positional encoding196 × 12825,088Transformer block 1LayerNorm + multi-head attention196 × 128131,968Add & norm196 × 128256Feed forward network196 × 12833,024Transformer block 2LayerNorm + multi-head attention196 × 128131,968Add & norm196 × 128256Feed forward network196 × 12833,024ClassifierGlobal average pooling1280Dropout1280Dense (Softmax)3387 architecture summary in detail. Table 6 Model summary of proposed CNN-SE-TRANSFORMER. The model is trained using the three optimizers with an initial learning rate of and categorical cross-entropy loss. Data augmentation, including random horizontal/vertical flips, brightness and contrast adjustments, and random rotations up to 15, is applied during training to improve robustness against environmental variations. Early stopping and learning rate reduction on a plateau are employed to prevent overfitting. Model performance is evaluated using accuracy, precision, recall, and F1-score across all datasets, with special emphasis on generalization performance on unseen ROI images from the segmentation dataset. There are two models in this research. Experiments and results Experimental setup The experiments were conducted on a system equipped with a T4 GPU in Kaggle environment with the deep learning framework. The dataset was split into training, validation, and test sets with a ratio of 70:20:10(dataset1), 70:15:15(dataset2). All models were trained with a batch size of 32, and for 100 epochs. Optimization was performed using Adam (1 × 10⁻⁴) and categorical cross-entropy as the loss function. Data augmentation was applied on-the-fly during training to enhance model generalization, and early stopping was employed to prevent overfitting. Data augmentation techniques are included in Tables 3, 4 and 5. In these experiments, we used dataset1 for training and testing the model. The dataset details for testing and training are shown in Table 7 below. Dataset 2 is 16 class weed images with annotations. Figure 8 shows the distribution of the dataset 2 in detail. As original dataset comprises of 19,000 images we didn’t perform augmentation. Model was purely trained on original dataset. Dataset 2 distribution on 16 classes is shown. Performance analysis In Table 8, we provide the training, validation, and test performance, with the proposed model and various optimizers, for the three independent runs. The results show that the Adam optimizer has the training accuracy and the validation score almost at the maximum, with the test accuracy in the range of 94.90% to 97.45% (≈ 98.7%–99.3%). RMSprop shows comparatively lower performance, with the performance being more variable, especially in Run 2, where we see a drop in the validation score (93.04%) and test score (89.79%) for the validation accuracy. This shows Adam-based optimization has more stable (less variable) convergence and more stable performance in the various optimization parameters. Table 8The performance evaluation of the proposed model with different optimizers with multiple runs on dataset 1.ModelRunsTraining lossTraining accuracyValidation lossValidation accuracyTest accuracyTest lossProposed +adamRun 10.19360.99240.21510.98610.97450.2340Proposed +adamRun 20.20220.99300.23310.97910.96520.2542Proposed +adamRun 30.21270.98710.24020.97100.94900.2671Proposed +RMSPropRun 10.04270.98580.07240.97450.95590.0893Proposed +RMSPropRun 20.09180.97350.18950.93040.89790.4562Proposed +RMSPropRun 30.04120.98480.08030.97220.92580.1815Proposed +adam_label SmoothingRun 10.20220.99270.23340.97910.96290.3642Proposed +adam_label SmoothingRun 20.21270.98710.24060.97100.94900.2680Proposed +adam_label SmoothingRun 30.20480.99240.24230.97450.96750.2559 The performance evaluation of the proposed model with different optimizers with multiple runs on dataset 1. When model consider Adam with label smoothing, results achieve similar approximations as with standard Adam. Here it’s noted that the test accuracy remains bounded between 94.90% and 96.75%. In a few of the runs, label smoothing does appear to achieve a marginal improvement in the generalization, and noted improvement to be of almost no significance. In these experiments, Adam-based methods provide the best variance and show the least bias concerning convergence, accuracy, and stability. Here, the proposed lightweight CNN-Transformer architecture Adam optimization does provide the best results, while on the other hand, RMSprop optimization does provide the least consistent results. The results from Table 9 detail the outcomes from different optimization strategies of the recommended model on the MH-Weed 16-class dataset during three separate runs. For the optimization strategy of Adam, results show fairly consistent overall performance with test accuracies in the range of 84.50% to 86.66% and with validation accuracies showing even less variability in the range of approximately 84.11% to 86.56%. This illustrates that the model has the ability to consistently converge and offers fairly consistent generalization. Table 9The performance evaluation of the proposed model with different optimizers with multiple runs on dataset 2.ModelTraining lossTraining accuracyValidation lossValidation accuracyTest accuracyTest lossProposed +adamRun 10.25130.90600.40600.86560.86660.4244Proposed +adamRun 20.34440.87700.47850.84110.84500.4131Proposed +adamRun 30.25120.90950.40590.86550.86640.4313Proposed +RMSPropRun 10.34170.87540.47620.84990.85230.4787Proposed +RMSPropRun 20.36000.86900.44660.85750.84470.4777Proposed +RMSPropRun 30.43790.83890.54160.81810.84800.5421Proposed +adam_label SmoothingRun 10.66310.91060.89080.87530.87800.8840Proposed +adam_label SmoothingRun 20.73620.88580.93900.85470.85720.9455Proposed +adam_label SmoothingRun 30.74560.91430.94670.85120.85060.8964 The performance evaluation of the proposed model with different optimizers with multiple runs on dataset 2. In optimization strategy RMSprop, results show significantly lesser performance, and even more that lower performance consistently. Run 1 RMSprop achieves competitive performance at 85.23% accuracy however noticeable drops in performance are observed in subsequent runs. In Run 3, validation accuracy fell to 81.81%. This indicates less stability of convergence, generalization, and training behavior, among other factors Adam optimizations. Adam with label smoothing has the greatest test accuracy among all other optimization strategies at 87.80% in Run 1. This demonstrates that in some cases, generalization could certainly be improved, however, performance in and across runs does not match the drop in 85.06% test accuracy which illustrates the improvement must of must certainly encompass the overall less variability test accuracies. This in combination with the even greater loss values observed from label smoothing which is in and of itself regularization demonstrates that less variability is to be expected. As overall observed performance on the MH-Weed dataset, Adam methods provide the greatest improvement with the greatest consistency. Label smoothing does not seem to greatly improved accuracy however, the marginal improvements lead little to no variation. Most importantly, all results show that the recommended model maintained fairly consistent stability across Adam optimization strategies which once again emphasizes that the model sustained expected performance across different optimization strategies. Table 10 provides results for the statistical analysis of various optimization methods for the proposed lightweight model from three independent runs. The results are broken down by mean accuracy and standard deviation to show both performance and consistency. Adam with learning rate scheduling (Adam + LR) has the highest mean accuracy at 96.29%, and has thus been chosen as the baseline. Adam with label smoothing (Adam + LS + LR) has a comparable accuracy of 95.98%, but the standard deviation of 0.79% is the smallest, meaning the level of consistency and generalization is highest. RMSprop mean accuracy is only 92.65%, and has a high variability of 2.37%, meaning that convergence for the proposed architecture is the worst. Table 10Statistical test (adam as baseline)-dataset 1.MethodAccuracy (%)Std (%)p-value (vs. Adam)Cohen’s dInterpretationAdam + LR96.291.05––Best accuracyAdam + LS + LR95.980.790.07200.28Similar performance, more stableRMSprop + LR92.652.370.1311.89Lower accuracy, large performance gap Statistical test (adam as baseline)-dataset 1. In this analysis, Adam is the baseline and used to show the results of the paired t-test for statistical significance. Adam and Adam + LS have a high p-value of 0.720 which means there is no significant difference and Adam has no real value. This means that Adam has a significant value when compared to RMSprop and p-value. This is due to the 3 runs. This shows that Adam is the best optimizer when considering accuracy and Adam with label smoothing is best option considering stability and provides consistent optimizations. The model we propose has a total of around 0.52 million parameters. This means that it has a model size that is approximately 100 times smaller than some of the deep learning models being used today. This allows a very compact model that consists of lightweight convolutional blocks, squeeze and excitation attention, and a transformer encoder that is very shallow. For feature extraction, it provides a reasonable trade-off in cost. There is also a very special computational cost that occurs due to the use of the Adam optimizer. This special computational cost is a result of training that is performed in a very stable manner and a cost that is due to the total time spent optimizing the model. For the optimizer, Adam also proved to be a very reliable optimizer because it has a very low variance and was able to achieve a very high accuracy. In general, in comparison with RMSprop, Adam was able to achieve a high accuracy with a very low variance which means that it was able to be very reliable and very efficient. The use of a learning rate scheduler is consistent with the other design requirements. Overall, these features reflect the fact that we have designed a model that is lightweight. This model is also very computationally efficient due to the design of the features within the normal operating parameters of the model. Table 11 compares the different optimization strategies on Dataset 2, taking the Adam optimization strategy as a reference point. Adam shows a high degree of accuracy and consistent performance with mean accuracy of 85.93% with very small standard deviation, meaning its converging behaviour is very stable. Adam with label smoothing shows a slight increase in mean accuracy to 86.19%. However, this is not a statistically significant improvement with a p-value of 0.620, and given the small effect size of Cohen’s d = 0.22, both in practical terms, this is not an improvement. Table 11Statistical test (adam as baseline)-dataset 2.MethodAccuracy (%)Std (%)p-value (vs. Adam)Cohen’s dInterpretationAdam + LR85.931.00––Strong performance, best convergenceAdam + LS + LR86.191.150.6200.22Slightly higher accuracy, similar performanceRMSprop + LR84.830.310.1801.30Lower accuracy, large performance gap Statistical test (adam as baseline)-dataset 2. On the other hand, RMSprop has a mean accuracy of 84.83% which is not as high as Adam, but does have the lowest variance among the different runs. The p-value of 0.180 is not statistically significant but the large effect size of Cohen’s d = 1.30 indicates a practical difference in this case to Adam for explaining this underperformance. Therefore, it can be stated that based on the number of runs, the statistical power is that RMSprop is consistently performing less than Adam based optimization strategies. There is sufficient evidence to conclude that optimization based on Adam is a good trade-off between accuracy and converging for the algorithms, other than improvement which can be expected from label smoothing. Hence, these results validate choice of Adam as the optimization strategy that can be used for the proposed model for Dataset 2. Qualitative analysis on dataset 1 In Fig. 9, the training and validation accuracy and loss graphs from applying the Adam optimizer are depicted for three distinct runs on Dataset 1. Training accuracy shows marked improvement for most of the epochs, exceeding 90% in just a few iterations and faintly plateauing in the range of 98–99%. This indicates the model has learned and converged rapidly. The validation dropped loss follows training loss fairly closely and also confirmed this observation. Training loss has a smooth downward trend and validation loss showed a little bit of up and down most likely due to effects of the data. The fewer the difference in the level of training and validation accuracies, the better the balance between overfitting the data and generalizing the model. As shown in the figure, the Adam optimizer has variable performance for different runs. However, the variance in performance is associated with different levels of losses in the model. Thus, the performance loss of the model is stable and gives reliable values in the end. Proposed training and validation curve for adam optimizer-dataset 1. Figure 10 shows the training and validation loss for each of the three runs of the proposed model using the Adam optimizer with label smoothing for the first dataset. The training loss shows a decline and convergence toward lower training loss values as the number of epochs increases. This shows a stable optimization behavior. Standard Adam shows loss values which are expected to be lower than this. Training loss shows a regular behavior with label smoothing. Training loss has a similar behavior to validation loss with the exception of the fluctuations with validation loss. Label smoothing shows the opposite of overconfidence which helps generalization. Even with these fluctuations, the validation loss remains stable during the later epochs with less of a gap between training and validation loss. This indicates less of an overfitting behavior. The loss curves show that the configuration of Adam optimizer with label smoothing shows both stability and generalizability of the model with the control over the training and validation loss gap across all runs. Proposed training and validation curve for Adam_ls optimizer-dataset 1. The model-trained using the RMSprop optimizer overview from continual training/validation loss graphs in Fig. 11 for the three repeats documented for Dataset 1. The training loss demonstrates a continuous drop which shows learning the data has been successfully accomplished. But the behavior does not hold for the validation loss which shows a turbulent trend with especially obvious volatility for early- to medium- epoch models. A few of the modeled runs show extremely large volatility. Proposed training and validation curve for Rms_porp optimizer-dataset1. The validation loss volatility indicates RMSprop has trouble calibrating its generalization ability to time-variant data. Despite the fact the validation loss appears to finally calm down in the later epochs, the loss shows erratic, effects of disruption compared to the calm convergence seen with the Adam optimization methods. The phenomenon of the training loss and validation loss disparity shows at times, severe overfitting and insufficient learning. It can be generally assessed from the curves that the training convergence using the RMSprop optimizer is as good as the RMSprop algorithm validity concerns. Figure 12 shows the confusion matrices of the proposed model using the Adam optimizer on Dataset 1 over three different independent trials, with an impressive presence of diagonal values in all three runs. Such a phenomenon indicates a high degree of true positive class predictions. The sample class predictions were correct for the majority of the three classes. The class Grass surpassed the other classes with correct predictions, achieving an accumulation of more than 140 correct predictions across the model trials and signifying a thorough learning of the model features and the ability to differentiate between classes. The class Sorghum and the class BroadLeafWeed showed the same high class prediction accuracy. As these classes were confused, an even smaller sample prediction of the class Sorghum was predicted to be Grass or confused with BroadLeafWeed. This relative confusion is a phenomenon observed across all model iterations and shows a consistency of the model. Adam optimizer with proposed model in all three run predictions in confusion matrix -dataset 1. The matrices of confusion confirm that the Adam optimizer is capable of model class predictions with high confidence and less between class predictions, thus confirming the model to be accurate in predicting class labels for the different plant classes within Dataset 1. Table 12 shows the classification report in detail. Table 12Classification Report of proposed with adam -dataset 1. Significant values are in bold.ClassMetricRun 1Run 2Run 3Mean ± StdSorghumPrecision 0.9645 0.94960.94740.9538 ± 0.0076Recall 0.9714 0.94290.90000.9381 ± 0.0294F1-score 0.9680 0.94620.92310.9458 ± 0.0184GrassPrecision0.95970.98610.95920.9683 ± 0.0127Recall0.97280.96600.95920.9660 ± 0.0056F1-score0.96620.97590.95920.9671 ± 0.0068BroadLeafWeedPrecision1.00000.95950.94040.9666 ± 0.0249Recall0.97920.98610.98610.9838 ± 0.0033F1-score0.98950.97260.96270.9749 ± 0.0110Accuracy 0.9745 0.96520.94900.9629 ± 0.0105Macro Avg F1 0.9746 0.96490.94830.9626 ± 0.0108Weighted Avg F1 0.9746 0.96520.94860.9628 ± 0.0107 Classification Report of proposed with adam -dataset 1. Significant values are in bold. In Fig. 13, confusion matrices describe outcomes of three trials of the suggested model utilizing the Adam optimizer with label smoothing on Dataset 1. All trials show same strong diagonal dominance as results with the standard Adam. This indicates a high rate of successful predictions of the three classes. Grass and BroadLeafWeed remain well classified and are predicted correctly more than 140 times, which indicates excellent discriminability. Adam_ls optimizer with proposed model in all three run predictions in confusion matrix-dataset 1. When compared to standard Adam results, label smoothing has the predicted outcomes of overall less concentration in the Sorghum class, leading to more balanced prediction distribution. There also exists slight prediction toward BroadLeafWeed class. This is an expected outcome as label smoothing encourages the model to be less certain and to produce a less pronounced prediction. Other than this adjustment, the confusion matrices suggest a similar classification result across trials. The matrices show the prediction concentration and the presence of small consistent off-diagonal values indicative of an improved. Table 13 shows the classification report in detail. Table 13The proposed model with Adam_ls optimizer-dataset 1.ClassMetricRun 1Run 2Run 3MeanSorghumPrecision0.950.950.950.95Recall0.940.900.950.93F1-score0.940.920.950.94GrassPrecision0.990.960.990.98Recall0.970.960.970.97F1-score0.980.960.980.97BroadLeafWeedPrecision0.950.940.970.95Recall0.990.990.980.99F1-score0.970.960.970.97Accuracy0.960.950.970.96Macro Avg F10.960.950.970.96Weighted Avg F10.960.950.970.96 The proposed model with Adam_ls optimizer-dataset 1. Figure 14 shows confusion matrices from the proposed model with the RMSprop optimizer from three separate runs on Dataset 1. While the matrices show effects of the model’s ability to classify to a certain extent, the diagonally dominated elements show the model’s classification ability, the degree of consistency in performance compared to the Adam-based methods is evident. Run 2 had a marked increase in misclassification of the Sorghum class, with a large number of samples falsely predicted as BroadLeafWeed. Run 3 also had this same phenomenon, but is to a lesser degree. RMS_Prop optimizer with proposed model in all three run predictions in confusion matrix-dataset 1. The Grass class, on the other hand, being consistently well classified throughout the runs, with a high degree of true positive prediction, exemplifies a class that is easier to classify than the rest of the subjects. But the higher values of the Sorghum and BroadLeafWeed classes, which were classified erroneously, shows that there was a greater degree of confusion among the classes, which point to the fact that the classes were visually similar. This situation illustrates better the degree of inconsistency that RMSprop has when trained on the popular datasets. The confusion matrices depict the situation perfectly RMSprop is slightly accurate in terms of classification, but this classification results either going to be accurate or completely wrong as the number of runs increases. This depicts the situation perfectly, RMSprop is not the best optimizer compared to the Adam-based class of optimization functions when the proposed model is run on Dataset 1. The classification reports shown in Table 14. Table 14The proposed model with RMS_Prop optimizer-dataset 1.ClassMetricRun 1Run 2Run 3MeanSorghumPrecision0.940.920.930.93Recall0.930.790.850.86F1-score0.940.850.890.89GrassPrecision0.970.920.940.94Recall0.960.950.950.95F1-score0.970.930.940.95BroadLeafWeedPrecision0.950.860.900.90Recall0.980.960.940.96F1-score0.970.900.920.93Accuracy0.960.900.930.93Macro Avg F10.960.900.920.93Weighted Avg F10.960.900.930.93 The proposed model with RMS_Prop optimizer-dataset 1. The graphs labeled 15–20 show Receiver Operating Characteristic (ROC) and Precision–Recall (PR) curves for the proposed model with the Adam, Adam with label smoothing, and RMSprop optimizers, respectively. For all graphs, the ROC curves reflect high true positive rates and low false positive rates, suggesting the model has excellent multi-class classification ability. For the Adam optimizer (Figs. 15 and 16), the ROC curves are almost always near the upper left corner, whereas PR curves take the upper range suggesting high values of precision, indicating significant and stable performance across all the classes. For Adam with label smoothing (Figs. 17 and 18), there are similar ROC features, while PR curves show that there is a slight improvement in generalization for some classes due to removal of over confidence in the predictions. ROC curve of adam optimizer with proposed model-dataset 1. Precision recall curve of adam optimizer with proposed model-dataset 2. Precision-recall curve of adam_ls optimizer with proposed model-dataset 1. ROC curve of adam_ls optimizer with proposed model-dataset 1. On the other hand, the RMSprop optimizer (Figs. 19 and 20) shows almost the opposite in the ROC and PR performances and the curves are far from the preferred region with high variations throughout the runs. This shows that there is less consistency and a poorer separation of classes as compared to all variants of the Adam optimizer. Precision-recall curve of adam_ls optimizer with proposed model-dataset 1. Precision-recall curve of adam_ls optimizer with proposed model-dataset 1. The ROC and PR analyses conclude that Adam-based optimization, with label smoothing, is more consistent, and offers better classification performance than Robprop. Although RMSprop is a usable optimizer, it is less applicable and most optimization techniques provide more consistent reliability on the classification tasks than these optimizers. Qualitative analysis of dataset 2 The results of the 16-class weed dataset for the proposed model are clearly visualized in the training curves (Figs. 21 and 22), the confusion matrix (Fig. 23), the ROC analyses (Fig. 24), PR-Curve (Fig. 25) and the metrics per class. The training and validation curves illustrate further improvements across epochs with an approximate 88–90% training accuracy and 83–86% validation accuracy. This makes the model an example of stable convergence. The associated curves for accuracy demonstrated in these analyses appears to decline, with training accuracy decreasing smoothly and validation accuracy showing moderate perturbations. This suggests efficient learning with minimal underfitting. The ROC curves for all three analyses demonstrate good discriminative performance with many classes having curves that approach the top left corner with AUC values nearing 0.98–1.00.98.00. This shows the model to be a strong classifier, even in a multi-class scenario with a high degree of intra-class similarity. Proposed with adam optimizer- dataset 2 16 class. Proposed model with RMSProp-dataset 2 16 class. Confusion matrix of proposed model with adam optimizer -dataset 2. mh weed-adam optimizer with proposed based RoC-Curve. mh weed-adam optimizer with proposed based PR-Curve. The confusion matrix also showed a strong diagonal performance with high classification accuracy for most classes. Excellent performance was demonstrated for classification accuracy for the classes Harali (98%), LamberQuarter (93%), Asian_Pigeonwings(94%), and Bilayat (96%) indicating strong feature separability. The classes Choti_dudhi (62%) and LittleMallow (78%) showed low diagonal values indicating high misclassification. Of particular note was Kena and Choti_dudhi, which suggets a visual similarity that was a function of overlapping class morphology. In a general view, the results show strong multi-class classification capability of the proposed model, the lightweight CNN–SE–Transformer, especially for most classes with high/model consistent accuracy. The model shows good generalization with a small degree of confusion with a few visually similar weed species. In terms of accuracy, generalization, and computational efficiency, the model strikes a good balance which demonstrates its usability in real-life farming situations. The detailed results from the classification report in Table 15 show the performance for each individual class. The model demonstrates significant precision and recall for a number of classes, with Harali achieving perfect recall (1.000) and the best F1-score (0.985), followed by Bilayat (F1 = 0.958) and LamberQuarter (F1 = 0.952). On the contrary, Choti_dudhi has the lowest precision (0.527) and F1-score (0.628), which shows the class was the most challenging to identify. The performance of LittleMallow (F1 = 0.713) is considered to be moderate. This is possibly caused by the small number of samples, which affected the class and the intra-class similarity. Table 15The proposed model with adam optimizer-dataset 2 classification report on best run 1. Significant values are in bold.ClassPrecisionRecallF1-scoreSupportKena0.8830.7670.821257Lavhala0.8380.7870.812178LamberQuarter0.9690.9350.952232LittleMallow0.6600.7750.71340Moti_dudhi0.7820.8140.79897Obscure_morning_glory0.7690.8410.804107Asian_Pigeonwings0.9490.9440.947197Bilayat0.9520.9640.95883Choti_dudhi0.5270.7780.628126Digitaria0.9070.8700.888315Gajar_gavat0.9310.8940.912451Graceful_Sandmart0.9000.8420.870247Sicklepod0.8450.9510.895103Harali0.9701.000 0.985 130Dwarf_cassia0.8650.8850.87587Punarnava0.8170.8170.817235 The proposed model with adam optimizer-dataset 2 classification report on best run 1. Significant values are in bold. MetricValueAccuracy0.867Macro Avg Precision0.848Macro Avg Recall0.866Macro Avg F1-score0.855Weighted Avg Precision0.876Weighted Avg Recall0.867Weighted Avg F1-score0.869 The ablation experiment was conducted with this setting. Tables 15 and 16 report the results of the comprehensive ablation study that was performed to determine the effect of each individual component in the given architecture. A number of key modules were added in turn to the architecture, including the Transformer, SE attention, residual connections, and additional depth to the CNN. Table 16Ablation studies of dataset 1.ExperimentCNN depthResidualSE blockTransformeroptimizerTest accuracyE13 layers✗✗✗Adam32%E23 layers✗✗✓Adam58%E33 layers✗✓✓Adam88%E43 layers✓✓✓Adam88%E510 layers✓✓✓Adam97%E612layers✓✓✓Adam92%E78 layers✓✓✓Adam89% Ablation studies of dataset 1. 12 layers The model started with a shallow baseline CNN with three convolutional layers (E1). With this architecture, the model performed poorly (32% with Adam and 22% with RMSProp). This confirms that shallow architectures have little capacity to learn complex visual features. After adding the Transformer module (E2), there was a significant performance increase with the Adam with label smoothing configuration, now at 58%. This confirms the benefit of capturing global contexts. However, the shallow backbone still limits the performance that can be attained. The addition of the SE attention mechanism (E3) resulted in a significant performance increase (88% with Adam + label smoothing and 63% with RMSProp). This demonstrates the utility of feature recalibration to improve the discriminative power of representations. The addition of residual connections (E4) provided only a small increase in accuracy but contributed to a more stable training process and a gradual increase in the depth of features that were learned (Table 17). Table 17Ablation studies of dataset 1.ExperimentCNN depthResidualSE blockTransformerOptimizerAccuracyE13 layers✗✗✗Adam22%E23 layers✗✗✓Adam18%E33 layers✗✓✓Adam63%E43 layers✓✓✓Adam65%E510 layers✓✓✓Adam86% Ablation studies of dataset 1. The largest increase in performance was seen as a result increasing the CNN depth from 3 to 10 layers (E5). With this change, the model achieved 92% accuracy on Dataset 1 and 76% accuracy on Dataset 2. This points to the importance of deeper hierarchical feature extraction in the capture of complex visual patterns in the agricultural imagery. The ablation results corroborate that despite individual components like the Transformer and SE blocks adding positively, integration of all of them into a deeper CNN backbone result in optimal performance. In addition, Adam optimizer with label smoothing demonstrates improved convergence and generalization which is why it consistently outperformed other settings. Gradcam for dataset 1 using adam Gradcam dataset 2 using adam The Fig. 26 for dataset 1 shows adam optimizer understand the exact features. This explainability concludes that dataset with medium count like 5000, adam provides robust performance. Similarly, while training large images adam optimizers performs well in classification tasks as seen in Fig. 27. But model performance was light weight, generalized and scalable in both dataset. This proves proposed model can handle all type of weed data in short time. GradCam for dataset 1. Gradcam explainable figure for dataset 2. Table 18 describes the model/inference efficiency and economy of various standard lightweight models versus the proposed model. The proposed CNN–SE–Transformer model has a parameter count of only 2.1 Million, and a FLOP count of only 1.2 Billion. It is also very fast, with an inference time of only 18 milliseconds per image, which clearly makes it a very good candidate for real time usage in agriculture. In MobileNetV3 and EfficientNet-B0, The proposed architecture outperforms the cross point, where computational efficiency and classification effectiveness, for resource limited environments, with the most value. Table 18The parameters and inference time of proposed with standard backbone model for dataset 1.ModelParameters (M)FLOPs (G)Inference time (ms/image)HardwareAccuracyProposed model~ 2.1~ 1.218 msGPU (T4)97MobileNetV3~ 5.4~ 0.22~ 12 msGPU (T4)33EfficientNet-B0~ 5.3~ 0.39~ 15 msGPU (T4)99 (overfit) The parameters and inference time of proposed with standard backbone model for dataset 1. Inference results on unseen data The image Fig. 28. was taken from google on model prediction test. The original plant name is sorghum. The image was not preprocessed. It is tested on its original light condition. The proposed model identifies the sorghum plant with high probability. The prediction was broadleaf: 18.21%, grass: 47.56% and sorghum: 72.34%. Proposed model with adam best model weight analysis. Table 19 compares the results of existing works with proposed work. The proposed model is unique and lightweight without using any backbone and complex models. Table 19Comparison of existing works with proposed work.ModelDatasetAccuracyResnet 101 + ASPP17UAV data-Wisconsin90%Hybrid vit+ lite conv21Real field data88.4%GoogleNet21Real field data82%ResNet1821Real field data84%WeedNet22 MAE-TransformerWeed species multiple93%Proposed lightweight CNN-TransformerSorghum97%Proposed lightweight CNN-TransformerMHWEED-16class87% Comparison of existing works with proposed work. Discussion It is shown in the experimental results the proposed architecture of the CNN-SE-Transformer is lightweight across both datasets. In regards to the first dataset (3-class classification), the model reaches high accuracy (up to 93%) showcasing great potential discriminating ability for simple classification tasks. In the training and validation curves there is stable convergence and less than average overfitting especially for Adam with label smoothing, overconfidence in the predictions is reduced which improves generalization. On the contrary with the second dataset (16-class classification) a more complex model is proposed and it achieves a competitive outcome of 78% accuracy on the test with Adam and label smoothing, despite the class diversity and imbalance. This shows the gap of the performance of model in the first dataset to the second one is increased because of the imbalance class, with the increase in the similarity between classes, this confirms the high difficulty in weed classification in day-to-day agricultural environments, this is supported by the analysis of the confusion matrices. There’s abundant evidence from ablation studies assessing the effectiveness of individual structural components. This is particularly true for the case of Block Transformers, which enhance structural feature representation globally, whereas with SE blocks this enhances structural representation with an individual degree. Thus, it can be stated that the residual connections enable stable training of the architecture, especially when the components are increased in depth. Additionally, with the increase of structural design components in the CNN Block, increasing from 3 to 10 individual components gives significant performance increases. This demonstrates that in combination with an efficient design, the removal of structural components is highly additive to overall performance. The proposed CNN–Transformer model is lightweight in design and applies to multi-class classification of weeds and achieves stellar results. However, the biggest drawback of the current research is the reliance on RGB data, which only captures the visible light and can lose valuable data to fully capture the spectral dimensions of the data, especially when needing to identify weeds that are visually identical based on lighting and soil conditions and the growth stages. The recent literature find that multispectral information improves the performance of machine learning algorithm. The data RafanoSet (Rana et al., 2024)23 proposed to incorporate the Near Infrared (NIR) and RedEdge spectral bands, which are known to provide additional physiological and structural information of the crops. This allows to better separate the crops from weeds, especially in the presence of complex real agricultural scenes. The proposed model performs well on RGB datasets, but generalization can be improved by adding multispectral data. In addition, this work will sweat architectural limitations by adding multi-channel spectral data to improve the reliability and scalability of weed detection through more varied agricultural conditions. The results from the trials show that from an optimization point of view, the test runs supported the conclusion that the Adam methods were better than the RMSprop methods. While the RMSprop methods were more consistent meaning lower the variance, it resulted in the lower average test accuracy and greater average test loss. In compared to the rest of the methods, the Adam methods achieved a best average test loss meaning the most optimal balance between the convergence and the performance. The results of adding label smoothing with Adam showed a small improvement on the test accuracy and increased the generalization. The results of the test showed that the two methods created a statistically insignificant improvement p > 0.05, therefore, were fairly equal. Label smoothing decreased the confidence in the prediction that caused an improvement in the robustness of the worst-performing classes. The proposed model maintains a light architecture and the computational effort does require more than average effort, and therefore, the speed of an inference is goal is less than ~ 18ms, meaning that it is suitable to be deployed in a real time situation. While the model is less than highly optimized ones like the ones based on the MobileNetV3 architecture, the model offers a nice tradeoff with other features like the SE attention with the convolutional feature extraction with the Transformer global context, meaning more than average computational effort. Conclusion The proposed lightweight CNN-SE-Transformer model is an innovative architectural development which combines the strengths of CNNs in localizing weeds from images, and the long-range dependencies of Transformers with the additional ability to capture contextual features through self-excitation (SE) mechanisms. A major advantage of this architecture is the ability to perform at high levels of accuracy while remaining computationally efficient enough for use in real-world agricultural settings. The experimental evaluation conducted to evaluate the effectiveness of the proposed lightweight CNN-SE-Transformer model was conducted on two different datasets which represented two differing levels of complexity. These results demonstrated that the lightweight CNN-SE-Transformer architecture was capable of producing accurate classification results regardless of the level of complexity inherent in the dataset used. The proposed architecture produced very good classification results across all datasets tested; specifically, the highest level of accuracy achieved by the proposed model across all datasets tested was 93% for Dataset 1, and 78% for Dataset 2. An ablation study was performed to determine how much each of the components contributed to overall performance. Specifically, the impact of residual connections, SE blocks, and Transformer modules were determined to have a significant positive effect on improving performance. In addition, the proposed model also found that employing Adam with label smoothing enhanced generalizability and stability; especially when dealing with large disparities in data distributions (i.e., class imbalance). In addition to achieving high levels of accuracy, the proposed architecture has a number of other advantages. One of these advantages is a low parameter count, and another advantage is fast inference times. Both are critical factors for many types of real-time agricultural applications such as precision farming and fully automated weed detection systems. Finally, the authors employed Grad-CAM to produce visualizations of their predictions. Visualizations provide valuable insights into why a particular prediction was made. Future work can expand by exploring more advanced methods to balance the classes such as generating synthetic training samples using Generative Adversarial Networks (GANs), or Diffusion Models. Another example includes developing more efficient versions of Transformers or developing alternative attention mechanisms to reduce computational requirements while enhancing performance. In addition, deploying the model in real-world field conditions on edge devices will help demonstrate its feasibility. More broadly, additional modalities including spectral (hyperspectral), temporal, or combinations thereof can potentially be integrated into the model to enhance classification accuracy in increasingly complex agricultural environments. Finally, adapting the model to detect and/or segment objects will allow for even more comprehensive solutions to support weed management. Acknowledgements The authors gratefully acknowledge the use of the publicly available datasets that supported this research and Vellore Institute of Technology for institutional support. Author contributions P.G. conceived the study, designed the methodology, conducted the experiments, analysed the data, and wrote the original draft. V.T. supervised the research, provided critical review and editing, and validated the results. Both authors reviewed and approved the final manuscript. Funding Open access funding provided by Vellore Institute of Technology. Data availability The datasets supporting the findings of this study are publicly available from the Mendeley Data repository, at https://data.mendeley.com/datasets/4gkcyxjyss/1. The MH-Weed16 dataset is utilized from https://data.mendeley.com/datasets/d3n3mgjjbv/1. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. References Associated Data Data Availability Statement The datasets supporting the findings of this study are publicly available from the Mendeley Data repository, at https://data.mendeley.com/datasets/4gkcyxjyss/1. The MH-Weed16 dataset is utilized from https://data.mendeley.com/datasets/d3n3mgjjbv/1.