Domain-Specific Self-Supervised Pre-training for Agricultural Disease Classification: A Hierarchical Vision Transformer Study
Arnav Sonavane · 2026
Read Domain-Specific Self-Supervised Pre-training for Agricultural Disease Classification: A Hierarchical Vision Transformer Study by Arnav Sonavane (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11612.txt.
Introduction
Arnav Sonavane pre-trains vision models on farm images for disease work. The 2026 preprint uses hierarchical vision transformers for the task. Agricultural disease classification guides treatment and saves crops. Labels for sick plants are costly to gather by hand.
Self-supervised pre-training learns from unlabeled images alone. Sonavane tests domain-specific pre-training with the SimCLR method. Just 3,000 unlabeled farm images drive the pre-training. The gain in accuracy is 4.57 percent in the study. That beats the 3.70 percent from architecture design. The benefit holds across architectures, not just one model. Swin-Base gains over 4 percent from the same step. Domain data, it seems, beats generic scale here.
Catalog record
| CREATOR | Arnav Sonavane |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11612
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Domain-Specific Self-Supervised Pre-training for Agricultural Disease Classification: A Hierarchical Vision Transformer Study Arnav Sonavane Abstract We investigate the impact of domain-specific self-supervised pre-training on agricultural disease classification using hierarchical vision transformers. Our key finding is that SimCLR pre-training on just 3,000 unlabeled agricultural images provides a +4.57% accuracy improvement—exceeding the +3.70% gain from hierarchical architecture design. Critically, we show this SSL benefit is architecture-agnostic : applying the same pre-training to Swin-Base yields +4.08%, to ViT-Base +4.20%, confirming practitioners should prioritize domain data collection over architectural choices. Using HierarchicalViT (HVT), a Swin-style hierarchical transformer, we evaluate on three datasets: Cotton Leaf Disease (7 classes, 90.24%), PlantVillage (38 classes, 96.3%), and PlantDoc (27 classes, 87.1%). At matched parameter counts, HVT-Base (78M) achieves 88.91% vs. Swin-Base (88M) at 87.23%, a +1.68% improvement. For deployment reliability, we report calibration analysis showing HVT achieves 3.56% ECE (1.52% after temperature scaling). Code: https://github.com/w2sg-arnav/HierarchicalViT 1 Introduction Agricultural crop diseases pose a significant threat to global food security, causing substantial economic losses and threatening crop yields worldwide. Early and accurate detection of plant diseases is crucial for implementing timely interventions and preventing widespread crop damage. Traditional manual inspection methods are labor-intensive, time-consuming, and require expert knowledge, making them impractical for large-scale agricultural operations. Recent advances in deep learning, particularly in computer vision, have shown promising results in automating plant disease detection 14 ; 5 . Vision Transformers (ViTs) 4 have emerged as powerful alternatives to convolutional neural networks (CNNs) for image classification tasks, demonstrating superior performance on various benchmarks. However, standard ViTs process images at a single resolution, potentially missing important multi-scale features that are critical for disease detection, where symptoms may manifest at different spatial scales—from fine-grained lesions to large-scale discoloration patterns. In this work, we investigate a practical research question: How much does domain-specific self-supervised pre-training help agricultural disease classification, and how does it compare to architectural improvements? Using HierarchicalViT (HVT), a Swin-style hierarchical transformer, we find that: • SSL pre-training is the dominant factor: SimCLR pre-training on 3,000 unlabeled agricultural images provides +4.57% accuracy gain, exceeding the +3.70% from hierarchical architecture (Table 4 ). • Parameter-matched comparison: HVT-Base (78M params) achieves 88.91% vs. Swin-Base (88M) at 87.23%—a +1.68% improvement at comparable scale (Table 9 ). • SSL corpus size ablation: We find diminishing returns beyond 2,000 unlabeled images for our dataset scale, providing practical guidance for data collection (Table 5 ). • Cross-dataset evaluation: Results on PlantVillage (96.3%) and PlantDoc (87.1%) demonstrate generalization, though we note PlantVillage is a saturated benchmark. Our contribution is primarily empirical: demonstrating the value of domain-specific SSL for agricultural applications, rather than proposing novel architectural components. The hierarchical architecture (adapted from Swin Transformer 13 ) serves as a vehicle for this study. 2 Related Work Plant Disease Detection. Deep learning has revolutionized automated plant disease detection 14 ; 5 ; 12 . Traditional approaches use CNNs (ResNet 9 , DenseNet 10 , EfficientNet 16 ), which may have limitations in capturing long-range dependencies crucial for distinguishing subtle disease symptoms. Vision Transformers and Hierarchical Architectures. Vision Transformers 4 leverage self-attention 17 to model global context effectively. Hierarchical variants like Swin Transformer 13 and PVT 18 combine local and global processing through progressive spatial reduction and shifted window attention. Our HVT architecture closely follows the Swin Transformer design ; our primary novelty lies in combining this established architecture with domain-specific SimCLR pre-training for agricultural applications. We include an optional cross-attention module for potential multi-modal extensions, though it provides only marginal improvement (+0.57%) in current RGB-only experiments. Self-Supervised Learning and Multi-Modal Fusion. Self-supervised learning (SimCLR 3 , MAE 8 ) leverages unlabeled data for representation learning, particularly valuable in agriculture 2 where unlabeled data is abundant. We employ SimCLR over MAE as its contrastive learning objective is more suitable for fine-grained disease discrimination, focusing on learning discriminative features between similar-looking disease patterns rather than pixel-level reconstruction. Multi-modal fusion 21 ; 7 combines RGB, multispectral, and hyperspectral data; we include hooks for future multi-modal extension but focus on RGB-only evaluation in this work. 3 Method 3.1 Overall Architecture Input RGB Image 448\times 448 Patch Embedding P=14 , 32^{2} tokens Stage 1 Depth: 3 Dim: 192 Res: 32\times 32 Stage 2 Depth: 6 Dim: 384 Res: 16\times 16 Stage 3 Depth: 24 Dim: 768 Res: 8\times 8 Stage 4 Depth: 3 Dim: 1536 Res: 4\times 4 Classification Head GAP + Linear Predictions Softmax 7 classes \downarrow Merge 2×2 \downarrow Merge 2×2 \downarrow Merge 2×2 Hierarchical Multi-Scale Processing Transformer Block Components \bullet Multi-Head Self-Attention \bullet Feed-Forward Network (4× expansion) \bullet DropPath + Residual Connections \bullet Layer Normalization 36 blocks total across 4 stages Training Strategy \star SSL Pre-training: SimCLR (80 epochs) \star Fine-tuning: Focal Loss (100 epochs) \star Augmentation: MixUp + CutMix \star Inference: EMA + TTA ensemble Total parameters: 158M Figure 1 : HierarchicalViT-XL architecture overview. The model processes 448\times 448 RGB images through patch embedding and four hierarchical transformer stages with progressive spatial downsampling ( 32^{2}\to 16^{2}\to 8^{2}\to 4^{2} tokens) and channel expansion ( 192\to 384\to 768\to 1536 dimensions). Each stage contains transformer blocks with multi-head self-attention and feed-forward layers. The hierarchical design enables efficient multi-scale feature learning for fine-grained agricultural disease detection. Table 1 : Notation Summary Symbol Description \mathcal{D}(\cdot) DropPath (stochastic depth) operation \mathcal{C}(\cdot) Cross-attention module (optional) \mathcal{M}(\cdot) Patch merging (spatial downsampling) \mathcal{A}(\cdot) Multi-head self-attention \mathcal{F}(\cdot) Feed-forward network (MLP) S Number of hierarchical stages (=4) D_{s} Channel dimension at stage s L_{s} Number of transformer blocks at stage s H_{s}\times W_{s} Spatial resolution at stage s Our HVT backbone (Figure 1 ) processes input images \mathbf{I}\in\mathbb{R}^{H\times W\times 3} through: (1) patch embedding, (2) four hierarchical transformer stages, (3) optional cross-attention refinement, and (4) classification head. Given input resolution H\times W=448\times 448 and patch size P=14 , we divide the image into non-overlapping patches, resulting in N=\frac{HW}{P^{2}}=1024 tokens. Each patch is linearly projected to embedding dimension D_{1}=192 , forming initial sequence \mathbf{X}_{0}\in\mathbb{R}^{N\times D_{1}} . 3.2 Hierarchical Transformer Stages Unlike flat ViTs, our architecture employs S=4 hierarchical stages with progressive downsampling (Table 1 ). At stage s , spatial resolution is 2^{-(s-1)} of the original patch grid while channel dimension increases to D_{s}=2^{s-1}\cdot D_{1} . Each stage has L_{s} transformer blocks with stochastic depth: \displaystyle\mathbf{Z}^{\prime}_{l} \displaystyle=\mathbf{Z}_{l-1}+\mathcal{D}(\mathcal{A}(\text{LN}(\mathbf{Z}_{l-1}))), (1) \displaystyle\mathbf{Z}_{l} \displaystyle=\mathbf{Z}^{\prime}_{l}+\mathcal{D}(\mathcal{F}(\text{LN}(\mathbf{Z}^{\prime}_{l}))), (2) where \mathcal{D}(\cdot) applies stochastic depth for regularization, \mathcal{A}(\cdot) computes multi-head self-attention, and \mathcal{F}(\cdot) applies the feed-forward network. See Section B for detailed formulations. Multi-Head Self-Attention. For input \mathbf{X}\in\mathbb{R}^{N\times D} with N tokens and dimension D , we compute queries \mathbf{Q} , keys \mathbf{K} , and values \mathbf{V} : \mathbf{Q}=\mathbf{X}\mathbf{W}_{Q},\quad\mathbf{K}=\mathbf{X}\mathbf{W}_{K},\quad\mathbf{V}=\mathbf{X}\mathbf{W}_{V}, (3) where \mathbf{W}_{Q,K,V}\in\mathbb{R}^{D\times D} are learned projection matrices. For h attention heads, we reshape to \mathbb{R}^{N\times h\times d_{h}} with per-head dimension d_{h}=D/h , then compute: \mathcal{A}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_{h}}}\right)\mathbf{V}. (4) Patch Merging. Between stages, patch merging \mathcal{M}(\cdot) reduces spatial resolution by concatenating 2\times 2 neighborhoods and projecting to 2D_{s} channels. For HVT-XL ( P=14 , input 448\times 448 ): depths [3,6,24,3] , heads [6,12,24,48] , dimensions [192,384,768,1536] , spatial resolutions [32\times 32,16\times 16,8\times 8,4\times 4] tokens. We include an optional cross-attention module for future multi-modal extensions (e.g., RGB + spectral); in this work we focus on RGB-only evaluation where it contributes only +0.57%. 3.3 Self-Supervised Pre-training We employ SimCLR 3 for self-supervised pre-training on unlabeled agricultural data, as