Segregation and Context Aggregation Network for Real-time Cloud Segmentation
Yijie Li, Hewei Wang, Jiayi Zhang, Jinjiang You, Jinfeng Xu, Puzhen Wu, Yunzhong Xiao, Soumyabrata Dev · 2025
Read Segregation and Context Aggregation Network for Real-time Cloud Segmentation by Yijie Li, Hewei Wang, Jiayi Zhang, Jinjiang You, Jinfeng Xu, Puzhen Wu, Yunzhong Xiao, Soumyabrata Dev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14178.txt.
Introduction
Yijie Li and co-authors split sky from cloud at the speed of sight in a 2025 paper. Ground shots of the sky feed weather lore and sun craft, but past nets trade truth for speed and stall on small gear.
Their net first drafts a rough map, then splits it in two: a sky face and a cloud face, each weighed on its own terms. The split lets fine rims stay sharp while the whole stays light. The large form, with a few mil parts, matched the best nets with near a third less bulk. The lite form, a sliver of that size, ran past a thou frames a blink on lean gear. A warm-up train trick lifts the score with no big bank of past shots.
Catalog record
| CREATOR | Yijie Li, Hewei Wang, Jiayi Zhang, Jinjiang You, Jinfeng Xu, Puzhen Wu, Yunzhong Xiao, Soumyabrata Dev |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-14178
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Segregation and Context Aggregation Network for Real-time Cloud Segmentation Yijie Li, Hewei Wang, Jiayi Zhang, Jinjiang You, Jinfeng Xu, Puzhen Wu, Yunzhong Xiao, Soumyabrata Dev Abstract Cloud segmentation from intensity images is a pivotal task in atmospheric science and computer vision, aiding weather forecasting and climate analysis. Ground-based sky/cloud segmentation extracts clouds from images for further feature analysis. Existing methods struggle to balance segmentation accuracy and computational efficiency, limiting real-world deployment on edge devices, so we introduce SCANet, a novel lightweight cloud segmentation model featuring Segregation and Context Aggregation Module (SCAM), which refines rough segmentation maps into weighted sky and cloud features processed separately. SCANet achieves state-of-the-art performance while drastically reducing computational complexity. SCANet-large (4.29M) achieves comparable accuracy to state-of-the-art methods with 70.9% fewer parameters. Meanwhile, SCANet-lite (90K) delivers 1390 fps in FP16, surpassing real-time standards. Additionally, we propose an efficient pre-training strategy boosting performance even without ImageNet pre-training. Our code is publicly available at: https://github.com/Att100/SCANet . 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding authors: Soumyabrata Dev ([email protected]) Keywords: cloud segmentation, machine learning, segregation and context aggregation module. 1 Introduction Understanding cloud-sky relationships is crucial for climate modeling, solar energy forecasting, and extreme weather prediction. Advances in computer vision and machine learning have improved meteorology estimation ( Dev et al., 2019c , Jiang et al., 2025 , Tulpan et al., 2017 , Akrami et al., 2022 ) and weather prediction ( Manandhar et al., 2019 , Yu et al., 2024 , Wang et al., 2021b , McNicholl et al., 2021 ) , offering insights into cloud status. While satellites provide valuable cloud data, they are costly and storage-intensive. Ground-based sky/cloud segmentation ( Jain et al., 2021 , Dev et al., 2015 , Dev et al., 2018 , Zhang et al., 2022 ) , supported by datasets like SWIMSEG ( Dev et al., 2016 ) , SWINSEG ( Dev et al., 2017 ) , and SWINySEG ( Dev et al., 2019b ) , offers a cost-effective, high-resolution alternative, enhancing climate applications. Sky/cloud segmentation, a binary semantic task, has evolved with fully convolutional networks (FCN) ( Long et al., 2015 ) . Real-time segmentation strategies include (a) lightweight backbones and decoders, e.g., DeepLab ( Chen et al., 2014 , Chen et al., 2018 ) , and (b) encoder-decoder architectures like ICNet ( Zhao et al., 2018 ) and BiseNet ( Yu et al., 2018 , Yu et al., 2021 ) . However, existing methods either lose information due to attention mechanisms or suffer from slow inference, limiting real-time applications. A detailed discussion of related works, including their limitations, can be found in Appendix A.1 , further motivating the design of SCANet. To address these challenges, we propose SCANet, a lightweight yet effective model integrating MobileNetV2 ( Sandler et al., 2018 ) and EfficientNet-B0 ( Tan and Le, 2019 ) with a Segregation and Context Aggregation module (SCAM). SCANet, following a U-Net ( Ronneberger et al., 2015 ) structure, refines cloud-sky features via SCAM decoders. Unlike prior methods, SCAM enhances feature separation and aggregation, while supervision at the last three stages accelerates convergence. By improving segmentation accuracy and efficiency, SCANet supports climate modeling, renewable energy forecasting, and extreme weather monitoring, demonstrating a direct pathway from machine learning to climate impact. The main contributions of our SCANet are twofold: • SCANet, a lightweight CNN-based model, achieves state-of-the-art performance with 70.68% fewer parameters while exceeding real-time standards, incorporating a new pre-training strategy for sky/cloud segmentation when ImageNet pre-training is unavailable. • A novel SCAM decoder with segregated branches for information processing, enabling precise segmentation while maintaining real-time efficiency. Figure 1: #Params vs. SWINySEG Accuracy. Our proposed SCANet model successfully achieves a balance between the model size and accuracy. SCANet-large can achieve 97.0\% of accuracy in SWINySEG with 4.29 million parameters, while SCANet-lite can achieve 94.4\% of accuracy with only 90k parameters. Figure 2: The overall architecture of SCANet and SCAM. (a) presents the pipeline of our SCANet; (b) details the design of our proposed SCAM module; (c) depicts the decoder structure preceding the SCAM modules. Additionally, The architecture of inverted residual block ( Sandler et al., 2018 ) is demonstrated in Fig. 4 in Appendix A.2 . The up-sample block consists of an inverted residual block paired with a bilinear up-sample layer. 2 SCANet Our SCANet is designed based on the U-Net ( Ronneberger et al., 2015 ) structure. We use different backbone networks to extract high-dimensional features in our experiments. The architecture of SCANet is shown in Fig. 2 (a). We equip our SCANet with MobileNetV2 ( Sandler et al., 2018 ) and EfficientNet-B0 ( Tan and Le, 2019 ) for different settings. We also propose SWINySEG pre-training (SWPT) for SCANet-lite, demonstrated in Appendix A.3 . 2.1 Segregation and Context Aggregation Module (SCAM) The Segregation and Context Aggregation Module (SCAM) is a lightweight decoder for sky/cloud segmentation, as shown in Fig. 2 (b), operating in the 2nd, 3rd, and 4th stages. Given the binary nature of sky/cloud segmentation, SCAM processes these categories separately by first segregating features based on rough segmentation results from the previous stage and then aggregating them. It takes two inputs: a concatenation of the U-Net shortcut and feature maps from the previous layer, c_{i-1} , and the segmentation prediction from the prior stage, s_{i-1} . The main branch is formulated as f_{i}=\mathrm{Cat}(\mathrm{Conv}(c_{i-1}),\mathrm{Conv}(c_{i-1}\times s_{i-1})) , where f_{i} represents foreground (sky) features, and \mathrm{Cat} denotes channel-wise concatenation. The background mask is computed as m_{i}=\mathrm{Sigmoid}(\mathrm{Conv}(c_{i-1}\times(1-s_{i-1}))) . To extract background features, we apply the background mask to the core branch, formulated as b_{i}=f_{i}\times m_{i} , improving background representation. Finally, element-wise addition aggregates f_{i} and b_{i} , producing the output o_{i} as o_{i}=\mathrm{Conv}(\mathrm{UpSample}(\mathrm{InvRes}(b_{i})+\mathrm{InvRes}(f_{i}))) , while the stage prediction is obtained through a convolutional layer followed by a sigmoid activation: s_{i}=\mathrm{Sigmoid}(\mathrm{Conv}({o}_{i})) . 2.2 Loss Functions In our research work, we use binary cross entropy (BCE) and Intersection of Union (IOU) loss as the loss function in the training of SCANet. These loss functions can be defined as follows: \mathcal{L}_{\mathrm{bce}}(p,y)=-\frac{1}{N}*\sum_{j=1}^{N}(y_{j}*\log{p_{j}}+(1-y_{j})*\log{(1-p_{j})}) (1) \mathcal{L}_{\mathrm{iou}}(p,y)=1-\frac{1}{n}\sum_{j=1}^{N}(\frac{y_{j}\times p_{j}}{y_{j}+p_{j}-y_{j}\times p_{j}}) (2) then our total training loss under deep supervision can be formulated as: \mathcal{L}(p,y)=\sum_{i=1}^{4}\alpha_{i}*(\mathcal{L}_{\mathrm{bce}}(p_{i},y_{i})+\mathcal{L}_{\mathrm{iou}}(p_{i},y_{i})) (3) in which \alpha_{i} represents the coefficient of i th SCAM or decoder. Figure 3: Qualitative comparison of SCANet-large with state-of-the-art approaches on day-time (rows 1–2) and night-time (rows 3–4) images from the SWINySEG dataset. 3 Datasets & Configurations 3.1 Dataset We use the Singapore Whole Sky Nychthemeron Image SEGmentation Database (SWINySEG) as our training dataset, which consists of 6078 day-time and 690 night-time cloud images captured in Singapore using a calibrated camera. Following ( Zhang et al., 2022 ) , we split the dataset into training and testing sets with a 9:1 ratio. For evaluation, SCANet is tested on three subsets: day-time images (augmented SWIMSEG), night-time images (augmented SWINSEG), and the full SWINySEG dataset. Notably, SCANet is trained only once on the complete SWINySEG dataset. 3.2 Implementation Details We implement SCANet using PaddlePaddle and conduct training on a single NVIDIA Tesla V100-SXM2 16GB GPU. The model is trained for 100 epochs with a batch size of 16. We use the Adam optimizer with an initial learning rate of 1e-3 , \beta_{1}=0.9 , \beta_{2}=0.999 , and epsilon set to 1e-8 . The learning rate follows an exponential