Zero-Shot Segmentation through Prototype-Guidance for Multi-Label Plant Species Identification

Luciano Araujo Dourado Filho, Almir Moreira da Silva Neto, Rodrigo Pereira David, Rodrigo Tripodi Calumby · 2025

Read Zero-Shot Segmentation through Prototype-Guidance for Multi-Label Plant Species Identification by Luciano Araujo Dourado Filho, Almir Moreira da Silva Neto, Rodrigo Pereira David, Rodrigo Tripodi Calumby (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-19957.txt.

Introduction

Luciano Araujo Dourado Filho and three co-authors entered the PlantCLEF 2025 challenge. In it, one high-resolution photo of a vegetation plot can hold many plant species at once. Their method never trains on such plots head on. It first builds class prototypes. It clusters features from single-species training images to do so. Then it teaches a narrow segmentation transformer to rebuild those prototypes from plot images.

The transformer's patch layer is a frozen DINOv2 model. It was pretrained to classify single species. The attention scores it yields mark where each species likely sits. Those marks guide the final call on the plot. The system placed fifth on the private leaderboard. It sat close behind the winning entry. The code is public.

Catalog record

CREATORLuciano Araujo Dourado Filho, Almir Moreira da Silva Neto, Rodrigo Pereira David, Rodrigo Tripodi Calumby
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-19957

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Zero-Shot Segmentation through Prototype-Guidance for Multi-Label Plant Species Identification Luciano Araujo Dourado Filho, Almir Moreira da Silva Neto, Rodrigo Pereira David, Rodrigo Tripodi Calumby Abstract This paper presents an approach developed to address the PlantClef 2025 challenge, which consists of a fine-grained multi-label species identification, over high-resolution images. Our solution focused on employing class prototypes obtained from the training dataset as a proxy guidance for training a segmentation Vision Transformer (ViT) on the test set images. To obtain these representations, the proposed method extracts features from training dataset images and create clusters, by applying K-Means, with K K equals to the number of classes in the dataset. The segmentation model is a customized narrow ViT, built by replacing the patch embedding layer with a frozen DinoV2, pre-trained on the training dataset for individual species classification. This model is trained to reconstruct the class prototypes of the training dataset from the test dataset images. We then use this model to obtain attention scores that enable to identify and localize areas of interest and consequently guide the classification process. The proposed approach enabled a domain-adaptation from multi-class identification with individual species, into multi-label classification from high-resolution vegetation plots. Our method achieved fifth place in the PlantCLEF 2025 challenge on the private leaderboard, with an F1 score of 0.33331. Besides that, in absolute terms our method scored 0.03 lower than the top-performing submission, suggesting that it may achieved competitive performance in the benchmark task. Our code is available at https://github.com/ADAM-UEFS/PlantCLEF2025 . [orcid=0000-0002-0507-2201, [email protected], url=https://github.com/FalsoMoralista, ] *1 [orcid=0009-0008-5042-5556, [email protected], url=https://github.com/almirneeto99/, ] [orcid=0000-0001-9218-8191, [email protected], url=https://github.com/rpdavid78/, ] [orcid=0000-0001-8515-265X, [email protected], url=https://www.rtcalumby.com.br] keywords vision transformer ,multispecies classification ,plant identification ,fine-grained classification ,prototype learning ,clustering † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: CLEF 2025 Working Notes, 9 – 12 September 2025, Madrid, Spain † † address: ADAM - Advanced Data Analysis and Management, University of Feira de Santana (UEFS), Feira de Santana, Brazil † † address: National Institute of Metrology, Technology and Quality (Inmetro) † † corresp: Corresponding author. 1 Introduction The automated identification of plant species from images of complex vegetation plots presents a significant challenge in ecological research and biodiversity monitoring. Initiatives like the PlantCLEF aim to advance this field by providing benchmark datasets and tasks, fostering innovation in multi-label classification from high-resolution plot imagery 6 ; 8 ; 12 ; 10 . A primary challenge in this task is the notable domain shift between training data, which mostly consist of images of individual plants, and test data, comprising dense, multi-species vegetation plots captured under varied environmental conditions 12 ; 10 ; 4 ; 5 . Therefore, it demands robust methods that can effectively generalize and differentiate target species within cluttered natural scenes 2 . Vision Transformers (ViTs), particularly those pre-trained using self-supervised learning, have demonstrated remarkable effectiveness in generating discriminative visual features for various tasks, including large-scale plant identification 3 ; 9 . In the broad context of image classification, models such as DINOv2 11 has demonstrated to offer powerful backbones for feature extraction. Specifically in The PlantCLEF 2025, organizers supported participants by providing ViT models pre-trained on relevant flora, considering a DINOv2-based architectures 7 . Our proposed method goes beyond the usual direct end-to-end classification of the entire plot images (which can be confounded by background noise and species overlap) by focusing on a preliminary segmentation-driven filtering step. We propose a “decrease-and-conquer” strategy which relies in a dedicated ViT that first identifies plant-relevant regions, thereby simplifying the subsequent classification task for specialized heuristics. This ViT leverages its attention mechanisms as a proxy for spatial segmentation, aiming to isolate pertinent areas from irrelevant background elements. A crucial aspect of our methodology is the training of this segmentation ViT without direct pixel-level supervision. To overcome the absence of segmentation labels, we introduce a novel proxy task as our core contribution. We trained a narrow, customized ViT to reconstruct a predetermined representation of the training dataset —specifically, class prototypes derived from k-Means clustering on DINOv2 embeddings of the original training set images. Uniquely, this ViT learns this reconstruction objective using, as its input, feature embeddings extracted by a separate, frozen DINOv2 model from test images . Our hypothesis is that for the ViT to successfully map features from unseen test plots back to these known training set prototypes, it must learn to generate attention scores that highlight plant-relevant patches while attenuating signals from the background (as illustrated in Figure 3 ). This paper provides a detailed exposition of this proxy task-driven ViT technique. The generation of the target class prototypes, the specific architecture of the custom ViT, and the operational steps of its unique training regimen as detailed in Section 3 . Furthermore, we describe how the learned attention maps are utilized within our inference pipeline to select regions of interest prior to applying classification heuristics for final species identification. The experimental settings, key hyperparameters, and the results achieved in the PlantCLEF 2025 challenge are presented and discussed, offering insights into the effectiveness of this approach for multi-label classification in complex ecological imagery. 2 Task Description The PlantCLEF 2025 challenge addresses the intricate problem of multi-label species identification within high-resolution images of vegetation plots. The core objective is to predict all plant species present in a given plot image, a fundamental task to ecological research, biodiversity assessment, and long-term environmental monitoring. The primary difficulty of the challenge stems from a significant domain shift between the provided datasets. A comprehensive training dataset comprising roughly 1.4 million images representing 7,806 plant species. These training images typically feature single plant species from various perspectives. In contrast, the test dataset consists of 2,105 high-resolution plot images from Mediterranean and Pyrenean regions. These test images depict complex, multi-species scenes, often captured from a top-down perspective within delimited sampling areas (quadrats), and under varying environmental conditions that can introduce shadows or blurry areas. Two pre-trained ViT models based on the DINOv2 architecture were provided along with the dataset. These models were not only pre-trained on a massive dataset of images but were also specifically fine-tuned on the PlantCLEF 2024 training data which includes isolated plant samples, considering multiple organs and perspectives. The fine-tuned models available were: a) one where only the final classification layer was trained; b) another where all model layers were unfrozen and fine-tuned for the task. These models offer a powerful, ready-to-use foundation resources avoiding new investigations to demand the extensive computational resources needed to train such large-scale models from scratch. The classification effectiveness was evaluated using the macro F1 score averaged per plot, a metric designed to balance precision and recall for each individual plot images, with results tracked on a public leaderboard. The final score is computed as described in Equation 1 , where N is the number of transects (plots) , T^{i} is the number of quadrats in transect i , \frac{1}{T_{i}}\sum_{j=1}^{T_{i}}F1^{j} is the macro-averaged F1-score per sample of transect i and F1^{j} is the F1 score for test image j . Score=\frac{1}{N}\sum_{i=1}^{N}(\frac{1}{T_{i}}\sum_{j=1}^{T_{i}}F_{1}^{j}) (1) For each test image j the F1 score is described as follows in Equation 2 . Macro\ F_{1}^{j}=\frac{2\times Precision_{j}\times Recall_{j}}{Precision_{j}+Recall_{j}} (2) The

Read the complete text