Efficient Curation of Invertebrate Image Datasets Using Feature Embeddings and Automatic Size Comparison
Mikko Impiö, Philipp M. Rehsen, Jenni Raitoharju · 2024
Read Efficient Curation of Invertebrate Image Datasets Using Feature Embeddings and Automatic Size Comparison by Mikko Impiö, Philipp M. Rehsen, Jenni Raitoharju (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-15844.txt.
Introduction
Mikko Impiö, Philipp M Rehsen, and Jenni Raitoharju speed up the cleaning of insect photo collections used to train vision systems. Environmental monitoring now gathers huge image sets, but models are only as good as the data's labels, and cleaning is usually ad hoc and unpublished. Their 2024 method targets collections with many shots of the same creatures on plain backgrounds. A pretrained network turns each photo into a feature vector; near-duplicate vectors flag repeats of the same specimen. Automatic size comparison adds a second check.
Curators can then drop duplicates and errors in bulk instead of by eye. Published curation methods let other teams judge and reuse the hygiene steps. The authors treat data cleaning as a research product, not a chore. Their approach suits any archive of look-alike specimens. Cleaner sets mean honest accuracy numbers downstream.
Catalog record
| CREATOR | Mikko Impiö, Philipp M. Rehsen, Jenni Raitoharju |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-15844
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Efficient Curation of Invertebrate Image Datasets Using Feature Embeddings and Automatic Size Comparison Thanks: This work was funded by Research Council of Finland project 333497. Mikko Impiö, Philipp M. Rehsen, Jenni Raitoharju Abstract The amount of image datasets collected for environmental monitoring purposes has increased in the past years as computer vision assisted methods have gained interest. Computer vision applications rely on high-quality datasets, making data curation important. However, data curation is often done ad-hoc and the methods used are rarely published. We present a method for curating large-scale image datasets of invertebrates that contain multiple images of the same taxa and/or specimens and have relatively uniform background in the images. Our approach is based on extracting feature embeddings with pretrained deep neural networks, and using these embeddings to find visually most distinct images by comparing their embeddings to the group prototype embedding. Also, we show that a simple area-based size comparison approach is able to find a lot of common erroneous images, such as images containing detached body parts and misclassified samples. In addition to the method, we propose novel metrics for evaluating human-in-the-loop outlier detection methods. The implementations of the proposed curation methods, as well as a benchmark dataset containing annotated erroneous images, are publicly available in https://github.com/mikkoim/taxonomist-studio . I Introduction The need for global biodiversity monitoring efforts is widely recognized, as the state of natural environments continues deteriorating worldwide and major fractions of biodiversity remain unknown [ 1 , 2 , 3 ] . Insects and other arthropods are groups of special interest in these monitoring efforts. Insects are the most diverse animal group and of great importance in food webs and as providers of ecosystem services [ 4 ] . Monitoring these groups with traditional methods is laborious, and approaches using new technologies like computer vision have been proposed in the recent years [ 5 ] . Computer vision can address challenges such as species classification, enumeration, and biomass estimation. Computer vision methods usually rely on deep learning, which in turn needs a lot of training data. Several approaches for digitizing arthropod samples for computer vision purposes have emerged in the past few years, making collection of larger datasets easier [ 6 , 7 , 8 , 9 , 10 ] . The amount of image data is thus growing, but often raw datasets collected with imaging devices contain erroneous images. These images might be debris, air bubbles, or anything else that does not match the taxonomic class label given to the image. Removing these images through curation is important in order to produce high-quality training datasets, which usually lead to better classification performance [ 11 , 12 ] . Dataset curation remains a laborious process. In practice, most curation solutions are developed as custom one-off approaches for specific datasets and problems. Often curation work is also done manually, which can be very time-consuming for large datasets. Some curation studies exist in the medical imaging domain [ 12 ] and for content-based curation of datasets [ 13 ] , but most curation solutions are not published. (a) Ilybius (b) Ischnura elegans (c) Phryganea Fig. 1 : Examples of successful BIODISCOVER images In this paper, we propose methodology for curating large-scale arthropod datasets imaged with a device, such as BIODISCOVER [ 8 ] , that (semi-)automatically takes multiple images of presorted samples. A couple of examples of successfully imaged arthropods using the BIODISCOVER imaging device are shown in Fig. 1 . The curation needs may arise from, e.g., debris or misclassified samples among the samples taxonomically presorted by a human expert, detached body parts, or air bubbles in the ethanol used for imaging the wet samples. Examples of different types of erroneous content are shown in Fig. 2 . Fig. 2 : Examples of erroneous content in images captured by a BIODISCOVER imaging device: A: Bubbles, B: Detached body parts, C: Forceps, D: Misclassifications Our proposed curation approach is especially useful for initial detection of erroneous content, where a human expert makes the final decision on the removal of the detected candidates. We show that good results can be achieved using deep networks pretrained on unrelated generic datasets and, thus, the approach does not require any training. The approach is suitable for any dataset that contains multiple images of the same taxon and/or specimens and have relatively uniform background in the images. In addition to the proposed dataset curation approach, our contributions include publishing a novel real-life dataset suitable for evaluating data curation approaches and proposing three novel metrics suitable for the evaluation task. The rest of the paper is organized as follows. In Section II , we present our approach feature embedding and size-based dataset curation. In Section III , we introduce the novel benchmark dataset. The proposed novel metrics are introduced in Section IV along with overall experimental setup and experimental results that show the effectiveness of the proposed dataset curation approach via experiments on two real-life arthropod datasets. Finally, in Section V , we conclude the paper. II Proposed Dataset Curation Approach We use two different methods for dataset curation: the first is based on image content comparison using feature embeddings and the second is based on size comparison. Both of the methods are designed for datasets that contain multiple images representing a single individual and/or taxon. II-A Dataset Curation via Feature Embeddings Image content can be compared by calculating feature embeddings for the images in the dataset. The embeddings are dense, real valued vectors representing the original images and their contents. We propose extracting the feature embeddings from a pretrained deep neural network that consists of a feature encoder f and a classification head g , so that their composition maps image inputs \mathbf{x}_{i} from a dataset \mathcal{X}=\{\mathbf{x}_{i}\}_{i=1}^{N} to a target class y_{i}\in\mathcal{Y} , i.e., g\circ f:\mathcal{X}\rightarrow\mathcal{Y} . Both f and g can be deep neural networks, however g usually consists of a single fully-connected feed-forward layer. The intermediate output \mathbf{z}_{i}=f(\mathbf{x}_{i}),\mathbf{z}_{i}\in\mathbb{R}^{M} is an M -dimensional vector usually called a feature vector , or an embedding . The similarity of two feature embeddings \mathbf{z}_{i} and \mathbf{z}_{j} can be compared by calculating their cosine distance from each other. Cosine distance for vectors \mathbf{z}_{i} and \mathbf{z}_{j} is d_{\cos}(\mathbf{z}_{i},\mathbf{z}_{j})=1-\frac{\mathbf{z}_{i}\cdot\mathbf{z}_{j}}{||\mathbf{z}_{i}||_{2}||\mathbf{z}_{j}||_{2}}, (1) meaning that orthogonal vectors have a distance of 1. In our experiments, we use feature embeddings extracted from a simple MobileNet-v3 [ 14 ] model pretrained on ImageNet-1k dataset [ 15 ] . However, the data curation approach can be used with any deep neural network model pretrained on any dataset. Our experimental results demonstrate that a network pretrained using a generic unrelated dataset can produce meaningful feature embeddings for our dataset curation approach, i.e., no training on the dataset to be curated is required. After extracting the feature embeddings for each dataset image, we choose a grouping (for example based on a taxon or a specimen) for which a mean feature embedding is calculated, and each image (feature embedding) in the same group is compared to. This is similar to calculating a ”prototype” vector for a group, as formulated in [ 16 ] . We assume that this mean feature embedding encodes some properties of the group and meaningful distances to it can be calculated, although