Crowdsourcing of Real-world Image Annotation via Visual Properties

Xiaolei Diao, Fausto Giunchiglia · 2026

Read Crowdsourcing of Real-world Image Annotation via Visual Properties by Xiaolei Diao, Fausto Giunchiglia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14449.txt.

Introduction

Diao and Giunchiglia tackle the semantic gap in image annotation, where visual data and words connect in tangled many-to-many ways that bias object recognition. Their method joins knowledge representation, language processing, and computer vision, constraining annotators with visual properties to curb subjectivity. An interactive crowdsourcing setup asks questions on the fly from a fixed object-category hierarchy plus annotator feedback, steering labeling by visual traits. Tests show the method works, and annotator feedback points to ways of tuning the setup.

Catalog record

CREATORXiaolei Diao, Fausto Giunchiglia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41208
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14449

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Crowdsourcing of Real-world Image Annotation via Visual Properties Xiaolei Diao, Fausto Giunchiglia Abstract Recent advances in data-centric artificial intelligence highlight inherent limitations in object recognition datasets. One of the primary issues stems from the semantic gap problem, which results in complex many-to-many mappings between visual data and linguistic descriptions. This bias adversely affects performance in computer vision tasks. This paper proposes an image annotation methodology that integrates knowledge representation, natural language processing, and computer vision techniques, aiming to reduce annotator subjectivity by applying visual property constraints. We introduce an interactive crowdsourcing framework that dynamically asks questions based on a predefined object category hierarchy and annotator feedback, guiding image annotation by visual properties. Experiments demonstrate the effectiveness of this methodology, and annotator feedback is discussed to optimize the crowdsourcing setup. 1 Introduction In recent years, the computer vision area has experienced significant advancements, which are utilized in numerous practical application scenarios and bring convenience to daily life. As a data-driven science, machine learning models rely on high-quality datasets for training and evaluation. Driven by data-centric artificial intelligence 16 ; 37 , the importance of data quality in improving the performance of object recognition models 31 has rekindled the interests of researchers. A critical analysis 41 of the construction process of benchmark image datasets, exemplified by ImageNet 3 , reveals systemic flaws that could be a pivotal challenge in impeding the continued progress of object recognition. The construction of non-generative image datasets in the field of computer vision typically involves crowdsourcing techniques 22 . During the crowdsourcing annotation process, annotators are presented with images and asked to select the most appropriate label from a set of words that are predefined categories, as introduced in some datasets, e.g., ImageNet 3 and CIFAR 20 . The labelling of images in such datasets is obtained by requiring annotators to verify if the image matches a predefined category or synonym set. For instance, if the synonym set is “bear,” all images deemed by annotators to contain ”bear” are categorized under this label. A widely discussed issue in this construction methodology is the many-to-many matching problem between categories and images 41 ; 9 . This issue, where an image containing multiple objects is described by a single label, introduces confusing erroneous information into object recognition models, negatively impacting recognition accuracy. The advent of datasets designed for object detection tasks, such as COCO 24 and PASCAL VOC 7 , has somewhat alleviated the problem of information clutter caused by multiple objects. The method of marking specific objects with polygonal boxes allows images to be labelled one-to-one. However, the approach to single-object category labelling remains unchanged, with the annotation process still matching images to predefined words. Figure 1 : Images and their labels in existing image datasets. (a) This image appears in three categories of ImageNet, i.e., “Musical Instrument”, “Guitar”, and “Acoustic Guitar”. (b)-(e) These four images are all labelled as the “Brown Bear” category in the Open Image Dataset. We observe that the process of annotating these image datasets is somewhat subjective, relying on the annotator’s interpretation to label the visual content. This subjectivity introduces a series of inconsistencies, as different annotators may perceive and label the same visual content with different criteria and in various ways. We give some examples in Figure 1 . We found that Figure 1 (a) in ImageNet falls under three categories, being labelled as “Acoustic Guitar”, “Guitar”, and “Musical Instrument”. In our opinion, this situation occurs due to the level of categorization granularity provided subjectively by the annotators. While we cannot declare that these labels are incorrect, it is undeniable that the fine-grained inconsistency in image categories may negatively impact object recognition results. In addition, in the Open Images Dataset 18 , we discovered that Figure 1 (b), (c), (d) and (e) are all labelled under the category “Brown Bear”, despite significant differences: Figure 1 (b) is an animal , Figure 1 (c) is a toy and also labelled as “Teddy Bear”, Figure 1 (d) is a stick figure of a cartoon bear, and Figure 1 (e) looks like a person playing the role of a brown bear doll. The reason these four images are categorized under the same label is due to the lack of constraints in upper-layer domain categories 10 , allowing annotators to subjectively classify at their discretion. Such subjective annotation introduces significant challenges in training machine learning models, which require precise and consistent data for optimal performance. The complexity and polysemy of natural language indeed introduce ambiguity when describing the visual information in images and their semantic interpretation, leading to the well-known issue of the semantic gap problem (SGP) 1 1 1 The SGP is the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation. 38 . This gap exacerbates the challenge of annotation inconsistency in image datasets. A fundamental issue with existing annotation methodologies is the lack of a clearly defined process for categorizing and standardizing image annotations. As a result, there is a pressing need for an image annotation methodology that minimizes the impact of subjective interpretations by annotators, aims to ensure consistency across annotators with diverse backgrounds, and maintains high quality in image datasets. To address these issues, we propose a new methodology for dataset construction that combines the strengths of knowledge representation, natural language processing, and computer vision technologies within a crowdsourcing framework, aiming to significantly improve the quality of image datasets. The integration of multiple technologies is intended to create a more systematic and objective method of image annotation. Specifically, we utilize knowledge representation technology 33 to construct a hierarchy of object categories 6 , which helps establish and define the relationships between different visual elements and their semantic labels. For each category within the hierarchy, we define visual features that can determine its parent categories and distinguish it from sibling categories, presented through structured natural language. This will assist annotators in better understanding and standardizing the annotation process. During the construction of the dataset, we rely on crowdsourcing technology, a powerful tool for collecting large-scale data, to collect labels for the image dataset. To fully leverage its potential, guiding the crowdsourcing process through explicit protocols and quality control measures is vital. In this paper, we explore how to effectively implement a defined integrated annotation method in a crowdsourcing environment, thereby providing more reliable and consistent datasets for machine learning applications in object recognition. Additionally, our dataset will eventually acquire multi-level labels, including fine-grained category labels at different levels, visual property labels, and natural language descriptions of visual features. This will make the dataset not only suitable for object recognition but also applicable to multiple tasks in the vision and language fields, including fine-grained image recognition, zero-shot image recognition, image captioning, image generation, etc. The remainder of the paper is organized as follows. Section 2 introduces the overall methodology we proposed for image dataset construction. We illustrated the detailed labeling strategy based on crowdsourcing in Section 3. Detailed experiments and analysis of the dataset quality evaluation are given in Section 4. Section 5 gives the related work, and Section 6 concludes the paper.

Read the complete text