Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation

Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait, Ansar Khangeldin · 2026

Read Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation by Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait, Ansar Khangeldin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13223.txt.

Introduction

Eldesokey and colleagues argue that judging text to image models needs annotation matched to each skill. Uniform rating scales ignore how skills differ, so scores can mislead even as models converge. Skill aligned annotation gives steadier signals, higher agreement between raters, and more stable rankings across models. They also build an automated pipeline that applies the protocol at scale with spatially grounded feedback.

Catalog record

CREATORAbdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait, Ansar Khangeldin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49673
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.13223

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait, Ansar Khangeldin Abstract Text-to-image (T2I) generation has advanced rapidly, making reliable evaluation critical as performance differences between models narrow. Existing evaluation practices typically apply uniform annotation mechanisms, such as Likert-scale or binary question answering (BQA), across heterogeneous evaluation skills, despite fundamental differences in their nature. In this work, we revisit T2I evaluation through the lens of skill-aligned annotation, where annotation strategies reflect the underlying characteristics of each evaluation skill. We systematically compare skill-aligned annotation against uniform baselines and show that it produces more consistent evaluation signals, with higher inter-annotator agreement and improved stability across models. Finally, we present an automated pipeline that instantiates the proposed evaluation protocol, enabling scalable and fine-grained evaluation with spatially grounded feedback. Our work highlights that improving the foundations of image evaluation can increase reliability and efficiency without simply scaling annotation effort. We hope this motivates further research on refining evaluation protocols as a central component of reliable model assessment. 1 Introduction Text-to-Image (T2I) generation has recently undergone rapid progress with the emergence of diffusion models [ 11 , 30 , 28 , 27 , 2 , 7 ] . As performance differences between models continue to narrow, reliable evaluation becomes increasingly critical for guiding meaningful progress and identifying failure modes. A large body of work has therefore focused on evaluating T2I systems across multiple axes, including semantic fidelity, compositional correctness, and visual quality [ 16 , 39 , 37 , 20 , 22 , 12 , 3 , 9 , 35 , 10 , 29 ] . In practice, the gold standard for T2I evaluation relies on human feedback, where human annotators assess different evaluation skills ( e.g. spatial arrangement, text rendering, visual quality) using uniform annotation mechanisms , such as Likert-scale ratings, ranking, or binary question answering (BQA). However, these mechanisms are typically applied uniformly across heterogeneous skills, despite fundamental differences in their structure. The recent work Gecko [ 35 ] ) showed that annotation design can significantly affect both inter-annotator agreement and evaluation outcomes, raising a central question: “ Are standard annotation mechanisms uniformly effective in capturing heterogeneous skills in T2I generation? ” Figure 1 : Overview of the proposed Text-to-Image evaluation protocol using skill-aligned annotation, yielding more consistent and stable evaluation signals than uniform baselines. Current evaluation pipelines implicitly assume that a single annotation strategy can capture diverse evaluation skills. This assumption is often invalid in practice. For example, assessing landmark or style recognition via BQA presumes annotators possess sufficient prior knowledge, which is rarely verified. Conversely, compressing spatially localized artifacts into a single Likert score discards fine-grained evidence and can lead to disproportionate penalties for minor defects. These mismatches suggest that annotation design is a critical but underexplored component of evaluation methodology . In this work, we revisit T2I evaluation from the perspective of skill-aligned annotations, where annotation mechanisms are chosen to match to the nature of each evaluation skill. To achieve this, we first define a taxonomy of evaluation skills spanning semantic and aesthetic dimensions, and use it to systematically tag image generation text prompts with different skills. Building on this taxonomy, we design annotation strategies tailored to different skill types. Crucially, we treat annotation design as a methodological variable and conduct controlled comparisons between skill-aligned and uniform annotation protocols. We show that skill-aligned annotation produces more consistent stable outcomes that converge with a few number of annotators, while achieving higher inter-annotator agreement. Finally, to support large-scale evaluation, we instantiate the proposed protocol in an automated pipeline that enables scalable, fine-grained, and spatially grounded assessment, achieving strong alignment with human judgments across most skills. We hope this work encourages a shift toward more principled evaluation design in T2I generation, emphasizing the role of annotation protocols in determining evaluation reliability 1 1 1 The source code is available at https://github.com/abdo-eldesokey/skill-aligned-eval . 2 Related Work 2.1 Evaluation Axes of T2I Generation Evaluating T2I models requires defining axes that capture distinct aspects of image quality. Early and widely adopted approaches [ 16 , 37 , 36 , 25 ] formulate evaluation as a global human preference task, where annotators rank images according to overall appeal. While such signals capture aggregate user preference, they do not disentangle the underlying factors that contribute to image quality, limiting their usefulness for diagnosing model behavior or enforcing task-specific requirements. To address this limitation, a large body of work focuses on more targeted evaluation axes, most notably semantic fidelity to the input prompt. These approaches assess whether generated images satisfy explicit textual constraints, such as object presence, counts, attributes, and spatial relationships [ 3 , 22 , 12 , 9 , 35 , 21 , 14 , 24 , 17 , 42 ] . Such formulations improve diagnostic granularity but often abstract away perceptual qualities of the image. Complementary work therefore considers perceptual axes, including realism, aesthetics, and visual artifacts, either independently or jointly with semantic fidelity [ 41 , 26 , 13 , 39 , 38 , 20 ] . As T2I models continue to improve and performance gaps narrow, the primary challenge is no longer the introduction of additional evaluation axes, but the design of evaluation protocols that measure these axes in a consistent and reliable manner. In contrast to prior work that focuses on what to evaluate, we focus on how to evaluate each axis, and propose a skill-aligned evaluation methodology that adapts annotation protocols to the structure of each evaluation skill. 2.2 Annotation Strategies in T2I Evaluation Human evaluation remains the gold standard for assessing T2I generation, and most prior work relies on large-scale user studies to collect evaluation signals. A common approach is comparative evaluation, where annotators choose between images generated by different models [ 16 , 37 , 36 , 25 ] . Such comparisons reduce absolute calibration requirements and often yield consistent judgments. However, comparative evaluation is inherently relative : it depends on the choice of baselines and does not provide direct insight into which specific aspects of an image drive a preference [ 26 ] . Moreover, as the number of models increases, pairwise comparisons scale quadratically and become prohibitively expensive, while offering limited diagnostic value for identifying failure modes. For these reasons, we focus on absolute evaluation, where annotators assess individual images against predefined criteria using mechanisms such as Likert-scale ratings or binary question answering (BQA) [ 3 , 12 , 9 , 39 , 35 , 29 ] . Unlike pairwise comparisons, this paradigm supports fine-grained, skill-level analysis and direct attribution of errors, while avoiding dependence on explicit baselines. However, its effectiveness depends strongly on annotation mechanism design, since uniform protocols may not measure heterogeneous skills with equal reliability. 2.3 Automated Evaluation of T2I Models To reduce the cost and latency of human evaluation, prior work seeks to automate T2I assessment using models trained on signals derived from user studies. A common approach is to fine-tune CLIP-based models or related architectures as preference or scoring functions [ 16 , 39 , 36 , 41 ] , or to employ learned models to answer task-specific VQA-style queries that probe semantic correctness [ 3 , 12 , 21 , 24 ] . While these methods improve

Read the complete text