TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design

Haonan Zhu ∗, Elad Hirsch ∗, Alexandria Minetti, Allison Nulty, Purvanshi Mehta † · 2026

Read TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design by Haonan Zhu ∗, Elad Hirsch ∗, Alexandria Minetti, Allison Nulty, Purvanshi Mehta † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20731.txt.

Introduction

Haonan Zhu and five colleagues release TASTE, a designer-annotated preference dataset for AI-generated graphic design. Text-to-image models now produce design at scale, yet supervision still comes from photo-style sets with one overall verdict per comparison. Designers instead judge typography, layout, colour harmony, and more as separate axes. TASTE has two cohorts of five professional designers each ranking outputs from four models across nine criteria, with hallucination flags per image.

Catalog record

CREATORHaonan Zhu ∗, Elad Hirsch ∗, Alexandria Minetti, Allison Nulty, Purvanshi Mehta †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69994
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20731

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TASTE: A Designer-Annotated Multi-Dimensional Preference Dataset for AI-Generated Graphic Design Thanks: ∗ Equal contribution. † Corresponding author. Haonan Zhu ∗, Elad Hirsch ∗, Alexandria Minetti, Allison Nulty, Purvanshi Mehta † Abstract Text-to-image models now generate graphic design at production scale, yet their supervision still comes primarily from photo-style preference datasets with a single overall verdict per comparison. Designers evaluate designs along several distinct axes (e.g., typography, layout, color harmony) that a single preference label collapses. We release TASTE (Typography, Aesthetics, Spatial, Tone, Etc.) , a multi-dimensional preference dataset in which two disjoint cohorts of five professional designers each ranked outputs from four current text-to-image models across nine criteria along with per-image hallucination flags. We pair the dataset with two contributions. First, a criterion-agnostic signal-validation framework based on Kendall’s τ \tau , majority-vote probability, and Condorcet cycles against exact iid-uniform nulls; the analysis reveals significant but moderate designer agreement, with every TASTE criterion rejecting the random-rater null. Second, we benchmark preference models on TASTE and find that off-the-shelf VLM judges and dedicated T2I scorers fail to reach majority agreement with the designer panel, while a small MLP head trained directly on TASTE substantially narrows the gap to the single-rater ceiling, setting a baseline for future TASTE-trained preference models. TASTE: Designer-Annotated Preference for AI Design † † email: [email protected] † † email: [email protected] † † email: [email protected] † † email: [email protected] † † email: [email protected] † † affiliation: Lica World, San Francisco, USA. † † affiliation: Contra.Work Inc., Brooklyn, NY, USA. 1 Introduction Text-to-image (T2I) models have crossed from research artifact to deployed design tool. Current systems generate UI mockups, posters, ad creatives, and typographic compositions end-to-end and ship them to end users in production design pipelines; a recent survey of AI-driven graphic design [ 42 ] catalogues the scope of this shift. Yet these systems are still largely evaluated and optimized using preference signals derived from photographic generation. This creates a misalignment between what models are rewarded for and what downstream design users actually value, potentially favoring photorealistic plausibility over typography, hierarchy, or structural fidelity. The public preference datasets that drive evaluation, reward modeling and preference optimization are collected on photo-style generation rather than on design. HPSv2 and HPSv3 [ 36 , 28 ] , Pick-a-Pic [ 17 ] , ImageReward [ 38 ] , RichHF-18K [ 24 ] , MHP/MPS [ 39 ] , and VisionReward [ 37 ] all collect preferences over photo-style outputs. Survey work [ 42 ] confirms the gap, and DesignPref [ 31 ] describes a UI-design pairwise dataset whose data has not yet been released. A complementary line of recent work targets the structural and task-level side of design: LICA [ 13 ] provides 1.55M layered design compositions with structural metadata, and Graphic-Design-Bench [ 8 ] evaluates current models on 50 design tasks across layout, typography, infographics, template semantics, and animation. These resources test what models can do ; neither records what designers prefer . Sub-dimensional preference remains a missing piece. Photo-style preference is dominated by a small set of objective cues such as anatomical plausibility and prompt alignment, so a single “which is better” label captures most of the signal. Graphic design splits along several partly independent axes including typography rules, visual hierarchy, color harmony, mood and tone, spatial layout fidelity, and description-fidelity criteria of the form does the color requested in the prompt actually appear in the output . A single overall label averages over these axes and loses the per-axis signal. A layout may preserve spatial structure while violating color intent; another may satisfy prompt semantics but break typographic hierarchy. Both could receive the same overall preference label despite failing for different reasons. Work on visual importance and saliency in graphic design supports treating these axes as distinct cognitive processes [ 5 , 9 ] , and multi-axis benchmarking is standard practice in adjacent modalities, for example VBench and VBench-2.0 in video [ 14 , 40 ] . Building a design-preference dataset is not sufficient on its own: the ratings must encode stable, learnable signal rather than arbitrary ratings. Similar aggregate disagreement can reflect random ratings, a weak shared preference with rater-specific noise, or designer groups with internally consistent but incompatible tastes. We propose a three-part diagnostic for separating these cases: graded agreement (per-prompt Kendall’s \tau ), majority decisiveness (majority-vote probability p_{\max} ), and triadic intransitivity (Condorcet cycle existence). Tested against exact null distributions at (p=4,R=5) , every TASTE criterion rejects the random-rater null on at least two of the three statistics. Three subsampled reference datasets at the same shape (Sushi [ 15 ] , MovieLens [ 12 ] , and the HPSv2 test data [ 36 ] ) place designer agreement on graphic design between food and movie preferences and below photo-style image-quality preference. Signal level varies across criteria, with typography and spatial accuracy carrying roughly twice the signal of color harmony. Beyond TASTE, we argue that this kind of analysis should be standard practice on any new preference dataset before model training begins, since it answers the prior question of whether learnable signal is present. Off-the-shelf systems do not yet read this signal. Across six open-weight VLM judges spanning four families and three dedicated T2I-preference scorers (HPSv2.1, PickScore-v1, LAION-Aesthetic-V2), no pre-trained system exceeds 0.55 macro agreement with the 5-designer majority on TASTE under a position-bias-corrected protocol [ 41 ] . Within the VLM slate, position-bias rate and content sensitivity trade off (Spearman \rho{=}{+}0.94 ), so scaling moves the operating point along this frontier rather than to a better one. TASTE is designed not only as a benchmark, but as a decision layer for design generation systems. Its criterion-level structure enables fine-grained evaluation of model strengths and weaknesses, allowing practitioners to select or route among generators based on task-specific priorities (e.g., typography fidelity, spatial accuracy, or color control) rather than relying on a single aggregate preference score. This same structure provides supervision for training preference judges, reward models, and alignment objectives that optimize for specific design dimensions, enabling more controllable and designer-aligned generation. Hence, our work makes the following contributions: 1. We release TASTE 1 1 1 Code: https://github.com/purvanshi-lica/taste . Data: https://huggingface.co/datasets/purvanshi/TASTE . (Typography, Aesthetics, Spatial, Tone, Etc.) , a designer-annotated preference dataset for AI-generated graphic design recording one rating per criterion across nine criteria, with ten professional designers ranking outputs from four current T2I models (1,600 ratings per criterion, plus hallucination flags on overall-preference cohorts). Both pairwise comparisons and derived 4-way rankings are released. 2. We introduce a criterion-agnostic diagnostic framework for validating preference-signal quality in new datasets, based on graded agreement, majority decisiveness, and triadic intransitivity. Applied with cross-domain anchors from Sushi, MovieLens, and HPSv2-test at the same shape, every TASTE criterion rejects the random-rater null and designer agreement on graphic design sits between subjective-taste and photo-quality regimes. 3. We benchmark preference models on TASTE. Nine pre-trained systems, three dedicated T2I-preference scorers (HPSv2.1, PickScore-v1, LAION-Aesthetic-V2) and six open-weight VLMs prompted as judges across four families and an order-of-magnitude range in active parameter count, all fall below 0.55 macro agreement with the 5-designer majority, and scaling within the VLM slate does not close the gap. A small MLP head trained directly on TASTE reaches 0.611 pairwise accuracy, approaching the 0.741

Read the complete text