MaSC: A Masked Similarity Metric for Evaluating Concept-Driven Generation
Patryk Bartkowiak, Lennart Petersen, Bartosz Kotrys, Dominik Michels, Soren Pirk, Wojtek Palubicki · 2026
Read MaSC: A Masked Similarity Metric for Evaluating Concept-Driven Generation by Patryk Bartkowiak, Lennart Petersen, Bartosz Kotrys, Dominik Michels, Soren Pirk, Wojtek Palubicki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22469.txt.
Introduction
Bartkowiak and colleagues introduce MaSC, a masked metric for text-to-image concept generation. Existing metrics judge the whole image, not subject versus background, and match human views poorly. MaSC uses foreground masks to split evaluation into concept preservation and prompt following. Concept preservation is scored by masked matching of foreground patches. Prompt following compares background and subject-stripped prompt embeddings. On DreamBench++ ratings, MaSC reaches Krippendorff alpha of 0.471, beating non-LLM baselines and GPT-4V. On ORIDa, it reaches AUC of 0.992, separating same from cross-subject pairs.
Catalog record
| CREATOR | Patryk Bartkowiak, Lennart Petersen, Bartosz Kotrys, Dominik Michels, Soren Pirk, Wojtek Palubicki |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56688 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22469
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MaSC: A Masked Similarity Metric for Evaluating Concept-Driven Generation Patryk Bartkowiak, Lennart Petersen, Bartosz Kotrys, Dominik Michels, Soren Pirk, Wojtek Palubicki Abstract Evaluating single-concept personalization in text-to-image diffusion has seen two categories of quantitative metrics: Concept Preservation (CP) measures identity fidelity to a reference while Prompt Following (PF) measures whether the generated scene matches the prompt. Personalization papers have commonly computed these signals using three separate backbones: CLIP-I and DINO for CP, CLIP-T for PF. In this paper, we show that existing metrics fall short of correlating with human perception because they attend to the image as a whole, instead of distinguishing the concept subject from the background. This distinction is important from a human perception point of view as the concept subject in the output image should be very similar to the input concept (CP), whereas the output background should adhere closely to the text prompt (PF). To improve personalization evaluation in this way, we introduce MaSC , a unified metric that attends to CP and PF by differentiating concept subject regions. Specifically, given an externally provided foreground concept mask, MaSC computes the CP and PF scores from a single forward pass of a frozen SigLIP2 encoder per image. On DreamBench++ human ratings, MaSC reaches Krippendorff α = + 0.471 \alpha=+0.471 on CP — beating every non-LLM baseline tested (DINOv3, DreamSim, AM-RADIO, DIFT-SDXL, DINO-I, CLIP-I) and GPT-4V, while sitting within Δ α = 0.028 \Delta\alpha=0.028 of GPT-4o. To distinguish human perception bias we also evaluate our method, as well as common SOTA baselines, on ORIDa, a real-photo benchmark of identity preservation across physical environments. In this experiment MaSC reaches AUC = 0.992 =0.992 almost perfectly identifying concept subjects. The PF score, obtained by MaSC without a second encoder forward pass, beats the CLIP-T baseline shipped with DreamBench++. In summary, our comprehensive evaluations demonstrate that MaSC establishes a new state-of-the-art for non-LLM concept preservation, while providing an efficient, unified standard for personalization evaluation. We release MaSC as a pip-installable Python package, alongside independent reproductions of every comparator’s published numbers. 1 Introduction The ability to personalize text-to-image (T2I) generation to user-supplied subjects has fundamentally transformed generative AI. Methods such as DreamBooth 25 , Textual Inversion 6 , and IP-Adapter 33 have made subject-driven generation a core research pillar and a standard T2I capability. However, reliably evaluating the success of these personalizations remains a difficult task. A complete evaluation requires capturing two distinct signals: Concept Preservation (CP), measuring the identity fidelity of the rendered subject to its reference, and Prompt Following (PF), measuring how well the generated scene satisfies the prompt. In the past, the field computed these signals using three independent backbones inherited from DreamBooth: CLIP-I 20 and DINO 2 for CP, and CLIP-T for PF. Recently, DreamBench++ 19 exposed the severe limitations of this standard trio, revealing that its correlation with human judgments plateaus at a Krippendorff 8 \alpha\approx 0.3 on both signals—roughly half the inter-annotator ceiling ( \alpha=0.658 for CP, 0.563 for PF). To improve alignment with human judgments, the community has generally taken two approaches: updating the representation encoders, or using Large-Language Models (LLMs) like GPT-4V 18 and GPT-4o 17 as evaluators. The first approach offers only minor improvements, while the second performs well but relies on API-bound, non-differentiable scoring that lacks per-region attribution. The plateau is not a limitation of the encoders. Global pooling introduces a distinct failure for each evaluation signal: (i) Global cosine for CP averages over the whole image, including background variation that humans correctly ignore when judging identity; (ii) Global text-image cosine for PF combines two distinct quantities: scene adherence to the prompt and the presence of the subject class (a static factor, since the subject is fixed across all test prompts for a given concept). Both failures are consequences of the aggregator and the input space, not the backbone — any encoder under mean or [CLS] pooling inherits them. These limitations are addressable at inference time, without retraining, via spatial decomposition. We introduce MaSC , which uses three lightweight interventions: (i) for each foreground reference patch, taking the maximum cosine against any output patch and averaging under the foreground mask; (ii) applying SigLIP2’s trained attention pooler over the patch tokens with foreground patches masked, yielding a background-only embedding via the pooler’s native masking support; and (iii) excluding the canonical subject name from the prompt, eliminating the static baseline on the text side. A single SigLIP2 SO400M-NaFlex 30 forward pass per image then produces both scores: CP from the patch grid via masked max-cosine, and PF from the masked attention pool against the stripped-prompt text embedding. MaSC’s spatial decomposition closes most of the performance gap with LLM judges in personalization evaluation. On DreamBench++, MaSC is the only non-LLM metric to outperform GPT-4V on CP. On ORIDa 7 , a real-photo identity-discrimination benchmark across 50 subjects in 10 environments, MaSC is the first non-LLM metric to surpass GPT-4o. Furthermore, the PF score outperforms the CLIP-T baseline shipped with DreamBench++. Contributions. We propose MaSC, a unified personalization metric that produces CP and PF scores from a single SigLIP2 SO400M-NaFlex forward pass per image, given an externally supplied foreground concept mask. We make five contributions: (1) On DreamBench++ human ratings, MaSC achieves Krippendorff \alpha=+0.471 on CP — outperforming every non-LLM baseline tested, surpassing GPT-4V, and reaching 72\% of the human inter-rater ceiling; (2) On ORIDa, MaSC is the first non-LLM CP metric to outscore GPT-4o, reaching AUC =0.992 ; (3) Same-backbone ablations isolate the contribution to our spatial decomposition strategy rather than the encoder choice, with this approach accounting for \Delta\alpha=+0.102 on DreamBench++ and \Delta\,\mathrm{AUC}=+0.038 on ORIDa; (4) At a matched parameter budget, MaSC outperforms state-of-the-art distilled vision transformers (e.g., AM-RADIO 21 ), demonstrating that late-stage explicit masking combined with patch features consistently outperforms internalized mask supervision for personalization evaluation; (5) The PF score, obtained from the same forward pass without an additional vision-encoder call, beats the CLIP-T baseline shipped with DreamBench++. 2 Related Work Subject-driven text-to-image personalization spans optimization-based methods that fine-tune or augment the generator with a reference subject (DreamBooth, Textual Inversion) and adapter-based methods that inject reference features at inference (IP-Adapter, BLIP-Diffusion 10 , Emu2 28 ). While these generative techniques form the core of the personalization literature, our work focuses strictly on the evaluation metrics used to benchmark them. DreamBooth introduced the evaluation protocol of using CLIP-I and DINO-I for image-image fidelity alongside CLIP-T for image-text adherence. This protocol has since been universally adopted. However, DreamBench++ 19 recently benchmarked these metrics against human judgments across an evaluation grid of 7 personalization methods, 150 subjects, and 9 prompts. They demonstrated that with CLIP-I at Krippendorff \alpha=+0.135 , DINO-I and CLIP-T plateau near \alpha\approx 0.3 , roughly half the human inter-annotator ceiling ( \alpha=0.658 for CP, 0.563 for PF). Our work directly addresses this critical evaluation gap. To complement DreamBench++, we also evaluate on ORIDa, a real-photo benchmark of identity preservation across natural environments. While ORIDa lacks human ratings, its inclusion of ground-truth per-object segmentation masks makes it an ideal real-world out-of-distribution test for CP. Several non-LLM alternatives have been proposed for image-image fidelity beyond CLIP-I and DINO-I. DreamSim 5 integrates CLIP, DINO, and OpenCLIP backbones and fine-tunes on the NIGHTS triplet dataset