MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans

Shubhankar BorseSeokeon ChoiSunghyun ParkJeongho KimShreya Kadambi, Risheek Garrepalli Sungrack Yun Munawar Hayat Fatih Porikli Qualcomm AI Research · 2025

Read MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans by Shubhankar BorseSeokeon ChoiSunghyun ParkJeongho KimShreya Kadambi, Risheek Garrepalli Sungrack Yun Munawar Hayat Fatih Porikli Qualcomm AI Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20879.txt.

Introduction

Shubhankar Borse and co-authors built a benchmark for testing how well image generators handle pictures with several people in them in a 2025 paper. Keeping each person's face recognizable while they perform actions is a known weak point, and no shared test existed. The new set, MultiHuman-Testbench, contains 1,800 prompts ranging from simple to complex actions.

The prompts are matched with 5,550 face photos chosen for balance across age, background, and gender, plus pose images picked by people to fit each prompt. Scoring covers face count, identity match, prompt fit, and action detection. The team tested many models, with and without extra layout guidance, and found that isolating regions with segmentation and careful matching lifted identity scores a lot.

Catalog record

CREATORShubhankar BorseSeokeon ChoiSunghyun ParkJeongho KimShreya Kadambi, Risheek Garrepalli Sungrack Yun Munawar Hayat Fatih Porikli Qualcomm AI Research
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20879

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MultiHuman-Testbench: Benchmarking Image Generation for Multiple Humans Shubhankar BorseSeokeon ChoiSunghyun ParkJeongho KimShreya Kadambi, Risheek Garrepalli Sungrack Yun Munawar Hayat Fatih Porikli Qualcomm AI Research Abstract Generation of images containing multiple humans, performing complex actions, while preserving their facial identities, is a significant challenge. A major factor contributing to this is the lack of a a dedicated benchmark. To address this, we introduce MultiHuman-Testbench, a novel benchmark for rigorously evaluating generative models for multi-human generation. The benchmark comprises 1800 samples, including carefully curated text prompts, describing a range of simple to complex human actions. These prompts are matched with a total of 5,550 unique human face images, sampled uniformly to ensure diversity across age, ethnic background, and gender. Alongside captions, we provide human-selected pose conditioning images which accurately match the prompt. We propose a multi-faceted evaluation suite employing four key metrics to quantify face count, ID similarity, prompt alignment, and action detection. We conduct a thorough evaluation of a diverse set of models, including zero-shot approaches and training-based methods, with and without regional priors. We also propose novel techniques to incorporate image and region isolation using human segmentation and Hungarian matching, significantly improving ID similarity. Our proposed benchmark and key findings provide valuable insights and a standardized tool for advancing research in multi-human image generation. 1 Introduction Figure 1: MultiHuman Testbench . Our MultiHuman Testbench consists of 5,550 IDs across 1800 samples, including captions describing a scene with of 1-5 humans. While current text-to-image diffusion models can generate high fidelity images, generating scenes featuring multiple humans (from provided reference images) performing text-described actions still remains a challenge. It requires simultaneously preserving visual characteristics of multiple subjects, accurately rendering their relative positions and interactions, and seamlessly integrating them into the synthesized background. However, current methods [ 41 , 12 , 40 ] frequently exhibit issues such as identity blending, generating the incorrect number of humans, or difficulties in composing the scene according to the text. To make the task easier, some works [ 19 , 6 ] adopt regional priors as an input to the model, such as human poses, bounding boxes or segmentation masks. While this improves performance, it hinders usability as pose or mask information might not be readily available. A major challenge in multi-human generation is the lack of a comprehensive and standardized benchmark, along with well defined metrics. Existing benchmarks typically focus on single-subject fidelity [ 9 ] , general text-to-image quality [ 4 , 46 ] , or multi-object compositional tasks [ 22 , 11 ] . However, none of the currently available open benchmarks address the added complexity of generating multiple distinct humans. To address this issue, we introduce MultiHuman-Testbench, a novel and challenging benchmark. It is built upon a dataset of 1800 samples, which include carefully crafted text prompts describing scenes with 1 to 5 humans, paired with 5,550 reference human faces, sampled from open-source datasets. We ensure diversity in age, ethincity, and gender. As many current works rely on regional priors in multi-human scenes [ 19 , 6 ] , we provide pose conditioning images. Additionally, we propose a multi-faceted evaluation framework designed to capture the nuances of multi-human generation. We propose four complementary metrics: Count Accuracy, Hungarian ID similarity, Human Preference Score, Multimodal LLM (MLLM) question-answering to probe the correctness of simple and complex actions. The proposed testbench has four different tasks: 1) Reference-based Multi-Human Generation in the wild. 2) Reference-based Multi-Human Generation with Regional Priors. 3) ID-Consistent Multi-Human Generation without Reference Images. 4) Text-to-Image Multi-Human Generation. We benchmark current models and identify key areas for improvement. Overall, most methods without regional priors struggle in generating the correct number of people. While proprietery models such as GPT-Image-1 generates plausible images, it lacks preserving facial features and has poor ID retention. We also study biases in current models, in terms of gender, age, status, and ethnicity. Reference-based Multi-Human Generation in the wild, is the most challenging and least restrictive task in our testbench. We propose new techniques (Sec. 3 ) to adapt current methods for improving their performance this task. Specifically, for unified multi-modal architectures [ 41 , 27 ] , we propose a method to isolate the reference images to impact only a specific region within the latent space. To match each reference image to regions, we propose an implicit Hungarian matching guided by human segmentation. Our method enhances the ability to maintain individual identities, reducing subject leakage and improving ID similarity. We extend our proposed techniques to two models, OmniGen [ 41 ] and IR-Diffusion [ 10 ] , resulting in our proposed MH-OmniGen and MH-IR-Diffusion. In summary, our contributions are: • Introduction of a novel, large-scale benchmark for multi-human ID image generation, featuring diverse subjects, text, and pose conditioning. • A comprehensive evaluation suite designed to assess multi-human generation fidelity, including people count accuracy, ID similarity, text-alignment, and MLLM-based assessment. • An extensive empirical evaluation and thorough analysis of 30 state-of-the-art zero-shot and training-based generative methods on four different tasks. • A novel training-free enhancement for existing multi-human generation methods, utilizing regional isolation and matching for improved identity and compositional control. 2 Related Work Native Text-to-Image models: Multiple diffusion based models have been proposed recently [ 29 , 32 , 33 , 18 , 2 ] . These models exhibit excellent text-to-image generation ability and can be used as base models for generating multiple humans. Multi-human Generation with native Text-to-Image models: To generate ID-consistent or subject driven images with text-to-image models, recent works employ auxiliary models such as IP-Adapter [ 43 ] or ControlNet [ 45 ] . There are also tuning-based approaches which exist such as LoRA [ 13 ] or MudI [ 36 ] for this purpose. These methods typically fall short on multi-human generation in the wild, without any regional priors. ID-Consistent Multi-Human Generation without reference images: Methods such as Consistory [ 36 ] , DreamStory [ 11 ] , IR-Diffusion [ 10 ] and StoryDiffusion [ 47 ] have recently gained popularity in ID-consistent multi-subject generation without reference images. These methods generate human faces and use these faces to generate multiple images for tasks like storytelling. Multi-Object Generation: Recent approaches e.g., MS-Diffusion [ 38 ] , MIP-Adapter [ 14 ] , Lambda-Eclipse [ 28 ] , have shown significant performance gain for incorporating multiple objects in the same scene. These can include daily items and in some cases, pets such as dogs and cats. However, they struggle to adapt to multi-human generation as zero-shot ID-preservation is a highly challenging task. Multi-human Generation with native Multi-Modal models: Unified multimodal models, such as OmniGen [ 41 ] , Show-O [ 42 ] , OneDiffusion [ 23 ] ACE++ [ 26 ] , GPT-Image-1 [ 27 ] and JanusFlow [ 25 ] process the text and vision via same transformer backbone and have shown promises for subject-driven generation. These methods input both the reference images and text prompt in a unified token space, removing the need for additional auxiliary task-specific networks such as IP-Adapter/ControlNet. Omnigen [ 41 ] was further tuned for ID-preservation. Our evaluations show that among all open-source models, Omnigen produces best results. Regional Isolation: For networks generating images using simply text inputs, recent works such as IR-Diffusion

Read the complete text