3D Arena: An Open Platform for Generative 3D Evaluation
Dylan Ebert · 2025
Read 3D Arena: An Open Platform for Generative 3D Evaluation by Dylan Ebert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18787.txt.
Introduction
Dylan Ebert presents 3D Arena in 2025, an open platform that ranks 3D generation models by human taste rather than by formula scores. Set measures often miss what people find good in a made object, ignoring shape or surface charm. Pairwise votes from real users offer a firmer yardstick.
Since mid 2024, the platform has gathered over one hundred twenty thousand votes from about eight thousand users across nineteen models. A chess style rating turns votes into ranks, and fraud checks keep voters honest. Textured models crushed plain ones in scores, and one output style beat mesh forms. Ebert shares lessons for builders.
Catalog record
| CREATOR | Dylan Ebert |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18787
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3D Arena: An Open Platform for Generative 3D Evaluation Dylan Ebert Abstract Evaluating Generative 3D models remains challenging due to misalignment between automated metrics and human perception of quality. Current benchmarks rely on image-based metrics that ignore 3D structure or geometric measures that fail to capture perceptual appeal and real-world utility. To address this gap, we present 3D Arena, an open platform for evaluating image-to-3D generation models through large-scale human preference collection using pairwise comparisons. Since launching in June 2024, the platform has collected 123,243 votes from 8,096 users across 19 state-of-the-art models, establishing the largest human preference evaluation for Generative 3D. We contribute the iso3d dataset of 100 evaluation prompts and demonstrate quality control achieving 99.75% user authenticity through statistical fraud detection. Our ELO-based ranking system provides reliable model assessment, with the platform becoming an established evaluation resource. Through analysis of this preference data, we present insights into human preference patterns. Our findings reveal preferences for visual presentation features, with Gaussian splat outputs achieving a 16.6 ELO advantage over meshes and textured models receiving a 144.1 ELO advantage over untextured models. We provide recommendations for improving evaluation methods, including multi-criteria assessment, task-oriented evaluation, and format-aware comparison. The platform’s community engagement establishes 3D Arena as a benchmark for the field while advancing understanding of human-centered evaluation in Generative 3D. 1 Introduction The field of Generative 3D has experienced rapid growth in recent years, with state-of-the-art models emerging every few months liu2024comprehensivesurvey3dcontent . These systems generate 3D assets—digital representations including meshes, point clouds, and volumetric data—that serve critical roles in downstream applications ranging from video game development and film production to architectural visualization and virtual reality experiences akenine2019real . While technical capabilities have advanced rapidly, standardized evaluation methods have failed to keep pace zhao2024challengesopportunities3dcontent . Current benchmarks rely predominantly on automated metrics that poorly capture real-world human preferences fardo2016formalevaluationpsnrquality ; wang2004image , creating an evaluation gap that hinders progress in the field. To address this challenge, we introduce 3D Arena 1 1 1 https://huggingface.co/spaces/dylanebert/3d-arena , an open platform for evaluating image-to-3D generation models through human preferences. Inspired by the success of Chatbot Arena chiang2024chatbot , our platform employs pairwise comparisons via an intuitive web interface where users compare anonymized 3D outputs side-by-side, as shown in Figure 1 . This approach enables direct assessment of subjective quality factors, including visual appeal, structural coherence, and perceived utility, which automated metrics fail to capture. Figure 1: The 3D Arena voting interface, showing side-by-side comparison of anonymized 3D outputs. Users can rotate, zoom, and examine 3D models before submitting their preference. Since launching in June 2024, 3D Arena has established itself as the largest human preference benchmark for Generative 3D. The platform’s impact extends beyond evaluation metrics, with research groups acknowledging the influence of 3D Arena rankings on their development priorities and research directions, demonstrating its practical value for guiding progress in the field. This paper presents an analysis of 3D Arena’s methods, data, and the insights we have derived from this human preference data. Our key contributions include: 1. Platform validation : Documentation of the methods used to create, operate, and maintain data integrity for crowdsourced 3D evaluation. 2. Evaluation insights : Analysis of human preference data revealing patterns in how humans perceive and evaluate 3D content quality. 3. Methods framework : Evidence-based recommendations for improving evaluation methods in Generative 3D, informed by observed preference patterns and cognitive mechanisms. Through analysis of this established benchmark, we provide insights for the Generative 3D community while advancing understanding of human-centered evaluation in this evolving domain. To ensure reproducibility and enable further research, we provide open access to both the platform at https://huggingface.co/spaces/dylanebert/3d-arena and the iso3d evaluation dataset at https://huggingface.co/datasets/dylanebert/iso3d . 2 Related Work 2.1 The Generative 3D Evaluation Challenge The field of Generative 3D has advanced rapidly in recent years, with state-of-the-art models emerging at an accelerating rate zhao2024challengesopportunities3dcontent . Recent advances range from Shap-E jun2023shap in 2023, to major leaps in quality like InstantMesh xu2024instantmesh in early 2024, to systems like TRELLIS xiang2025structured3dlatentsscalable , Hunyuan3D-2 zhao2025hunyuan3d20scalingdiffusion , and Hi3DGen ye2025hi3dgenhighfidelity3dgeometry released in late 2024 and early 2025. Concurrently, proprietary systems from organizations such as CSM csm2024 , Rodin rodin2024 , Tripo tripo2024 , and Meshy meshy2024 have demonstrated high-quality Generative 3D results. However, evaluation methods have failed to keep pace with this rapid advancement, creating a significant assessment bottleneck. Current approaches predominantly rely on traditional image-based metrics such as PSNR fardo2016formalevaluationpsnrquality and SSIM wang2004image , which evaluate only the final rendered output while ignoring the underlying 3D structure. While T3Bench he2024t3benchbenchmarkingcurrentprogress represents an important advancement by incorporating multi-view image consistency evaluation, these image-centric approaches fail to capture the underlying 3D structure. Attempts to address 3D structure have employed geometric metrics such as Chamfer Distance achlioptas2018learning , which measure shape accuracy through point-to-point distances. However, these geometric measures fail to capture crucial aspects of 3D quality including perceptual appeal, aesthetic coherence, and mesh topology, which are factors that determine real-world utility and professional applicability. The importance of mesh topology, in particular, has been largely overlooked by the Generative 3D research community despite its critical role in professional workflows akenine2019real . Industry practitioners consistently emphasize that clean topology with proper edge flow is essential for animation, deformation, and production pipelines parent2012computer . Recent work such as MeshAnything chen2024meshanything addresses this gap by prioritizing mesh topology over visual fidelity. Their evaluation approach involves user studies where participants vote on topology quality. However, these user studies are limited by scale and cost, requiring significant effort for a small number of models and participants. 2.2 Arena-Style Evaluation: Proven Success in Other Domains Arena-style evaluation has emerged as a powerful paradigm for large-scale model assessment, addressing the limitations of traditional benchmarking approaches through crowdsourced pairwise comparisons. Chatbot Arena chiang2024chatbot and GenAI Arena jiang2024genai have demonstrated the effectiveness of this method, providing complementary insights to expert-designed benchmarks by capturing subjective preference aspects that automated metrics cannot assess. Similar arena-style approaches have been applied across domains including image generation jiang2024genai , video generation, and text-to-image evaluation, establishing the broader applicability of community-driven preference data. These platforms leverage well-established ranking algorithms including ELO rating systems elo1978rating and the Bradley-Terry model 10.1093/biomet/39.3-4.324 , which infer relative model performance from sparse pairwise comparisons. The core method presents users with side-by-side anonymous outputs, enabling black-box preference judgments that aggregate into statistical rankings. The arena approach offers advantages for 3D evaluation: it captures subjective quality aspects that automated metrics cannot measure, scales to large participant populations for statistical reliability, and provides continuous assessment as new models emerge. These characteristics make arena-style evaluation well-suited for Generative 3D, where quality encompasses multiple dimensions including visual appeal, structural coherence, and downstream utility. Recent concurrent work has applied arena-style evaluation to Generative 3D through 3DGen-Bench zhang20253dgen . While 3DGen-Bench claims to introduce "the first 3D Arena," our 3D Arena platform launched earlier in June 2024 and has since accumulated significantly more preference data with 123,000 votes compared to 3DGen-Bench’s 13,800 votes. The platforms serve complementary roles: 3D Arena functions as an open, continuously-updated leaderboard with public voting focused specifically on image-to-3D generation, while 3DGen-Bench encompasses both text-to-3D and image-to-3D evaluation through closed expert annotation with an emphasis on an automated scoring model.