Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity

Yuhan Zhang 1 , 2 ∗ 1,2* Long Zhuo 2 ∗ 2* Ziyang Chu 2 , 3 ∗ 2,3* Tong Wu 4 † 4{\dagger} Zhibing Li 2 , 5 2,5 Liang Pan 2 † 2{\dagger} Dahua Lin 2 , 5 2,5 Ziwei Liu 6 † 6{\dagger} · 2025

Read Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity by Yuhan Zhang 1 , 2 ∗ 1,2* Long Zhuo 2 ∗ 2* Ziyang Chu 2 , 3 ∗ 2,3* Tong Wu 4 † 4{\dagger} Zhibing Li 2 , 5 2,5 Liang Pan 2 † 2{\dagger} Dahua Lin 2 , 5 2,5 Ziwei Liu 6 † 6{\dagger} (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05609.txt.

Introduction

Yuhan Zhang and seven co-authors build a grading system for computer-generated 3D content, in a 2025 paper. Software that creates three-dimensional objects from text is racing ahead, but judging the output still leans on flat image scores that miss depth, materials, and small parts. Their framework, Hi3DEval, grades at two levels: the whole object and its parts.

It also grades surface materials directly, judging color base, saturation, and metallic feel rather than looks alone. To train and test the grader, the team built a large dataset of varied 3D assets with careful labels, produced with the help of several AI agents. Their automated scorer watches rendered videos for whole-object and material grades and uses learned 3D features for part grades. In tests it matched human preferences better than image-based scores, offering a scalable stand-in for manual review.

Catalog record

CREATORYuhan Zhang 1 , 2 ∗ 1,2* Long Zhuo 2 ∗ 2* Ziyang Chu 2 , 3 ∗ 2,3* Tong Wu 4 † 4{\dagger} Zhibing Li 2 , 5 2,5 Liang Pan 2 † 2{\dagger} Dahua Lin 2 , 5 2,5 Ziwei Liu 6 † 6{\dagger}
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-05609

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity Yuhan Zhang 1 , 2 ∗ 1,2* Long Zhuo 2 ∗ 2* Ziyang Chu 2 , 3 ∗ 2,3* Tong Wu 4 † 4{\dagger} Zhibing Li 2 , 5 2,5 Liang Pan 2 † 2{\dagger} Dahua Lin 2 , 5 2,5 Ziwei Liu 6 † 6{\dagger} Abstract Despite rapid advances in 3D content generation, quality assessment for the generated 3D assets remains challenging. Existing methods mainly rely on image-based metrics and operate solely at the object level, limiting their ability to capture spatial coherence, material authenticity, and high-fidelity local details. 1) To address these challenges, we introduce Hi3DEval , a hierarchical evaluation framework tailored for 3D generative content. It combines both object-level and part-level evaluation, enabling holistic assessments across multiple dimensions as well as fine-grained quality analysis. Additionally, we extend texture evaluation beyond aesthetic appearance by explicitly assessing material realism, focusing on attributes such as albedo, saturation, and metallicness. 2) To support this framework, we construct Hi3DBench , a large-scale dataset comprising diverse 3D assets and high-quality annotations, accompanied by a reliable multi-agent annotation pipeline. We further propose a 3D-aware automated scoring system based on hybrid 3D representations. Specifically, we leverage video-based representations for object-level and material-subject evaluations to enhance modeling of spatio-temporal consistency and employ pretrained 3D features for part-level perception. Extensive experiments demonstrate that our approach outperforms existing image-based metrics in modeling 3D characteristics and achieves superior alignment with human preference, providing a scalable alternative to manual evaluations. Project page is available at https://zyh482.github.io/Hi3DEval/ . Figure 1: Overview of Hi3DEval , a unified framework for 3D generation evaluation with three key components: 1) Hierarchical evaluation protocols that jointly assess object-level and part-level quality, with extended material evaluation via reflectance cues. 2) A large-scale benchmark featuring a diverse set of 3D generative models and extensive human-aligned annotations generated via a multi-agent, multi-modal LLMs pipeline. 3) A hybrid automated scoring system that integrates video-based and naive 3D-based representations to enhance evaluators’ perceptions of 3D structure. 1 Introduction Creating vivid and high-fidelity 3D assets remains a fundamental yet challenging problem in computer vision and graphics, with wide-ranging applications in gaming, virtual and augmented reality, and robotics. In recent years, the field has witnessed significant breakthroughs, driven by the emergence of large-scale datasets Objaverse ; Objaverse-xl , expressive neural representations Nerf ; 3dgs , sophisticated optimization techniques Dreamfusion ; Mvdream ; Luciddreamer ; Magic3d ; Fantasia3d , and powerful network architectures Trellis ; CLAY ; hunyuan3d ; Triposr . As the visual realism of generated content continues to advance, establishing reliable and fine-grained metrics for evaluating and comparing different approaches has become a critical objective. Existing 3D evaluation frameworks can be broadly categorized into two paradigms: training-free protocols and data-driven evaluators. The former often extend conventional 2D metrics to 3D domains through handcrafted heuristics T3_Bench ; GT23D-Bench , while methods like GPTEval3D GPTEval3D leverage powerful Multimodal Language Models (MLLMs) such as GPT-4V 1 1 1 GPT-4V system card: https://openai.com/index/gpt-4v-system-card with multi-view renderings to enhance 3D reasoning. In pursuit of greater transparency and alignment with human judgment, recent efforts (e.g. 3DGen-Bench 3DGen-Bench , Gen3DEval Gen3DEval , T23DAQA T23DAQA ) explore training lightweight scoring models supervised by human annotations or pseudo-labels generated by GPT-4V. Despite these advances, current approaches remain coarse-grained and overlook material attributes when evaluating textures. Moreover, base on 2D renderings, these methods inherently struggle to capture the spatial continuity and structural complexity of 3D assets, limiting their reliability and robustness. To address these limitations, we present a unified evaluation framework, Hi3DEval, that supports both image- and text-conditional 3D generation and advances 3D evaluation in the following dimensions: 1) Hierarchical evaluation scheme across varying granularities. We introduce a hierarchical evaluation scheme, supporting both object-level and part-level assessment. Specifically, the object-level evaluation provides a holistic evaluation of generated 3D assets, considering geometry, texture, and prompt alignment. While the part-level evaluation enables localized diagnosis of quality issues within semantic regions, enhancing interpretability and failure analysis. Together, they provide a more comprehensive view of generation performance. 2) Physical material evaluation with reflectance cues. We propose a material-subject evaluation protocol that goes beyond aesthetic-level judgments to explicitly assess core physical properties, such as albedo, saturation, and metallicness. In practice, we use reflectance cues under diverse illumination to simulate realistic perception scenarios, enabling robust assessment across diverse 3D representations, even when PBR are not explicitly disentangled during generation. 3) Large-scale dataset with human-aligned annotation pipeline. We construct a large-scale comprehensive dataset involving all aforementioned dimensions, dubbed Hi3DBench. To trade off the subjectivity of purely manual labeling and the inconsistency of purely GPT-based labeling with human judgment, we introduce a multi-agent, multi-modal annotation pipeline, producing more consistent and faithful assessments. 4) Hybrid 3D representation-based automated scoring system. To overcome the limitations of static image representation, we propose a 3D-aware automated scoring system leveraging hybrid 3D representations. To be specific, we use video-based representations to enhance evaluators’ understanding of spatio-temporal consistency for object-level and material-subject evaluation, and we apply pretrained geometric embeddings for the part-level to achieve deep shape perception. Experimental results show that such hybrid 3D-aware signals yield more reliable and human-consistent evaluation than conventional static image-based approaches. Specifically, the proposed video-based scoring pipeline, adopted for the object-level and material-subject evaluations, achieves superior pairwise alignment with humans across all assessed dimensions. Furthermore, we conduct qualitative studies on the part-level scoring model, demonstrating that it exhibits capability in localizing generation flaws, enabling fine-grained diagnostic analysis. 2 Related works 3D object generation. Prior work in 3D content generation has explored a variety of approaches, including leveraging 2D generative models, incorporating 3D geometric priors, and directly learning from large-scale 3D datasets. DreamFusion Dreamfusion leverage text-to-image diffusion model diffusion ; Stable_diffusion to optimize differentiable 3D representations through Score Distillation Sampling (SDS). Subsequent methods Magic3d ; SJC ; ProlificDreamer ; Latent-NeRF ; Luciddreamer ; Dreamcraft3d ; Make-it-3d ; Zero-1-to-3 have refined optimization framework and SDS loss to enhance visual quality. Another line of research SyncDreamer ; One-2-3-45++ ; Mvdream ; Wonder3D focuses on adapting 2D generative models for multi-view synthesis, providing strong 3D priors that enable reconstruction of 3D assets via sparse-view reconstruction techniques or Large Reconstruction Models Lrm ; LGM ; Instant3d ; GRM ; Instantmesh ; SV3D ; Crm . More recently, native 3D generation approaches CLAY ; Triplane ; Trellis ; hunyuan3d ; Triposr ; 3dtopia-xl have achieved state-of-the-art performance by training directly on 3D data collections Objaverse ; Objaverse-xl ; Omniobject3d , significantly improving both geometric fidelity and computational efficiency. As generation techniques evolve, the need for an efficient, robust, and comprehensive 3D evaluation framework has become increasingly urgent. 3D generation evaluation. Early approaches to evaluating 3D content often relied on labor-intensive user studies or 2D-based metrics such as CLIP Score CLIPScore ; Clipsim and Aesthetic Score Laion-5B , which are insufficient for capturing the holistic structure of 3D assets. To address these limitations, T^{3} Bench T3_Bench proposed aggregating multi-view image features via hand-crafted formulations. With the growing adoption of the "LLM-as-a-Judge" LLM-as-a-Judge paradigm, GPTEval3D GPTEval3D utilized GPT-4V to perform pairwise comparisons and established a leaderboard using Elo scoring elo . However, reliance on a closed-source model raises concerns about reproducibility and transparency. Subsequent efforts bootstrap3d ; 3DGen-Bench ; Gen3DEval fine-tuned

Read the complete text