ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation

Tencent Hunyuan Team · 2025

Read ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation by Tencent Hunyuan Team (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04952.txt.

Description

Language models now write web pages that move and respond, but standard coding tests only check whether the logic runs, not whether the result looks or feels right. Chenchen Zhang and co-authors built ArtifactsBench to close that gap. Their system renders each model's output, screenshots its behaviour over time, and hands the pictures plus the code to a multimodal judge model armed with a per-task checklist.

The benchmark holds 1,825 varied tasks, and more than thirty leading models were run through it. The automated rankings matched a popular human-preference arena ninety-four percent of the time, and agreed with human experts on over ninety percent of paired comparisons. Generalist models often beat coding specialists. Everything is open-sourced for the community.

Catalog record

CREATORTencent Hunyuan Team
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04952

Browse the full library · About the Tower of Babel