DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain

Enhao Huang, Pengyu Sun, Shuxun Wang, Zixin Lin, Alex Chen, Kaichun Hu, Joey Ouyang, Frank Li, Zhiyu Zhang, Haobo Wang, Yiming Li, Zhan Qin, James Yi, Gang Zhao, Ziang Ling, Lowes Yang · 2025

Read DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain by Enhao Huang, Pengyu Sun, Shuxun Wang, Zixin Lin, Alex Chen, Kaichun Hu, Joey Ouyang, Frank Li, Zhiyu Zhang, Haobo Wang, Yiming Li, Zhan Qin, James Yi, Gang Zhao, Ziang Ling, Lowes Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16116.txt.

Introduction

Huang and co-authors build a benchmark for testing language models on Web3, the blockchain-based corner of the internet where mistakes cost real money, in a 2025 paper. General tests miss the specialized reasoning that smart contract audits and decentralized finance demand. Their suite spans nine subdomains, from infrastructure to token economics, mixing factual questions with open-ended problems modeled on real operations.

They evaluated thirty-one leading models with checks against contamination and scoring drift. Models handled basic infrastructure concepts well but faltered on high-reasoning tasks such as security auditing. The authors add a cost-versus-performance analysis and adversarial tests showing top scores require genuine reasoning, not memorization.

Catalog record

CREATOREnhao Huang, Pengyu Sun, Shuxun Wang, Zixin Lin, Alex Chen, Kaichun Hu, Joey Ouyang, Frank Li, Zhiyu Zhang, Haobo Wang, Yiming Li, Zhan Qin, James Yi, Gang Zhao, Ziang Ling, Lowes Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16116

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain Enhao Huang, Pengyu Sun, Shuxun Wang, Zixin Lin, Alex Chen, Kaichun Hu, Joey Ouyang, Frank Li, Zhiyu Zhang, Haobo Wang, Yiming Li, Zhan Qin, James Yi, Gang Zhao, Ziang Ling, Lowes Yang Abstract . The Web3 ecosystem, underpinned by cryptographic primitives and decentralized consensus, represents a high-stakes environment where software vulnerabilities and incentive misalignments translate directly into financial loss. As Large Language Models (LLMs) are increasingly integrated into this domain for tasks ranging from smart contract auditing to decentralized finance analytics, ensuring their reliability is paramount. However, general-purpose benchmarks fail to capture the specialized reasoning required for these adversarial and protocol-driven settings. To bridge this gap, we introduce DMind Benchmark , a comprehensive evaluation suite designed to rigorously assess LLM proficiency across the Web3 stack. DMind Benchmark encompasses nine distinct subdomains (spanning infrastructure, smart contracts, token economics, etc.) and combines objective knowledge retrieval with complex open-ended reasoning tasks that emulate real-world operational challenges. We conduct an extensive evaluation of 31 leading proprietary and open-weights models, employing a contamination-aware pipeline and verifying the statistical robustness of our scoring protocol through rigorous cross-judge consistency checks. Our analysis reveals a critical dichotomy: while models demonstrate competence in foundational infrastructure concepts, they exhibit significant vulnerabilities in high-reasoning tasks such as security auditing. Furthermore, we provide a Pareto analysis to guide cost-effective deployment and demonstrate through adversarial experiments that high performance on DMind Benchmark necessitates genuine reasoning rather than superficial memorization. Since its open-source release in April 2025, DMind Benchmark achieved the #1 trending position on Hugging Face for nearly a week and accumulated over 13k downloads by June 2026, establishing itself as a standard for advancing secure and trustworthy AI in Web3. The dataset and evaluation toolkit are publicly available at https://huggingface.co/datasets/DMindAI/DMind_Benchmark . Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 9–13, 2026; Jeju Island, Republic of Korea. Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD 2026), August 9–13, 2026, Jeju Island, Republic of Korea ISBN: 979-8-4007-2259-2/2026/08 DOI: 10.1145/3770855.3817512 Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3817512 ISBN: 979-8-4007-2259-2/2026/08 CCS: Computing methodologies Language resources Figure 1. Overview of the nine key Web3 subdomains covered and evaluated by the DMind Benchmark Overview of the nine key Web3 subdomains evaluated in the DMind Benchmark, showing their interconnections within the broader Web3 ecosystem. † † cc-license: by-nc-nd † † cc-license: by-nc-nd 1. Introduction Large Language Models (LLMs) have evolved from fluent text generators into general-purpose systems that can follow complex instructions and perform multi-step reasoning ( OpenAI et al., 2023 ; DeepSeek-AI et al., 2025 ; Touvron et al., 2023 ; Mesnard et al., 2024 ) . As these models become production-ready, they are increasingly embedded in domain-critical workflows such as biomedical decision support ( Singhal et al., 2023 ) , financial analysis ( Wu et al., 2023 ) , and legal reasoning ( Looijenga, 2024 ) . In these settings, broad language ability is necessary but not sufficient. Practical reliability requires precise domain knowledge, faithful numerical and logical reasoning, and robustness under ambiguity and adversarial inputs. As a result, domain-grounded evaluation has become a bottleneck for responsible deployment. In this paper, we study LLMs in the Web3 domain. Web3 seeks to re-architect the internet around decentralized ownership and trust minimization, replacing centralized intermediaries with cryptographic primitives and distributed consensus ( Buterin, 2013 ; Wood, 2014b ; Chatterjee and Ramamurthy, 2025 ; Geren et al., 2025 ) . Modern Web3 systems span smart contracts, DeFi protocols ( Ozili, 2022 ) , NFTs ( Wang et al., 2021 ) , DAOs ( Bellavitis et al., 2023 ) , on-chain governance, and privacy-enhancing infrastructures, all underpinned by cryptoeconomic incentives and an adversarial security environment. Web3 therefore combines software engineering with mechanism design, and it operates under unusually high stakes. Applications often manage real capital, are difficult to patch once deployed, and face constant attack. Even small reasoning errors can translate into exploitable contracts, misaligned incentives, or governance failures. These properties make Web3 a demanding setting for assessing whether LLMs can be trusted as assistants for developers, auditors, and end users. LLMs are already being integrated into Web3 workflows. Recent studies explore LLM-assisted smart contract development and refactoring ( Nijkamp et al., 2023 ; Nam et al., 2024 ; Zhong and Wang, 2024 ) , vulnerability understanding and audit support ( Wu et al., 2024 ) , and tooling for documentation and user support ( Suri et al., 2024 ; Dearstyne et al., 2024 ) . Beyond code, LLMs are used for blockchain data analytics ( Toyoda et al., 2024 ) , DeFi interaction agents ( Mothukuri et al., 2024 ) , and market-oriented tasks such as cryptocurrency forecasting ( Li et al., 2024 ) . Despite this momentum, progress is constrained by the lack of a shared, domain-grounded evaluation standard. General-purpose benchmarks such as MMLU ( Hendrycks et al., 2021 ) , BIG-Bench ( Kazemi et al., 2025 ) , and HELM ( Liang et al., 2023 ) provide valuable coverage of broad capabilities, but they do not directly test Web3-specific competencies such as EVM-level debugging, multi-step exploit reasoning, or incentive analysis under strategic behavior. In contrast to medicine and finance, where domain benchmarks have become a foundation for systematic progress ( Xie et al., 2024 ; Kim et al., 2024 ) , the Web3 ecosystem has lacked a comprehensive and practical evaluation suite. To address this gap, we introduce the DMind Benchmark ( Huang et al., 2025 ) , a holistic evaluation suite designed to assess LLM proficiency across the Web3 stack. DMind Benchmark is organized into nine domains: (1) blockchain fundamentals, (2) blockchain infrastructure, (3) smart contracts, (4) DeFi mechanisms, (5) DAOs, (6) NFTs, (7) token economics, (8) meme concepts, and (9) security vulnerabilities . We adopt this taxonomy for a simple reason: it mirrors how Web3 systems are built and how they succeed or fail in the real world. One lens follows the engineering path of a typical system, from shared concepts (fundamentals), to the execution environment and scalability constraints (infrastructure), to the programmable layer where behavior is encoded (smart contracts), and then to the dominant application families seen in practice (DeFi, DAOs, NFTs). A second lens captures the forces that shape outcomes even when code appears correct. Token economics formalizes incentives and strategic behavior; meme concepts capture fast-moving narratives that affect adoption, attention, and coordination; security vulnerabilities represent the adversarial pressure that stress tests every layer. Together, these nine domains offer compact coverage with clear diagnostic value, enabling evaluation that is both end-to-end and actionable. The construction of DMind Benchmark employs a rigorous pipeline characterized by strict provenance tracking and contamination awareness. We curate a whitelist of authoritative Web3 sources, preserve metadata for auditability, and apply sanitization, deduplication, and privacy redaction before expert review. Domain specialists convert high-value concepts into evaluation items using paraphrasing and option

Read the complete text