Deep-Bench: Deep Learning Benchmark Dataset for Code Generation
Alireza Daghighfarsoodeh, Chung-Yu Wang, Hamed Taherkhani, Melika Sepidband, Mohammad Abdollahi, Hadi Hemmati, Hung Viet Pham · 2025
Read Deep-Bench: Deep Learning Benchmark Dataset for Code Generation by Alireza Daghighfarsoodeh, Chung-Yu Wang, Hamed Taherkhani, Melika Sepidband, Mohammad Abdollahi, Hadi Hemmati, Hung Viet Pham (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18726.txt.
Introduction
Alireza Daghighfarsoodeh and co-authors set a harder code-writing test for AI in deep learning. Their 2025 paper notes that old benchmarks use small pre- and post-processing snippets. Real deep learning work spans data prep, model building, and training, across tables, images, and text. Their set, DeepBench, grades function-level code across all those cuts.
Top models stumbled: the best scored thirty-one percent, half its rate on the older test. Rivals trailed in step. Scores also swung by task type far more than by phase. A bug taxonomy from the failures shows where model code breaks. The authors offer DeepBench as a map of what AI still cannot write.
Catalog record
| CREATOR | Alireza Daghighfarsoodeh, Chung-Yu Wang, Hamed Taherkhani, Melika Sepidband, Mohammad Abdollahi, Hadi Hemmati, Hung Viet Pham |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-18726
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deep-Bench: Deep Learning Benchmark Dataset for Code Generation Alireza Daghighfarsoodeh, Chung-Yu Wang, Hamed Taherkhani, Melika Sepidband, Mohammad Abdollahi, Hadi Hemmati, Hung Viet Pham Abstract Deep learning (DL) has revolutionized areas such as computer vision, natural language processing, and more. However, developing DL systems is challenging due to the complexity of DL workflows. Large Language Models (LLMs), such as GPT, Claude, Llama, Mistral, etc., have emerged as promising tools to assist in DL code generation, offering potential solutions to these challenges. Despite this, existing benchmarks such as DS-1000 are limited, as they primarily focus on small DL code snippets related to pre/post-processing tasks and lack a comprehensive coverage of the full DL pipeline, including different DL phases and input data types. To address this, we introduce Deep-Bench, a novel benchmark dataset designed for function-level DL code generation. Deep-Bench categorizes DL problems based on three key aspects: phases such as pre-processing, model construction, and training; tasks, including classification, regression, and recommendation; and input data types such as tabular, image, and text. GPT-4o—the state-of-the-art LLM—achieved 31% accuracy on Deep-Bench, significantly lower than its 60% on DS-1000. We observed similar difficulty for other LLMs (e.g., 28% vs. 54% for Claude, 21% vs. 41% for LLaMA, and 15% vs. 20% for Mistral). This result underscores Deep-Bench’s greater complexity. We also construct a taxonomy of issues and bugs found in LLM-generated DL code, which highlights the distinct challenges that LLMs face when generating DL code compared to general code. Furthermore, our analysis also reveals substantial performance variations across categories, with differences of up to 7% among phases and 37% among tasks. These disparities suggest that Deep-Bench offers valuable insights into the LLMs’ performance and areas for potential improvement in the DL domain. 1 Introduction In recent years, machine learning (ML) and deep learning (DL) have advanced significantly and have been integrated into various fields ( Hordri et al., 2016 ; Kamilaris & Prenafeta-Boldú, 2018 ; Gamboa, 2017 ) . DL coding has its challenges ( Arpteg et al., 2018 ) , and because of its widespread use, many DL systems are developed by domain experts who are often not software developers ( Park et al., 2021 ; Singaravel et al., 2020 ) , which amplifies the problems even more. Recently, with the rise of Large Language Models (LLMs) such as ChatGPT, LLMs are considered among the best solutions for coding tasks ( Wang et al., 2021 ; Feng et al., 2020 ; Achiam et al., 2023 ) . As shown in Table 1 , numerous code generation datasets ( Hendrycks et al., ; Austin et al., 2021 ; Agashe et al., 2019 ; Lu et al., 2021 ; Yu et al., 2018 ; Du et al., 2024 ; Zhuo et al., 2024 ) benchmark LLMs’ code generation capabilities. However, DS-1000 ( Lai et al., 2023 ) is the only dataset offering a limited set of DL-specific code generation samples. Specifically, DS-1000 provides small code snippets, typically just a few lines, primarily focused on pre/post-processing tasks. Furthermore, they do not include categorizations such as DL phases (e.g., pre/post-processing, model construction, training, inference, evaluation) or input types (e.g., tabular, image, text), which could provide valuable insights for advancing DL code generation research. To address this gap, we introduce Deep-Bench, a novel dataset designed to benchmark DL code generation at a functional level. Deep-Bench includes the code generation prompt, the ground-truth code at the function level, an extensive set of unit tests, and three categorizations (DL phases, ML tasks, and input data types). Unlike DS-1000, Deep-Bench provides a more comprehensive set of entries that cover the full DL pipeline, encompassing code examples for various machine learning tasks. Additionally, it includes DL functions with diverse input data types, including tabular, image, and text, making it a more diverse benchmark for DL code generation evaluation. Specifically, each entry of Deep-Bench dataset is categorized based on these three categories which allow a more in-depth evaluation of future DL code generation techniques: (1) The DL/ML pipeline stages (consist of pre/post-processing, model construction, training, inference, and evaluation), (2) The DL/ML tasks (consist of classification, object detection, image segmentation, time-series prediction, recommendation, and regression), and (3) The input data types (consist of text, image, and array). Our study found that while GPT-4o demonstrated strong performance in various code generation tasks, it achieved only 31% accuracy on Deep-Bench, significantly lower than the 60% reported for DS-1000. We observed similar difficulty for other LLMs (e.g., 28% vs. 54% for Claude, 21% vs. 41% for LLaMA, and 15% vs. 20% for Mistral) which indicates that Deep-Bench contains more challenging data points. Furthermore, the difficulty of generating code varies significantly across categories. For example, GPT-4o reaches an accuracy of 33% for pre/post-processing tasks but only 26% for model construction. Additionally, the accuracies vary even more among task types, ranging from 58% for recommendation tasks to 21% for segmentation tasks. These large gaps in performance across categories highlight the insights that Deep-Bench can bring to help improve the LLM DL code generation capability. To further analyze the root causes of such a poor generation performance, we construct a bug taxonomy of the issues found in the generated DL code. Our qualitative analysis reveals that when compared to LLM-generated general code, LLM-generated DL code exhibits a higher frequency of deviation from the prompt issues. Furthermore, arithmetic and logical errors emerged as a new and frequently observed category in generated DL code. Deep-Bench’s data is available in our public repository 1 1 1 https://anonymous.4open.science/r/DL-Bench-D65E/ . Table 1: Code generation benchmarks (SO-Stack Overflow, EM-exact match, ES-edit similarity, CS-competition specifics). Benchmark Size Language Level Source Metrics Human Eval ( Chen et al., 2021 ) 164 Python Function Manual Pass@k MBPP ( Austin et al., 2021 ) 974 Python Function Manual Pass@k APPS ( Hendrycks et al., ) 10000 Python Function Manual Pass@k CoderEval ( Yu et al., 2024 ) 460 Python-Java Function GitHub Pass@k RepoEval ( Zhang et al., 2023 ) 1600 Python Function GitHub EM,ES DS-1000 ( Lai et al., 2023 ) 1000 Python Statement SO Pass@k MLE-Bench ( Chan et al., 2024 ) 75 Python Workflow Kaggle CS Deep-Bench 520 Python Function GH Pass@k 2 Related Works Code Generation for ML software : Shin et. al. ( Shin et al., 2023 ) have explored the effectiveness of neural code generation models on ML tasks, which differ significantly from general programming tasks. A key study evaluated six state-of-the-art models across popular ML libraries, highlighting the need for domain-specific benchmarks and improvements tailored to the complexities of ML. Our work provides such benchmarks specifically for ML and DL software. Recently, DS-1000 ( Lai et al., 2023 ) is the only benchmark designed for data science code generation. It contains 1000 problems sourced from StackOverflow with diverse, practical tasks. Similarly, MLE-bench ( Chan et al., 2024 ) targets evaluating ML engineering workflows in Kaggle competitions. However, Deep-Bench differs in key aspects: 1) it focuses on ML/DL tasks rather than general data science and MLE, 2) we categorize the data by ML phases, task types, and data types, and 3) our granularity is at the function level rather than script or workflow level. Additionally, unlike the other datasets, our dataset is based on GitHub repositories with