SBAN: A Framework & Multi-Dimensional Dataset for Large Language Model Pre-Training and Software Code Mining
Hamed Jelodar, Mohammad Meymani, Samita Bai, and Roozbeh Razavi-Far, Ali A. Ghorbani · 2025
Read SBAN: A Framework & Multi-Dimensional Dataset for Large Language Model Pre-Training and Software Code Mining by Hamed Jelodar, Mohammad Meymani, Samita Bai, and Roozbeh Razavi-Far, Ali A. Ghorbani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18936.txt.
Introduction
Hamed Jelodar and four co-authors introduce SBAN, a large dataset that links source code, binary code, assembly, and plain language descriptions of software. Each sample carries all four views, which lets models learn how the same program looks at very different levels.
With more than three million samples, split between benign programs and malware, SBAN supports training and testing of large language models for code analysis. Uses range from malware detection and code translation to code explanation and other software mining tasks. Its layered design suits transformer models and other deep systems that learn across data types.
Catalog record
| CREATOR | Hamed Jelodar, Mohammad Meymani, Samita Bai, and Roozbeh Razavi-Far, Ali A. Ghorbani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18936
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SBAN: A Framework & Multi-Dimensional Dataset for Large Language Model Pre-Training and Software Code Mining * SBAN: LLMs and Code Mining Thanks: Hamed Jelodar, Mohammad Meymani, Samita Bai, and Roozbeh Razavi-Far, Ali A. Ghorbani Abstract This paper introduces SBAN (Source code, Binary, Assembly, and Natural Language Description), a large-scale, multi-dimensional dataset designed to advance the pre-training and evaluation of large language models (LLMs) for software code analysis. SBAN comprises more than 3 million samples, including 2.9 million benign and 672,000 malware respectively, each represented across four complementary layers: binary code, assembly instructions, natural language descriptions, and source code. This unique multimodal structure enables research on cross-representation learning, semantic understanding of software, and automated malware detection. Beyond security applications, SBAN supports broader tasks such as code translation, code explanation, and other software mining tasks involving heterogeneous data. It is particularly suited for scalable training of deep models, including transformers and other LLM architectures. By bridging low-level machine representations and high-level human semantics, SBAN provides a robust foundation for building intelligent systems that reason about code. We believe that this dataset opens new opportunities for mining software behavior, improving security analytics, and enhancing LLM capabilities in pre-training and fine-tuning tasks for software code mining. I Introduction The rapid growth of software development and the increasing complexity of modern applications have heightened the importance of effective software code mining [ 1 ] . Mining software, including source code, binaries, and their natural language descriptions, is crucial for numerous tasks such as malware detection, program analysis, and automated code summarization [ 2 ] . However, constructing large-scale, high-quality datasets that capture the multifaceted nature of software remains a significant challenge. Existing datasets often focus on a single representation layer, limiting the scope of cross-modal learning and comprehensive analysis [ 3 ] . This fragmentation hinders the development of advanced machine learning models, particularly LLMs, which benefit from diverse and aligned data sources for pre-training and fine-tuning. Fig. 1: Relation between different sources in the SBAN dataset. Previous efforts have provided valuable resources for specific software representations, such as source code repositories or binary malware datasets [ 5 ] , but rarely integrate multiple software layers within a single benchmark [ 4 ] . This gap restricts the potential to leverage rich semantic relationships across different code modalities and hinders progress in unified code understanding and analysis. To address these challenges, we present SBAN (Source, Binary, Assembly, and Natural Language Description), a multi-dimensional dataset comprising over 3.3 million real-world software samples that encompass four distinct but interconnected representation layers: 1) source code, 2) binary, 3) assembly representations, and 4) natural language descriptions, as showed in the Figure 1. To the best of our knowledge, SBAN is the first dataset to integrate these layers into a unified benchmark, enabling rich cross-modal code mining and analysis. It is specifically designed to support the pre-training and evaluation of large language models for diverse software tasks, including code understanding, malware detection, and program analysis. Research Contributions Comprehensive software analysis requires datasets that integrate multiple code representations to enable effective cross-modal learning. SBAN addresses this need by providing a large-scale, four-dimensional dataset to advance LLM-based software mining and security research. This work makes the following key contributions: • We introduce SBAN , a large-scale, multi-dimensional dataset with over 3 million samples, covering binary, assembly, source code, and natural language descriptions of both benign and malware software. • To the best of our knowledge, SBAN is the first dataset to integrate these four distinct software representation layers in a unified benchmark, enabling rich cross-modal code analysis. • SBAN is specifically designed to support pre-training and evaluation of large language models (LLMs) for tasks such as code understanding, malware detection, and program analysis. • The dataset supports diverse data mining applications, including malware classification, code summarization, anomaly detection, and deep semantic code mining. It is available at https://www.unb.ca/cic/datasets/sban-dataset-2025.html . II Related work Due to the increasing sophistication of malware attacks and the growing risk of malware generation through LLMs, there is a clear need for research focused on the development of malware datasets for pre-training and fine-tuning LLMs. Various researchers have proposed approaches for utilizing diverse malware datasets to train and adapt LLMs for cybersecurity tasks. Recently, Jiang et al. [ 10 ] introduced Nova, a generative LLM tailored for assembly code analysis, addressing challenges such as low information density and diverse compiler optimizations. The authors employ extensive open-source C/C++ codebases, including “The Stack” and “AnghaBench.” By compiling the code at various optimization levels (O0 to O3), they generated approximately 4.3 million assembly functions. Nova employs a hierarchical attention architecture that captures semantics across tokens, instructions, and entire functions. To ensure robustness, it leverages a contrastive learning objective that encourages representations to remain consistent across different compiler optimizations. Following pre-training on this large corpus, Nova was fine-tuned for downstream tasks such as binary similarity detection and binary decompilation, aiming to recover high-level source code from assembly. In a related study, Li et al. [ 11 ] aimed to learn general-purpose embeddings for binary assembly instructions through pre-training for language models. They created a large, unlabeled corpus by compiling 3,266 benign software programs, including multiple versions of GNU Binutils, Coreutils, Diffutils, and Findutils, using various compiler optimization levels. This process generated approximately 2.25 billion assembly instructions. On this data, they pretrained a BERT-style model called PalmTree using self-supervised objectives such as masked token prediction and next-instruction prediction. PalmTree effectively captures the structural nuances of assembly language, including opcodes, operands, and addressing modes, outperforming prior embedding approaches. The study demonstrates that pre-training on binary code corpora can enhance downstream tools for malware detection and vulnerability analysis. Harang and Rudd [ 12 ] introduced SOREL-20M, a large-scale benchmark dataset designed for machine learning applications in malware analysis. It contains 20 million Windows PE files equally split between malware and benign samples, each annotated with extensive metadata and pre-extracted static features such as header information, imported functions, strings, and byte histograms. The dataset includes high-confidence labels derived from consensus among multiple antivirus engines. Notably, it also offers approximately 10 million “disarmed” malware binary files that retain executable structure but have been stripped of harmful functionality, making them safe for research and model training. Although SOREL-20M is not a traditional code pre-training corpus, it is widely used to develop and benchmark deep learning models for malware detection at scale. The authors also provide baseline models (e.g., neural networks and gradient-boosted trees) and open-source tooling to facilitate community experimentation and advancement. Demirci and Acarturk [ 13 ] framed the disassembled malware assembly code as a language corpus suitable for static analysis. They extracted instructions from the .text sections of approximately 822 malicious and 672 benign Windows PE files. The GPT-2 model was pretrained on the unlabeled assembly instructions to capture the underlying ”language” of malware. This model was then integrated with a stacked BiLSTM network for classification. Their method achieved strong performance in distinguishing between benign and malicious code by effectively modeling the syntactic and semantic relationships among instructions. The study highlights the potential of applying NLP techniques such as transformers and language modeling to malware detection, particularly in learning informative instruction embeddings that can handle challenges like