Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis
Michael J. Bommarito II · 2025
Read Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis by Michael J. Bommarito II (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17573.txt.
Introduction
Michael J. Bommarito builds tokenizers for a neglected input type: raw binary files. Language models read text through tokenizers trained on prose, which choke on arbitrary byte sequences and waste scarce context window on single bytes. Bommarito trains a family of byte-pair encoding tokenizers on a large corpus of executables spanning Linux, Windows, macOS, Android, and malware, across several architectures and operating systems. He releases them openly in five vocabulary sizes.
The learned tokens turn out interpretable, capturing file headers, instruction sequences, and cross-platform strings, and they pack several bytes per token. On typical uncompressed executables, the tokenizers fit roughly two to three times more binary content into a fixed context window than raw bytes. The release aims to serve as a drop-in foundation for binary-focused models used in content identification, malware detection, and reverse engineering.
Catalog record
| CREATOR | Michael J. Bommarito II |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-17573
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis Michael J. Bommarito II Abstract Sequence models for binary analysis are bottlenecked by byte-level tokenization: raw bytes waste precious context window capacity for transformers and other neural network architectures, and many existing text-oriented tokenizers fail on arbitrary 0x00–0xFF sequences. To address this issue, we introduce the Binary BPE family, a set of cross-platform Byte Pair Encoding (BPE) tokenizers for executables trained on a large corpus of binaries spanning multiple platforms, architectures, and operating systems, including Linux, Windows, macOS, Android, and malware sources. We release trained tokenizers with vocabularies of 4K, 8K, 16K, 32K, and 64K tokens, enabling both systematic scaling studies and practical deployment from resource-constrained edge devices to high-throughput datacenters. The family exhibits a nested hierarchy where each smaller vocabulary is a perfect prefix of larger ones, enabling seamless transfer of learned embeddings when scaling model capacity. These tokenizers discover interpretable patterns (ELF/PE headers, instruction sequences, cross-platform strings) while yielding multi-byte compression per token. On representative uncompressed executables (e.g., ELF/PE/Mach-O rather than compressed APKs), the Binary BPE family typically allows for roughly 2–3 × \times more binary content per fixed-length transformer context window than raw bytes, enabling more efficient research and practical deployment for content identification, malware detection, reverse engineering, and optimization. We release the trained Binary BPE tokenizers on HuggingFace and supporting code on GitHub, providing a drop-in, open-source foundation for binary-focused language models and context-efficient agentic tools. 1 Introduction Sequence models—including recurrent neural networks (RNNs), convolutional neural networks (CNNs), structured state-space models (SSMs), and most especially transformers—require tokenization to convert input data into discrete sequences. Existing tokenizers like BPE [ 19 ] , WordPiece [ 18 ] , and SentencePiece [ 11 ] are designed for natural language and source code, typically learning subword units that respect linguistic structure, word boundaries, and UTF-8 text properties. Binary executables, however, present a fundamentally different domain: raw byte sequences (0x00–0xFF) containing machine code instructions, file format headers, compressed resources, and encrypted sections with no linguistic regularities, word boundaries, or printable characters. Applying NLP tokenizers to binaries fails catastrophically, as they are optimized for human-readable text, not arbitrary byte distributions. Processing binaries as raw byte sequences also wastes tokens: a 100KB binary encoded byte by byte produces 100K tokens, quickly exhausting typical accelerator memory, especially when malware and production binaries are often orders of magnitude larger. Developing effective binary tokenization faces several unique challenges. First, platform diversity : Linux (ELF), Windows (PE), macOS (Mach-O), and Android (APK) employ distinct file format conventions with different header structures and section layouts. Second, architectural heterogeneity : x86-64, ARM, MIPS, and RISC-V instruction sets encode operations differently, creating ISA-specific byte patterns that must be learned jointly for cross-platform utility. Third, extreme length variation : binaries range from 10KB utilities to multi-megabyte applications, with sections (code, data, resources) exhibiting dramatically different compression characteristics. Finally, domain variability : system utilities, GUI applications, and malware samples contain distinct byte-level distributions. A successful tokenizer must learn patterns that generalize across these dimensions while preserving format- and ISA-specific structure. Binary analysis underpins tasks such as malware detection and classification, vulnerability discovery in proprietary software, and program understanding or reverse engineering of stripped or obfuscated executables. Recent advances demonstrate that multiple hybrid or transformer-based architectures with scaled attention can learn rich binary representations for these tasks, but their effectiveness is fundamentally constrained by tokenization. While tokenization critically impacts transformer performance on code [ 15 ] , existing approaches operate on disassembled text, limiting applicability to stripped or obfuscated binaries. Android-specific methods [ 17 ] apply BPE to decompiled manifest files but require high-level symbolic information. Raw byte transformers [ 13 ] avoid disassembly but lack learned vocabulary compression. The challenge remains: can one develop a byte-level BPE tokenizer that learns cross-platform patterns from raw binaries without disassembly or decompilation? Motivated by these opportunities, we present a family of cross-platform BPE tokenizers for raw binary executables—infrastructure enabling transformer-based and more broadly sequence-model-based binary analysis. Trained on 24 GB of data across 30,000 unique binaries from the Binary-30K dataset [ 6 ] , the family achieves roughly 2–3 \times effective compression on typical uncompressed binaries (ELF/PE/Mach-O), while compressed, high-entropy formats such as Android APKs naturally see smaller but still meaningful gains. Beyond compression, BPE tokenization also performs unsupervised pattern discovery that pre-encodes domain structure before transformer training begins. By learning frequent byte co-occurrences, BPE discovers and consolidates recurring patterns—x86-64 instruction prefixes (REX.W), file format headers (ELF magic), cross-platform string fragments—into single vocabulary tokens. This gives transformers meaningful semantic units rather than forcing models to rediscover "these bytes always co-occur" from scratch during training. For binary analysis, where ISA-specific encodings and format conventions create predictable byte-level structure, BPE effectively performs feature learning at the byte level, enabling more efficient downstream representation learning. The vocabulary family serves dual purposes. Scientifically , it enables controlled ablation studies of BPE scaling behavior on binary data, quantifying compression-vocabulary tradeoffs, sequence coverage dynamics, and learned pattern evolution across architectures—questions otherwise unanswerable with a single model. Practically , different deployment scenarios demand different resource-computation tradeoffs. Smaller vocabularies (4K, 8K) enable deployment in resource-constrained environments like network security appliances performing inline packet inspection, edge devices with limited memory footprints, or embedded malware scanners in gateways. Where tokenizer size and inference latency are critical constraints, these smaller vocabulary sizes provide an ideal footprint. Larger vocabularies (32K, 64K) maximize compression efficiency for datacenter-scale analysis, where accelerator memory and throughput dominate. This range supports both research and pedagogy on tokenization fundamentals, as well as real-world deployment from edge to cloud. Critically, the family exhibits a nested hierarchy: each smaller vocabulary is a perfect prefix of all larger ones, with identical token IDs mapping to identical byte sequences. This property enables seamless model scaling—practitioners can train a 1B-parameter model with 4K embeddings, then expand to a 7B-parameter model with 64K embeddings while preserving all originally learned representations, avoiding costly retraining of the base vocabulary. Beyond serving as a drop-in tokenizer for larger general-purpose models, the Binary BPE family is designed to support smaller specialist masked- and causal-language models that might act as tools inside larger analysis workflows; for example, plug-ins embedded in reverse-engineering frameworks, such as Ghidra or IDA Pro, or domain-specific binary “experts” orchestrated by agentic systems or human analysts. In this paper, we contribute: (1) a training methodology for BPE on diverse binaries (ELF, PE, Mach-O, APK) across x86-64, ARM, MIPS, and RISC-V architectures, with platform stratification to ensure cross-platform pattern learning; (2) an ablation study of vocabulary size (4K, 8K, 16K, 32K, 64K) quantifying compression, coverage, and token-length shifts; (3) analysis of learned patterns demonstrating instruction- and format-aware tokens across sizes; and (4) public release of the Binary BPE family on HuggingFace, together with replication for our publication experiments, figures, and tables. 2 Related Work 2.1 Tokenization in Natural Language Processing Subword tokenization through BPE [ 19 ] , WordPiece [ 18 ] , and SentencePiece [ 11 ] is standard for neural language models. These methods typically assume UTF-8 text with linguistic structure, integrate pre- and post-processing for word boundaries and character composition rules, and feature vocabulary sizes on the order of tens to hundreds of thousands of tokens to cover