Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model

1 st Thang Dang, 2 nd Akira Nakagawa, 3 rd Kenichi Kobayashi, 4 th Koichi Shirahata · 2026

Read Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model by 1 st Thang Dang, 2 nd Akira Nakagawa, 3 rd Kenichi Kobayashi, 4 th Koichi Shirahata (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30080.txt.

Introduction

Thang Dang, Akira Nakagawa, Kenichi Kobayashi and Koichi Shirahata propose Adaptive Targeted Dynamic Chunking for tokenization-free models that read raw bytes, not preset vocabularies. Byte models dodge vocabulary design and unknown-word errors, but their quality hinges on chunk compression ratios. Their method raises compression gradually during training through curriculum learning, and links the target ratio to bytes per innermost chunk. On FineWeb-Edu 100B, chunked models match byte- and token-level baselines on bits per byte, with steadier training and stronger downstream scores.

Catalog record

CREATOR1 st Thang Dang, 2 nd Akira Nakagawa, 3 rd Kenichi Kobayashi, 4 th Koichi Shirahata
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35984
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30080

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model 1 st Thang Dang, 2 nd Akira Nakagawa, 3 rd Kenichi Kobayashi, 4 th Koichi Shirahata Abstract Tokenization-free hierarchical models are emerging as a promising alternative to traditional Large Language Models (LLMs), addressing inherent preprocessing issues such as vocabulary design complexity, out-of-vocabulary (OOV) errors, and language-specific constraints. However, a significant challenge in these byte-level methods is the optimization of the compression ratio, a critical factor that dictates model performance for processing bytes data via chunks. In this paper, we propose Adaptive Targeted Dynamic Chunking (ATDC), a novel byte-compression control mechanism designed to enhance the effectiveness of dynamic chunking within hierarchical architectures. Our approach utilizes curriculum learning to progressively adjust the compression ratio during training, transitioning from low to high compression to stabilize the learning process. We provide an analysis establishing the relationship between the target compression ratio and Bytes-Per-Innermost-Chunk (BPIC), allowing for tracking of chunk-size evolution throughout the training phase. Evaluations conducted on the FineWeb-Edu 100B dataset demonstrate that hierarchical models equipped with ATDC achieve competitive Bits-Per-Byte (BPB) performance compared to conventional baselines operating at both byte and token levels. Furthermore, the proposed method exhibits more stable training dynamics and superior final performance across diverse downstream tasks compared to models using fixed compression ratios, while maintaining the inherent robustness and flexibility of byte-level processing. I Introduction The development of Large Language Models (LLMs) [ 3 , 10 ] has traditionally depended on subword tokenization techniques, including Byte-Pair Encoding (BPE) [ 11 , 19 ] , WordPiece [ 21 ] , and hybrid Byte-Word approaches [ 15 ] applied during preprocessing. Although these methods have proven effective, they impose notable limitations. Tokenization demands intricate vocabulary construction, which frequently results in out-of-vocabulary (OOV) problems and reduced resilience to typographical errors or morphological diversity [ 1 ] . Moreover, tokenizers exhibit pronounced bias toward high-resource languages [ 18 ] , exacerbating performance disparities in multilingual contexts. These intrinsic drawbacks, often termed the “overheads” of tokenization, have driven interest in tokenization-free architectures [ 16 ] that process raw bytes directly. While byte-level models hold considerable potential, they encounter a core efficiency hurdle: individual byte or meaningful chunks span numerous bytes, necessitating substantial compression to handle extended contexts and control computational overhead effectively. Hierarchical designs, such as H-Net [ 14 ] , mitigate this through mechanisms like dynamic chunking [ 8 ] or patching [ 16 ] to form higher-level abstractions from byte sequences. Nevertheless, determining and regulating the compression ratio in these systems remains predominantly heuristic and suboptimal. Existing approaches typically employ static compression rates (for example, compression rate N is equal to 6 indicates that average of 6 bytes are expected to be compressed into a single chunk) that fail to adapt to evolving data complexity throughout training, often causing unstable optimization or suboptimal dynamic compression. In this work, we address these shortcomings by proposing Adaptive Targeted Dynamic Chunking (ATDC). Our approach is motivated by the insight that a model’s compression capability should adapt in tandem with its growing comprehension of the data distribution. We conceptualize compression management as a form of curriculum learning, wherein the method gradually modulates the compression ratio and targeted chunk sizes. This enables a smooth progression from low-compression (fine-grained) processing to high-compression (coarse-grained) regimes as training advances. The primary contributions of this work are as follows: • A Novel Compression Control Method: We propose ATDC, which dynamically regulates byte compression in hierarchical architectures. • Indicator Framework: We provide a formal analysis establishing the relationship between a target compression ratio and the Bytes-Per-Innermost-Chunk (BPIC). • Empirical Validation: Through extensive training on 100B tokens of the FineWeb-Edu [ 17 ] dataset, we demonstrate that byte-level models using ATDC achieve bits-per-byte performance competitive with both byte-level H-Net [ 8 ] and token-level Llama3.2 [ 5 ] -based models. • Textual Robustness: We demonstrate that our adaptive approach leads to increased robustness against HellaSwag textual perturbations across a variety of tasks compared to models with a fixed compression rate. II Related works Tokenization has become a significant limitation in modern Large Language Models, particularly due to its sensitivity to spelling variations, typos, capitalization, and morphological changes [ 23 ] . Byte-level modeling, which directly processes raw bytes, offers a promising alternative by eliminating many of these preprocessing assumptions. MegaByte [ 24 ] introduced a multiscale decoder capable of modeling extremely long byte sequences (over 1 million bytes). However, its reliance on fixed-size patches and simple byte embedding concatenation limits flexibility. MambaByte [ 22 ] advanced this direction by training a Mamba-based [ 6 ] model directly on raw bytes, leveraging its efficient fixed-size state. Still, compressing all contextual information into a single fixed hidden state remains a potential bottleneck. More recently, the Byte Latent Transformer (BLT) [ 16 ] proposed a dynamic patching strategy that groups bytes according to next-byte prediction entropy. This approach adaptively assigns more computation to complex regions while using less for predictable sequences. However, BLT’s entropy-based method can suffer from instability or “drift,” especially on highly repetitive text such as in multiple-choice reasoning tasks. Dynamic Chunking [ 8 ] utilizes an end-to-end routing mechanism and exponential moving averages to learn chunk boundaries differentiably. However, its reliance on a fixed compression ratio introduces sensitive, manually-tuned hyperparameters. While H-Net++ [ 25 ] attempts to improve this via a lightweight Transformer context-mixer, its evaluation on limited data (1.4B tokens) and smaller model scales prevents a comprehensive performance comparison. Despite these advancements, Dynamic Chunking is limited by its use of a fixed compression rate for chunk size control. Such a mechanism results in highly empirical hyperparameters that often demand manual tuning. Our work is motivated by the need to move beyond these manual constraints, a goal we achieve through our proposed ATDC method. III Hierarchical Models H-Net [ 8 ] is a tokenizer-free hierarchical language model that directly processes raw byte sequences ( \mathcal{V}vocab=256 ). The primary innovation is Dynamic Chunking (DC), a mechanism that identifies semantic boundaries in byte sequences to facilitate efficient hierarchical processing. The general workflow is: \text{Input Bytes}\to\text{Encoder}\to\text{Routing Module}\to\text{Chunking}\to\text{Main Network}\to\text{Dechunking}\to\text{Decoder}\to\text{Output} . III-A Hierarchical Decomposition Given an input sequence \mathbf{x}=(x_{1},x_{2},\dots,x_{L}) of length L , H-Net constructs a hierarchical representation through recursive compression. At any stage s , the process is defined by: • Boundary Prediction: A routing module predicts discrete decisions \mathbf{b}^{(s)}\in\{0,1\}^{L_{s}} and continuous probabilities \mathbf{p}^{(s)}\in[0,1]^{L_{s}} from hidden states \mathbf{h}^{(s)}\in\mathbb{R}^{L_{s}\times d_{s}} . • Compression: The sequence length for the subsequent stage is determined by the number of boundaries, M_{s+1}=\|\mathbf{b}^{(s)}\|_{1} . For a K -stage hierarchy with downsampling factors \{n_{1},n_{2},\dots,n_{K}\} , the total compression ratio is given by \prod_{i=1}^{K}n_{i} . For example, a two-stage model with N_{1}=N_{2}=3 achieves approximately 9\times compression. III-B Boundary Detection via Semantic Discontinuity The routing module identifies boundaries by quantifying the semantic shift between consecutive hidden states using a learned cosine similarity: \sigma_{t}=\frac{\langle Q(\mathbf{h}_{t-1}),K(\mathbf{h}_{t})\rangle}{\|Q(\mathbf{h}_{t-1})\|\cdot\|K(\mathbf{h}_{t})\|} (1) where Q and K are learnable linear projections initialized as identity matrices. The boundary probability p_{t} and discrete decision b_{t} are then derived as: p_{t}=\text{clip}\left(\frac{1-\sigma_{t}}{2},0,1\right),\quad b_{t}=\text{argmax}(1-p_{t},p_{t}) (2) High similarity ( \sigma_{t}\approx 1 ) signals semantic continuity ( p_{t}\approx 0 ), while low similarity ( \sigma_{t}\approx-1 ) triggers a boundary ( p_{t}\approx 1 ). This formulation learns a data-dependent metric space where cosine distance

Read the complete text