DTM-Codec: Dynamic Token Masking for VFR Speech Coding with Efficient Boundary Selection

Hoyeol, Juhan · 2026

Read DTM-Codec: Dynamic Token Masking for VFR Speech Coding with Efficient Boundary Selection by Hoyeol, Juhan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29480.txt.

Introduction

Sohn and Nam present DTM-Codec, a neural speech codec that changes its frame rate as speech changes. A codec compresses speech into tokens and rebuilds it later. Variable frame rate coding spends fewer frames on steady sounds and more on rapidly changing ones. Earlier gains often shrank once the extra bits describing which time steps were kept were honestly counted.

Dynamic Token Masking keeps selected encoder tokens and fills the rest with a learned mask embedding. A binary keep-mask travels with the data so the decoder knows each token's position. Their Path Length Equalization selector picks boundaries in linear time and spreads segments evenly. Under a strict matched total bitrate protocol, the codec improves on fixed frame rate baselines broadly, in both reconstruction quality and intelligibility. The gains hold across several operating points. Both authors work at KAIST's Graduate School of Cultural Technology in South Korea.

Catalog record

CREATORHoyeol, Juhan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29480

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DTM-Codec: Dynamic Token Masking for VFR Speech Coding with Efficient Boundary Selection Hoyeol, Juhan Abstract Variable frame rate (VFR) coding has recently emerged in neural speech codecs, allocating fewer frames to redundant regions and more frames to rapidly changing speech. VFR must transmit side information about retained time steps, but prior gains are either not rigorously addressed or often minor once these overhead bits are included in total bitrate. We present Dynamic Token Masking (DTM)-Codec, a neural speech codec that demonstrates clear gains over fixed-frame-rate baselines under a strict matched-total-bitrate protocol. DTM keeps selected encoder tokens, fills masked positions with a learned <MASK> embedding, and transmits a binary keep-mask for position-aware decoding. We further introduce Path Length Equalization (PLE), a linear-time boundary selector for VFR coding that yields well-spread adaptive segments with negligible overhead. Across operating points, DTM-Codec broadly improves reconstruction quality and intelligibility over fixed-frame-rate baselines. Sohn Nam keywords speech codec, variable frame rate, token masking, low bitrate, speech tokenization † † address: 1 Graduate School of Cultural Technology, KAIST, South Korea † † email: {hoyso48,juhan.nam}@kaist.ac.kr 1 Introduction Neural audio codecs have become a central tokenization layer for speech language models [ 5 , 46 , 52 ] , text-to-speech synthesis [ 7 ] , and audio generation [ 22 ] . The field has advanced from multi-codebook residual vector quantization (RVQ) systems such as SoundStream and EnCodec toward single-codebook tokenizers and semantic-aware designs [ 53 , 11 , 23 , 51 ] , while recent surveys identify temporal token allocation as a key open challenge beyond codebook design [ 55 , 30 ] . Single-codebook systems such as BigCodec [ 50 ] , WavTokenizer [ 16 ] , and TAAE [ 33 ] simplify language-model integration by producing a flat token sequence, yet they operate at fixed frame rates ( e.g. , 25–100 Hz), assigning uniform temporal resolution regardless of information density. Variable frame rate (VFR) coding addresses this limitation by allocating temporal resolution adaptively: fewer tokens for silence or sustained vowels, more for rapid phonetic transitions. Several recent works explore paradigms for VFR neural codecs. TFC-Codec [ 56 ] augments a standard codec with a plug-in module that selects among predefined temporal granularities per region. VARSTok [ 58 ] fine-tunes a pretrained single-codebook codec to emit variable-length tokens by clustering adjacent frames and encoding duration implicitly within the token index. CodecSlime [ 47 ] applies VFR as a post-training plugin: an optimization-based scheduler determines merge patterns on frozen encoder features, after which the decoder is adapted via two-stage fine-tuning. FlexiCodec [ 25 ] adopts a dual-stream architecture with a frozen ASR encoder alongside a codec encoder, merging frames guided by semantic similarity to reach low frame rates. However, whether VFR yields consistent reconstruction gains at matched total bitrate remains an open question. A fair comparison must account for side-information bits (position or duration) together with content bits. VARSTok, TFC-Codec, and FlexiCodec each report bitrate-matched or reduced-rate settings, yet none provides a controlled same-architecture VFR-vs-FFR comparison in which timing-side-information overhead is explicitly included in the total bitrate [ 58 , 56 , 25 ] . CodecSlime includes a strict matched-rate comparison, but the reported gains are partial and concentrated on a subset of metrics [ 47 ] . In addition, optimization parity is not always controlled: VARSTok adapts a pretrained WavTokenizer, and CodecSlime applies an additional two-stage Melt-and-Cool adaptation on top of a pretrained FFR backbone to obtain its VFR variant [ 58 , 47 ] . This motivates a practical question: can VFR deliver reliable reconstruction improvements at matched total bitrate when side-information overhead is explicitly accounted for? We address this question with DTM-Codec, a 127M-parameter neural speech codec trained on LibriSpeech-960 alone [ 32 ] —modest in both model size and data compared with recent systems that leverage large-scale multi-domain corpora and semantic teacher models [ 50 , 52 , 25 ] . Despite these constraints, DTM-Codec outperforms or matches larger external codecs across multiple reconstruction metrics at matched bitrate, demonstrating that well-designed VFR mechanisms can compensate for scale [ 50 , 52 , 25 ] . Our contributions are as follows: 1. Dynamic Token Masking (DTM). Rather than merging or pooling features, we retain selected encoder tokens and fill missing positions with a learnable <MASK> embedding. The binary mask is transmitted as compact position bits, enabling position-aware decoding. This formulation yields the strongest overall reconstruction among common down/up-sampling alternatives at the same bitrate. 2. Path Length Equalization (PLE). We introduce an O(N) boundary selector that partitions the cumulative feature change along the encoder trajectory into equal-length segments. Compared with representative heuristic and optimization-based selectors—including those used by VARSTok, FlexiCodec, and CodecSlime—PLE achieves well-spread temporal coverage with negligible runtime overhead and a favorable quality–efficiency trade-off. 3. Strict matched-total-bitrate training and evaluation. We train and evaluate matched-rate VFR and FFR under a unified protocol that counts both content and timing-side-information bits across low-to-high frame-rate operating points. This controlled setup enables direct and fair comparisons and reveals clear and broadly VFR advantages over fixed-frame-rate baselines at low-to-mid bitrates. https://github.com/hoyso48/DTM-Codec . 2 Related Work 2.1 Neural Audio Codecs Neural audio codecs (NACs) typically follow the VQ-VAE/VQ-GAN paradigm [ 44 ] , learning an encoder–decoder with a discrete bottleneck [ 53 , 11 ] . SoundStream and EnCodec use residual vector quantization (RVQ) with multiple codebooks to scale bitrate [ 53 , 11 ] . Recent work pushes toward lower bitrates and LM-friendly token interfaces: TAAE uses a transformer autoencoder with an FSQ bottleneck and reports 400–700 bps operating points [ 33 , 29 ] ; BigCodec and WavTokenizer use single-codebook VQ tokenizers [ 50 , 16 ] . Multi-resolution designs such as SNAC assign different frame rates to different quantizer layers [ 40 ] . X-Codec [ 51 ] and X-Codec 2.0 [ 52 ] incorporate semantic feature distillation, while SpeechTokenizer [ 57 ] disentangles semantic and acoustic information across RVQ layers. DAC [ 23 ] improves codebook utilization within the RVQGAN framework. We build on the TAAE two-stage transformer template but use single-codebook VQ, an STFT/iSTFT front-end/back-end, and dynamic token masking for VFR coding (Section 3.1 ). 2.2 Variable Frame Rate Coding in Speech Codecs Variable-rate discrete representation has roots in learned compression of images and audio [ 12 ] ; variable-rate hierarchical CPC has demonstrated acoustic unit discovery in speech [ 9 ] , and recent SSL work shows that self-distillation induces syllable-level temporal structure [ 8 ] , providing linguistic motivation for adaptive-rate tokenization. Recent VFR speech codecs mainly differ along three axes: boundary selection, timing-side-information design, and training strategy. TFC [ 56 ] computes signal-level Shannon entropy over amplitude histograms to route each temporal window to one of three predefined granularities and fuses multi-resolution quantized tokens back to the finest timeline via granularity masks; the reported bitrate accounts for content bits only, without explicitly quantifying the mask overhead. VARSTok [ 58 ] applies density-peak clustering on encoder features to produce content-adaptive segments with non-uniform boundaries, and folds duration into an extended token index \mathrm{ID}=(d{-}1)\times K+k , enabling standard next-token prediction over an expanded vocabulary without a separate duration channel. CodecSlime [ 47 ] formulates boundary selection as a global L2-distortion minimization via dynamic programming and

Read the complete text