Principled Coarse-Grained Acceptance for Speculative Decoding in Speech

Moran Yanuka, Paul Dixon, Eyal Finkelshtein, Daniel Rotman, Raja Giryes · 2025

Read Principled Coarse-Grained Acceptance for Speculative Decoding in Speech by Moran Yanuka, Paul Dixon, Eyal Finkelshtein, Daniel Rotman, Raja Giryes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13732.txt.

Introduction

Moran Yanuka and 4 co-authors speed up speech generation with speculative decoding that accepts near-miss tokens. A small draft model proposes acoustic tokens and a large model verifies them, but exact matching rejects many tokens that sound the same. That strictness wastes most of the possible speedup for speech language models.

Their Principled Coarse-Graining verifies proposals at the level of acoustic similarity groups drawn from the target model's embedding space. Probability mass is split across the overlapping groups a token belongs to, and sampling runs on the group variable with an exactness promise at that level. On LibriTTS, acceptance and throughput rose over standard and speech-specific rivals while intelligibility and voice likeness held.

Catalog record

CREATORMoran Yanuka, Paul Dixon, Eyal Finkelshtein, Daniel Rotman, Raja Giryes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13732

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Principled Coarse-Grained Acceptance for Speculative Decoding in Speech Moran Yanuka, Paul Dixon, Eyal Finkelshtein, Daniel Rotman, Raja Giryes Abstract Speculative decoding accelerates autoregressive speech generation by letting a fast draft model propose tokens that a larger target model verifies. However, for speech LLMs that generate acoustic tokens, exact token matching is overly restrictive: many discrete tokens are acoustically or semantically interchangeable, reducing acceptance rates and limiting speedups. We introduce Principled Coarse-Graining (PCG), which verifies proposals at the level of Acoustic Similarity Groups (ASGs) derived from the target model’s embedding space. By splitting each token’s probability mass across the overlapping groups that contain it, we define an overlap-aware coarse-grained distribution and perform rejection sampling on the resulting group variable. This yields an exactness guarantee at the group level while allowing the accepted draft token to stand in for any member of the group in practice. On LibriTTS, PCG increases acceptance and throughput relative to standard speculative decoding and prior speech-specific relaxations while maintaining intelligibility and speaker similarity. These results suggest acoustically aware, group-level acceptance as a simple and general way to accelerate speech token generation while maintaining speech quality. † † address: 1 Apple, 2 Tel-Aviv University 1 Introduction Speculative decoding (SD) is a two-model approach for accelerating autoregressive decoding in large language models [ 1 , 2 ] . A small, fast draft model proposes token sequences that a larger, more accurate target model subsequently verifies or corrects. SD can deliver large wall-clock speedups while provably preserving the target model’s output distribution. This property is especially valuable for on-device TTS, where strict latency budgets and offline operation for privacy constrain compute. Figure 1 : Token-level probabilities used for the SD acceptance criterion: original vs PCG. Colors denote Acoustic Similarity Groups (ASGs); tokens may belong to multiple groups. Applying SD to autoregressive models over speech tokens poses distinct challenges. Many speech tokens are acoustically or phonetically interchangeable, so different tokens can yield perceptually similar audio [ 3 , 4 , 5 ] . Because standard SD accepts only exact token matches, it fails to exploit this many-to-one mapping and rejects draft tokens that are perceptually valid. The resulting low acceptance rates can erase the speed gains from accepted tokens. Prior adaptations of SD to speech synthesis address low acceptance by restricting sampling. Methods such as sequence-level Viterbi search [ 6 ] and tolerance-based validation [ 5 ] typically confine decoding to small top-k pools, which reduces output diversity [ 7 ] . SSD [ 3 ] instead increases acceptance by adding a constant probability bias, enabling long-tail sampling but ignoring acoustic similarity and risking erroneous acceptances. For visual AR models, LANTERN [ 8 ] relaxes acceptance by aggregating probability over latent neighbors and bounding distortion via total variation distance. In contrast, we introduce an explicit coarse-grained random variable and run acceptance/rejection on its induced distribution, guaranteeing exact sampling at the coarse level. While coarse-graining is well studied in statistical physics and explored in ML [ 9 , 10 , 11 ] , we are not aware of SD methods that define a group variable with overlap-aware acceptance and perform rejection sampling on its valid induced distribution. Figure 2 : Overview of speculative decoding with PCG. Tokens are first clustered into overlapping ASGs in the target model’s embedding space. The lightweight draft model then autoregressively samples these groups. Finally, the target model verifies all proposed groups in parallel. We propose Principled Coarse-Graining (PCG), a framework that replaces exact token matching with group-level verification. We construct Acoustic Similarity Groups (ASGs) in the target model’s token embedding space, capturing its internal organization of semantic and acoustic similarity. PCG performs speculative sampling on the coarse-grained distribution over ASGs and carries out rejection sampling at the group level. This yields a strong guarantee: the sequence of accepted groups is an exact sample from the target model’s coarse-grained distribution at each decoding step. Empirically, draft predictions are often correct at the group level, which substantially raises acceptance while maintaining diversity. Figure 1 illustrates the intuition. The top row shows draft and target distributions over individual tokens; the bottom row shows the same distributions after coarse-graining into ASGs. Under strict matching a draft sample like token 2 is likely rejected, whereas under PCG its group 1,2,3,4 is accepted because the target assigns high mass to the group even if not to the specific token. Overlap (e.g., token 4 belonging to multiple groups) preserves smooth acoustic neighborhoods. Our contributions are twofold: we introduce PCG, a group-level speculative decoding scheme that performs rejection sampling on a valid coarse-grained distribution over ASGs, guaranteeing exact coarse-level sampling at each step; and we provide a drop-in AR TTS implementation that raises acceptance and decoding speed while preserving target diversity, with ablations over group granularity and overlap. 2 Method Our framework addresses the challenges of speech token decoding through two key innovations. First, we define ASGs that capture how multiple discrete tokens map to perceptually similar sounds, with tokens belonging to multiple overlapping groups. Second, we introduce PCG, which performs group-level acceptance while accounting for these overlaps—preserving acoustic neighborhoods without arbitrary boundaries. See Figure 2 for an overview of speculative decoding with PCG. We denote the target model distribution by q and the draft model distribution by p . 2.1 Acoustic Similarity Groups (ASG) To capture the many-to-one mapping between discrete speech tokens and perceived sounds, we group tokens with similar embeddings from the target model q . Learned embeddings across text, vision, and audio exhibit rich semantic structure and cross-modal alignment, motivating embedding similarity as a perceptual proxy [ 12 , 13 , 14 ] . For a token t , its ASG is \mathcal{G}(t):=\{t^{\prime}\in\mathcal{V}:\cos(\text{Emb}_{q}(t),\text{Emb}_{q}(t^{\prime}))>\theta\}. (1) where \text{Emb}_{q}(\cdot) is the target model’s token embedding and \mathcal{V} is the vocabulary of the model. The threshold \theta controls the granularity: lower values create larger, more permissive groups. We find \theta\in[0.38,0.45] optimal in our experiments (Figure 3 ). The collection of distinct ASGs forms our group set \{\mathcal{G}_{k}\}_{k=1}^{M} , where each \mathcal{G}_{k} corresponds to a unique equivalence group and M\leq|\mathcal{V}| . 2.2 Principled Coarse-Graining (PCG) ASGs naturally overlap—tokens belong to multiple groups thus preserving acoustic neighborhoods without arbitrary boundaries. However, overlap prevents naive probability aggregation. PCG resolves this by splitting each token’s probability across its groups, yielding proper distributions P_{c} and Q_{c} for group-level acceptance. Overlap-aware coarse distributions. For a token t , let S(t):=\{k:t\in\mathcal{G}_{k}\} and N(t):=|S(t)| . We choose nonnegative weights w_{k,t} for k\in S(t) such that \sum_{k\in S(t)}w_{k,t}=1,\qquad w_{k,t}=0\;\text{if}\;t\notin\mathcal{G}_{k}. (2) Our default is equal-split, w_{k,t}=\frac{\mathbbm{1}\{t\in\mathcal{G}_{k}\}}{N(t)}. (3) Given w_{k,t} , define the coarse-grained group distributions P_{c}(\mathcal{G}_{k})=\sum_{t\in\mathcal{G}_{k}}p(t)\cdot w_{k,t},\qquad Q_{c}(\mathcal{G}_{k})=\sum_{t\in\mathcal{G}_{k}}q(t)\cdot w_{k,t}. (4) Because the weights in ( 2 ) sum to one per token, we have \sum_{k}P_{c}(\mathcal{G}_{k})=\sum_{t}p(t)=1 and \sum_{k}Q_{c}(\mathcal{G}_{k})=1 , hence no global normalization constants are needed. Under equal-split, P_{c}(\mathcal{G}_{k})=\sum_{t\in\mathcal{G}_{k}}\frac{p(t)}{N(t)},\qquad Q_{c}(\mathcal{G}_{k})=\sum_{t\in\mathcal{G}_{k}}\frac{q(t)}{N(t)}. (5) Proposal coupling and acceptance. At speculative step i , draw a draft token x_{i}\sim p_{i}(\cdot) . Because groups overlap, we then draw a group label K_{i} from the groups containing x_{i} using the same per-token weights: K_{i}\sim w_{k,x_{i}}\quad\text{over}\quad k\in S(x_{i}). (6) This coupling ensures \Pr(K_{i}=k)=\sum_{t}p_{i}(t)\cdot w_{k,t}=P_{c}^{(i)}(\mathcal{G}_{k}) , i.e., the proposal over groups equals the coarse-grained draft distribution at step i . The group-level acceptance ratio is then the standard speculative ratio for the same random

Read the complete text