Singular Proxies for Adaptive Caching in Diffusion Language Models Wenhao Sun, Rong-Cheng Tu §, Yifu Ding, Zhao Jin, Jingyi Liao, Yongcheng Jing, Dacheng Tao § Abstract While Diffusion Language Models (DLMs) offer a flexible, arbitrary-order alternative to the autoregressive paradigm, their non-causal nature precludes standard KV caching, forcing costly hidden state recomputation at every decoding step. Existing DLM caching approaches reduce this cost by selective hidden state updates; however, they are still limited by (i) costly token-wise update identification heuristics and (ii) rigid, uniform budget allocation that fails to account for heterogeneous hidden state dynamics. To address these challenges, we present SPA-Cache that jointly optimizes update identification and budget allocation in DLM cache. First, we derive a low-dimensional singular proxy that enables the identification of update-critical tokens in a low-dimensional subspace, substantially reducing the overhead of update identification. Second, we introduce an adaptive strategy that allocates fewer updates to stable layers without degrading generation quality. Together, these contributions significantly improve the efficiency of DLMs, yielding up to an 8 × 8\times throughput improvement over vanilla decoding and a 2 2 – 4 × 4\times speedup over existing caching baselines. The code repository is available at https://github.com/wenhao728/spa-cache . 1 Introduction Diffusion Language Models (DLMs) ( 2 ; 12 ; 27 ; 16 ) have recently emerged as a versatile alternative to the dominant autoregressive (AR) paradigm ( 6 ) , primarily characterized by their ability to decode in arbitrary orders and facilitate the potential of parallel decoding. By decoupling generation from a fixed left-to-right order, DLMs have demonstrated remarkable potential in handeling multi-modality tasks ( 49 ; 47 ) , overcoming the “reverse curse,” ( 5 ) enhancing diversity ( 17 ) , and scaling effectively across reasoning ( 53 ; 8 ; 54 ) . However, a significant performance gap remains: while proprietary industrial prototype models, such as Gemini Diffusion ( 11 ) and Mercury ( 25 ) , have begun to showcase ultra high throughput, current open-source implementations ( 30 ; 48 ) continue to suffer from prohibitive inference latency compared to their AR counterparts. This inefficiency is fundamentally rooted in the unpredictable decoding order and the resulting adaptation of bidirectional attention. Consequently, the standard KV-cache ( 31 ) , a cornerstone of efficiency in AR models, is rendered incompatible, as the hidden states of DLMs cannot be simply reused without recomputation to capture the evolving global dependencies. Recent efforts to address the DLM inference bottleneck have mainly focused on caching with heuristic-based updates, often assuming a locality bias, where only hidden states in the immediate vicinity of recently decoded tokens require updates ( 29 ; 24 ) . Others rely on restrictive block-wise semi-autoregressive decoding ( 46 ; 45 ; 35 ) , which compromises the arbitrary-order flexibility of DLMs and sacrifices global dependency variations. A more principled alternative ( 28 ) attempts to identify update-critical tokens by monitoring shifts in the Value states; however, this approach remains bottlenecked. First, the identification process is computationally heavy, requiring high-dimensional projection and similarity computations at every step. This overhead often offsets the gains from sparse updates as the generation lengths scale. Second, these methods typically apply a uniform update budget across all layers. As illustrated in Figure 2 , the fraction of drifting tokens, which necessitate updates, shows layer-wise heterogeneity. Consequently, a high fixed budget ( \eg , 30%) leads to computational redundancy in stable layers, whereas a lower budget ( \eg , 20%) triggers performance degradation due to the insufficient updating of highly unstable layers. To address these challenges, we present SPA-Cache , a framework that jointly optimizes update identification and budget allocation. We ground our approach in a formal analysis of the evolutionary dynamics of DLM hidden states, characterizing the statistical properties that dictate when and where updates are most critical. Building on these theoretical insights, we introduce a singular proxy that identifies update-critical tokens in the early stages of each layer ( Figure 1 ). Leveraging a singular approximation ( 37 ) of the identification process, our singular proxy retains the dominant directions of state drift in a low-dimensional manifold, thereby avoiding the computational bottleneck associated with high-dimensional identification. Furthermore, observing that hidden state stability fluctuates significantly across model layers, we depart from standard uniform update fractions in favor of an adaptive budget allocation scheme. By dynamically prioritizing computational resources for high-variance layers while aggressively caching stable ones, SPA-Cache maximizes throughput without sacrificing generation quality. Figure 1 : Adjacent-Step Similarities for LLaDA-8B-Instruct ( 30 ) . While input states exhibit uniformly high similarity, our singular proxy at early stage of the layer can efficiently uncover the drifts in the final FFN output. This provides a clear guidance for selective hidden state updates. Extensive evaluations on seven benchmarks demonstrate that SPA-Cache significantly optimizes DLM inference, delivering up to a 8\times increase in throughput with negligible impact on generation quality. Our SPA-Cache is fundamentally orthogonal to sequence-level optimizations and can be integrated with parallel decoding methods ( 46 ) to amplify gains to a total of 28\times . By drastically reducing inference overhead, this work paves the way and positions DLMs as a computationally viable alternative for high-throughput, real-world applications. 2 Related Work 2.1 Diffusion Language Models (DLMs) The transition of diffusion processes to language modeling originated with Gaussian-based continuous latent diffusion in Diffusion-LM ( 27 ) and structured discrete denoising in D3PM ( 2 ) . Subsequent frameworks such as CDCD ( 12 ) introduced score interpolation to bridge the discrete-continuous gap between latents and targets. Plaid ( 18 ) demonstrated that diffusion models follow predictable scaling laws, eventually outperforming smaller autoregressive (AR) models in zero-shot likelihood. More recently, MDLM ( 34 ) and LLaDA ( 30 ) have converged on iterative unmasking as a highly scalable DLM alternative to AR modeling, offering bidirectional context and inherent parallelism. However, the bidirectional nature of these models requires recomputing a forward pass over the entire sequence of length N at each decoding step, resulting in a severe inference bottleneck with O(T\times N^{2}) complexity for T decoding steps. Figure 2 : Distribution of Drift Across Layers. Rather than using a fixed threshold (dashed line) to allocate update budgets uniformly across layers, our SPA-Cache employs a dynamic threshold (solid line) that allocates more updates on the most volatile layers and fewer on the more stable layers. 2.2 Caching Mechanisms for Language Models KV cache ( 31 ) , a cornerstone of efficiency in AR language models, is fundamentally incompatible with DLMs due to the unpredictable decoding order and the subsequent reliance on full bidirectional attention ( 28 ) . Recent efforts, like dKV-Cache ( 29 ) , d2Cache ( 24 ) , and Fast-dLLM ( 46 ) propose window-based heuristics that selectively update tokens in the immediate vicinity of recently decoded ones while reusing distant hidden states. However, these methods often rely on a locality bias that lacks theoretical justification and may fail to capture complex global dependency variations. dLLM-Cache ( 28 ) monitors feature drift to identify and update critical tokens at arbitrary positions. Despite its flexibility, this approach remains limited by (i) the high overhead of high-dimensional similarity checks and (ii) a uniform update budget across all layers that fails to exploit the layer-wise stability profiles of DLMs. 2.3 Other Efficient Generation Techniques Beyond caching, researchers have explored cache eviction ( 35 ) and specialized attention variants ( 43 ; 52 ; 38 ; 39 ) to reduce the cost of bidirectional modeling. Furthermore, DLMs exhibit unique synergy with parallel decoding methods ( 22 ; 42 ; 46 ; 26 ) , where confidence-aware strategies are adopted to simultaneously decode multiple tokens. Caching methods are fully compatible with these parallel frameworks. By integrating our adaptive caching, SPA-Cache, with these parallel frameworks, we can further amplify throughput gains. 3 Method 3.1 Workflow Overview As discussed in Section 2.1 , the arbitrary-order decoding paradigm and the reliance on bi-directional self-attention in Diffusion Language Models (DLMs) render standard LM caching techniques inapplicable. To address this, we propose SPA-Cache for efficient DLM caching. The workflow for a single Transformer layer is partitioned into the following three distinct phases as shown in Algorithms 1 and 3 . Some operations ( \eg , residual connections, position embeddings) are omitted for clarity. Algorithm 1 SPA-Cache of a Transformer Block 0: Input states {\bm{H}} , cached KV {\bm{K}}^{c} , {\bm{V}}^{c} , cached FFN outputs {\bm{H}}^{c} , layer index l , maximum update ratio \rho_{p} 0: Output states {\bm{H}} Phase 1: Update Identification & Selection N\leftarrow\mathrm{len}({\bm{H}}) \triangleright Sequence length N \rho\leftarrow\mathrm{get\_ada\_rho}(\rho_{p},l) \triangleright Equation 5 ( Section 3.4 ) \mathcal{I}\leftarrow\mathrm{get\_update\_indices}({\bm{H}},k=N\cdot\rho) \triangleright Get update indices; Algorithm 2 ( Sections 3.2 and 3.3 ) {\bm{H}}_{\mathcal{I}}\leftarrow\mathrm{gather}({\bm{H}},\mathcal{I}) \triangleright \Pi_{\mathcal{I}} : Select states at indices \mathcal{I} ; \mathrm{len}({\bm{H}}_{\mathcal{I}}):k Phase 2: Attention with Partially Cached KV {\bm{Q}}_{\mathcal{I}},{\bm{K}}_{\mathcal{I}},{\bm{V}}_{\mathcal{I}}\leftarrow\mathrm{to\_qkv}({\bm{H}}_{\mathcal{I}}) {\bm{K}}^{c}\leftarrow\mathrm{scatter}({\bm{K}}^{c},\mathcal{I},{\bm{K}}_{\mathcal{I}}) {\bm{V}}^{c}\leftarrow\mathrm{scatter}({\bm{V}}^{c},\mathcal{I},{\bm{V}}_{\mathcal{I}}) \triangleright \mathrm{Upd} : Update KV; \mathrm{len}({\bm{K}}^{c}),\mathrm{len}({\bm{V}}^{c}):N {\bm{H}}_{\mathcal{I}}\leftarrow\mathrm{Attn}({\bm{Q}}_{\mathcal{I}},{\bm{K}}^{c},{\bm{V}}^{c}) Phase 3: FFN and Output States Update {\bm{H}}_{\mathcal{I}}\leftarrow\mathrm{FFN}({\bm{H}}_{\mathcal{I}}) {\bm{H}}^{c}\leftarrow{\bm{H}}\leftarrow\mathrm{scatter}({\bm{H}}^{c},\mathcal{I},{\bm{H}}_{\mathcal{I}}) \triangleright \mathrm{Upd} : Update output and its cache; \mathrm{len}({\bm{H}}):N Phase 1: Update Identification and Input States Selection. In this initial phase, the goal is to identify which tokens have drifted significantly to warrant recomputation. Since the drift in subsequent components is unknown beforehand, the challenge lies in determining the update priority efficiently. We employ an update identification Algorithm 2 , which will be detailed in Sections 3.2 and 3.3 , to select a set of update indices \mathcal{I} . These indices guide the selection of sparse inputs for computation while allowing the remaining tokens to stay dormant and use cache. Phase 2: Attention with Partially Cached Key-Value. The sparse inputs are used to generate a sparse Query {\bm{Q}}_{\mathcal{I}} , along with Key {\bm{K}}_{\mathcal{I}} and Value {\bm{V}}_{\mathcal{I}} states exclusively for the tokens indexed in \mathcal{I} . The Update ( \mathbf{\mathrm{Upd}} ) module then integrates these new KV states into the existing KV cache from previous decoding steps. This process refreshes the representations of the active tokens while reusing cached states for stable tokens. The Attention is computed between the sparse queries and the partially updated KV cache, yielding the sparse attention output. Phase 3: FFN and Output States Update. The final phase handles the feed-forward transitions. The sparse attention output is passed through a FFN component. Similar to Phase 2, an Update ( \mathbf{\mathrm{Upd}} ) module uses the indices \mathcal{I} to synchronize the newly computed output states with the cache and produce the final output states for the layer. This selective recomputation mechanism ensures that the computational complexity of both the Attention and FFN components scales with the number of updated tokens k rather than the total sequence length N . Figure 3 : Overview of a Single SPA-Cache Layer. Phase 1 : Input states are projected by f_{\mathrm{proxy}} to produce update scores. A Top- k selection identifies the update index set \mathcal{I} , which identifies positions whose cached states from the previous steps should be refreshed. Phase 2 : Queries are generated only from the selected states and attend to a key-value cache that is selectively updated at indices \mathcal{I} . Phase 3 : The resulting sparse attention outputs are fed into the MLP for the layer output. The output states is updated at \mathcal{I} , while non-selected positions reuse cached features. 3.2 On the Choice of Update Identifier Algorithm 2 Update Identification 0: Input states {\bm{H}}=[{\bm{h}}_{1},\ldots,{\bm{h}}_{N}] , cached proxy identifiers {\bm{P}}^{c}=[{\bm{p}}_{1}^{c},\ldots,{\bm{p}}_{N}^{c}] , update size k 0: Update indices \mathcal{I} for i=1 to N do {\bm{p}}_{i}\leftarrow f_{\mathrm{proxy}}({\bm{h}}_{i}) \triangleright Compute new identifiers {\bm{s}}_{i}\leftarrow\mathcal{S}_{\cos}({\bm{p}}_{i},{\bm{p}}_{i}^{c}) \triangleright Similarity scores end for {\bm{P}}\leftarrow[{\bm{p}}_{1},\ldots,{\bm{p}}_{N}] ; {\bm{S}}\leftarrow[{\bm{s}}_{1},\ldots,{\bm{s}}_{N}] \mathcal{I}\leftarrow\mathrm{topk}({\bm{S}},k,\text{smallest=True}) \triangleright Indices to update {\bm{P}}^{c}\leftarrow\mathrm{scatter}({\bm{P}}^{c},\mathcal{I},{\bm{P}}) \triangleright Update identifiers The core of our framework is the update identification mechanism in Phase 1, which selectively determines which tokens require recomputation. As detailed in Algorithm 2 , we first project the input hidden states [{\bm{h}}_{i}] into proxy identifier vectors [{\bm{p}}_{i}] via a proxy projection f_{\mathrm{\mathrm{proxy}}}(\cdot) . To quantify temporal changes, we compute the cosine similarity \mathcal{S}_{\cos}(\cdot) between the current identifier {\bm{p}}_{i} and the cached identifier {\bm{p}}_{i}^{c} from the previous decoding steps. The update set \mathcal{I} is then constructed by selecting the Top- k tokens with the lowest similarity scores, effectively prioritizing those with the most significant state deviations. 28 has empirically utilized Value states to guide the update of drifting states, \ie {\bm{p}}_{i}=f_{\mathrm{\mathrm{proxy}}}({\bm{h}}_{i})={\bm{W}}{\bm{h}}_{i}={\bm{v}}_{i} , where {\bm{W}}\in{\mathbb{R}}^{d\times d} is the Value projection matrix. However, such design choice was motivated by experimental correlations. In this section, we formalize the theoretical underpinnings of why Value states constitute a principled identifier for state updates. Specifically, we provide analyses of feature drift propagation through linear projections and non-linearities. Full proofs for the following theorems are provided in Appendix A . We first establish that the stability of the Value states is a sufficient condition for the stability of the attention output. Theorem 3.1 ( Bound on Attention Output Similarity ) . Consider the Value states {\bm{v}}_{i}^{t}\in\mathbb{R}^{d} and attention output {\bm{h}}_{i}^{t}\in\mathbb{R}^{d} for the i -th token at decoding steps t of a DLM. Under the following assumptions: 1. Bounded Norms: The Value norms \|{\bm{v}}_{i}\| and attention output norms \|{\bm{h}}_{i}\| are bounded. 2. Stable Attention: The total variation of the attention weights is bounded: \sum_{j}|\alpha_{ij}^{t+1}-\alpha_{ij}^{t}|\leq\delta_{A} . 3. Bounded Drift: The drift of output states is controlled by its value state: \sum_{j}\alpha_{ij}^{t+1}\|\mathbf{v}_{j}^{t+1}-\mathbf{v}_{j}^{t}\|\leq\lambda\|\mathbf{v}_{i}^{t+1}-\mathbf{v}_{i}^{t}\| , where \lambda\geq 1 is a relaxation constant. Then, the cosine dissimilarity of the output \mathbf{h}_{i} is bounded by the dissimilarity of the i -th value state \mathbf{v}_{i} : 1-\mathcal{S}_{\cos}({\bm{h}}_{i}^{t},{\bm{h}}_{i}^{t+1})\leq C\cdot(1-\mathcal{S}_{\cos}({\bm{v}}_{i}^{t},{\bm{v}}_{i}^{t+1}))+\epsilon, (1) where C>0 is a scaling constant determined by the norm bounds, and \epsilon>0 is a residual term capturing attention shifts and norm variance (detailed in Appendix A.1 ). This implies that if the Value states maintain high similarity, the resulting attention output drift is upper bounded. Thus, monitoring Value states provides a conservative guarantee for the stability of the attention output. Next, we argue that the FFN preserves this stability. Theorem 3.2 ( Bound on FFN Output Divergence ) . Let f_{\mathrm{FFN}}(\cdot) be a standard FFN component comprising linear projections, RMSNorm ( 50 ) , and SiLU ( 13 ) activation functions. Let {\bm{h}}_{1},{\bm{h}}_{2}\in\mathbb{R}^{d} be two FFN input vectors. The divergence of the FFN outputs is bounded by the cosine similarity of their inputs: \|f_{\mathrm{FFN}}({\bm{h}}_{1})-f_{\mathrm{FFN}}({\bm{h}}_{2})\|_{2}\leq C\cdot\sqrt{1-\mathcal{S}_{\cos}({\bm{h}}_{1},{\bm{h}}_{2})}+\epsilon, (2) where C>0 and \epsilon>0 are constants depending on the Lipschitz constant of the network and the feature norms. Remark 3.3 . More generally, this theorem holds for normalizations that project onto a bounded hypersphere ( \eg , LayerNorm ( 4 ) ) and Lipschitz-continuous activations ( \eg , GELU ( 20 ) ). The bounding property can also be extended to the more recent MoE ( 51 ) as they are intrinsically composed of standard FFNs. Theorem 3.1 and Theorem 3.2 together provide the theoretical guarantee for the Value identification mechanism: if the Value states maintain high cosine similarity, the final layer output drifts are bounded. Consequently, it is unnecessary (and impractical) to speculatively evaluate the whole layer to detect divergence; the similarity of Value states act as a sufficient proxy for determining which tokens can safely reuse cached states. Table 1 : Comparison of different identifier types on LLaDA-8B-Instruct ( 30 ) and GSM8K ( 10 ) . TPS: Tokens Per Second, TTFT: Time To First Token (in milliseconds). Identifier type TPS \uparrow TTFT \downarrow Accuracy \uparrow Baseline (None) 29.01 28.6 78.62\ _{(\pm 1.13)} Query 164.36 28.4 77.21\ _{(\pm 1.15)} Key 164.11 28.1 76.83\ _{(\pm 1.15)} Value 164.88 28.1 \mathbf{78.59\ _{(\pm 1.13)}} Attn. Input \mathbf{172.47} 28.6 77.29\ _{(\pm 1.14)} Attn. Output 139.18 28.0 73.92\ _{(\pm 1.21)} Emperical Evaluation of the Identifiers. To validate our theoretical findings, we perform an empirical comparison of various hidden state components as update identifiers in Table 1 . Specifically, we evaluate the efficacy when using Query, Key, Value, attention input, and attention output states as update identifiers. As demonstrated in Table 1 , the Value state outperforms all other candidates, providing the most reliable signal for hidden state drift. Query and Key serve as routing features; while shifts in their values may alter the attention score distribution, the resulting output remains stable if the attended Value vectors are semantically equivalent. The attention input state is an entangled representation of Query, Key, and Value states, making it less effective for isolating drift. Notably, using attention output states as identifiers yields the most significant accuracy degradation. This limitation stems from the Anisotropy Masking Effect ( 14 ; 15 ; 41 ) , in which the representation space of deeper features collapses into a narrow cone. This dominance of a few directions masks the unique features necessary for robust identification. We provide an in-depth analysis of this failure mode in Appendix B . 3.3 Singular Proxy Identifier Although the Value states are justified as a reliable identifier for update selection, projecting the whole input sequence to Values and computing similarity in the d -dimensional Value space are computationally expensive. In moderate-parameterized DLMs such as LLaDA-8B ( d=4096 ), the identification process in this d -dimensional space incurs an overhead that can offset the gains from sparse computation. To address this challenge, we introduce the singular proxy , a method that designed to identify update-critical tokens within a low-dimensional subspace r\ll d . We perform the Singular Value Decomposition (SVD) ( 37 ) on the Value projection matrix {\bm{W}}\in\mathbb{R}^{d\times d} such that {\bm{W}}\approx{\bm{U}}\mathbf{\Lambda}{\bm{V}}^{\top} , where {\bm{U}},{\bm{V}}\in\mathbb{R}^{d\times d} denote the left and right singular vectors, and \mathbf{\Lambda}=\mathrm{diag}(\lambda_{1},\ldots,\lambda_{d})\in\mathbb{R}^{d\times d} represents the singular values in descending order. We define a truncated projection {\bm{W}}_{r}\in\mathbb{R}^{r\times d} by retaining the top r right singular vectors and singular values. This yields the dimension-reduction projection, f_{\mathrm{\mathrm{proxy}}}:\mathbb{R}^{d}\to\mathbb{R}^{r} : f_{\mathrm{\mathrm{proxy}}}({\bm{h}}_{i})={\bm{W}}_{r}{\bm{h}}_{i}=(\mathbf{\Lambda}_{r}{\bm{V}}_{r}^{\top}){\bm{h}}_{i}, (3) where \mathbf{h}_{i}\in\mathbb{R}^{d} is the i -th input state. By projecting the input into the principal subspace, we reduce the computational complexity of the identifier mapping f_{\mathrm{\mathrm{proxy}}}(\cdot) from O(d^{3}) to O(rd^{2}) and the similarity computation \mathcal{S}_{\cos}(\cdot) from O(d) to O(r) . The remaining pipeline still follows the procedure outlined in Algorithm 2 . While truncated SVD preserves the majority of variance, it is critical to ensure that the topological structure remains intact within the reduced subspace. To formalize this, we establish the following bound on similarity divergence: Theorem 3.4 ( Similarity Preservation in the Truncated Projection ) . Let {\bm{h}}_{1},{\bm{h}}_{2}\in\mathrm{span}({\bm{V}}_{r}) be the input states. Denote {\bm{v}}={\bm{W}}{\bm{h}} and \hat{{\bm{v}}}={\bm{W}}_{r}{\bm{h}} as the Value states and our singular proxy, respectively. Their divergence of the cosine similarity is bounded by a factor independent of the input states {\bm{h}}_{1},{\bm{h}}_{2} : |\mathcal{S}_{\cos}({\bm{v}}_{1},{\bm{v}}_{2})-\mathcal{S}_{\cos}(\hat{{\bm{v}}}_{1},\hat{{\bm{v}}}_{2})|\leq 2\left(\frac{\lambda_{r+1}}{\lambda_{r}}\right)^{2}. (4) Remark 3.5 . The {\bm{h}}_{1},{\bm{h}}_{2}\in\mathrm{span}({\bm{V}}_{r}) assumption typically holds as the cumulative update to each neuron can be viewed as a weighted summation of the training input vectors ( 9 ) . The rigorous proof is provided in Appendix A.3 . By leveraging Theorem 3.4 , we can analytically ensure the singular proxy is an effective alternative for the Value proxy. This allows for computational reductions during the update identification process. 3.4 Adaptive Budget Allocation Table 2 : Comparison of LLaDA-8B-Instruct ( 30 ) and Dream-v0-Instruct-7B ( 48 ) across seven benchmarks. Compared to prior dLLM-Cache ( 28 ) and Fast-dLLM ( 46 ) , our SPA-Cache achieves better speedups while maintaining comparable accuarcy. Task Method LLaDA-8B-Instruct Dream-v0-Instruct-7B TPS \uparrow TTFT (ms) \downarrow Accuracy \uparrow TPS \uparrow TTFT (ms) \downarrow Accuracy \uparrow Mathematics & Science Baseline 29.67\ _{(1.0\times)} 28.0 78.62\ _{(\pm 1.13)} 17.86\ _{(1.0\times)} 23.6 75.21\ _{(\pm 1.19)} + dLLM-Cache 68.62\ _{(2.3\times)} 28.7 78.24\ _{(\pm 1.14)} 23.39\ _{(1.3\times)} 25.3 74.45\ _{(\pm 1.20)} + Fast-dLLM 93.86\ _{(3.2\times)} 28.1 76.80\ _{(\pm 1.16)} 25.53\ _{(1.4\times)} 24.6 79.23\ _{(\pm 1.12)} GSM8K + Ours \mathbf{190.73\ _{(6.4\times)}} 28.7 78.24\ _{(\pm 1.14)} \mathbf{45.97\ _{(2.6\times)}} 24.5 77.56\ _{(\pm 1.15)} Baseline 20.55\ _{(1.0\times)} 44.8 29.91\ _{(\pm 2.17)} 12.89\ _{(1.0\times)} 61.2 34.15\ _{(\pm 2.24)} + dLLM-Cache 27.29\ _{(1.3\times)} 44.9 29.01\ _{(\pm 2.15)} 25.40\ _{(2.0\times)} 61.4 33.04\ _{(\pm 2.22)} + Fast-dLLM 25.91\ _{(1.3\times)} 44.4 27.23\ _{(\pm 2.11)} 30.63\ _{(2.4\times)} 61.3 33.93\ _{(\pm 2.24)} GPQA + Ours \mathbf{39.19\ _{(1.9\times)}} 44.6 30.58\ _{(\pm 2.18)} \mathbf{45.97\ _{(3.6\times)}} 61.8 34.60\ _{(\pm 2.25)} Baseline 33.35\ _{(1.0\times)} 23.3 33.18\ _{(\pm 0.62)} 23.08\ _{(1.0\times)} 20.2 36.16\ _{(\pm 0.63)} + dLLM-Cache 74.26\ _{(2.2\times)} 23.2 33.14\ _{(\pm 0.62)} 42.08\ _{(1.8\times)} 21.9 32.86\ _{(\pm 0.63)} + Fast-dLLM 85.94\ _{(2.6\times)} 23.2 32.02\ _{(\pm 0.62)} 89.05\ _{(3.9\times)} 20.9 35.04\ _{(\pm 0.63)} MATH500 + Ours \mathbf{172.19\ _{(5.2\times)}} 24.8 33.44\ _{(\pm 0.62)} \mathbf{104.23\ _{(4.5\times)}} 20.8 34.84\ _{(\pm 0.63)} General QA Baseline 24.85\ _{(1.0\times)} 16.2 53.49\ _{(\pm 0.54)} 49.30\ _{(1.0\times)} 14.0 58.42\ _{(\pm 0.52)} + dLLM-Cache 40.48\ _{(1.6\times)} 16.5 52.74\ _{(\pm 0.55)} 54.65\ _{(1.1\times)} 14.0 58.22\ _{(\pm 0.52)} + Fast-dLLM 105.82\ _{(4.3\times)} 17.0 54.66\ _{(\pm 0.56)} 123.05\ _{(2.5\times)} 13.2 57.79\ _{(\pm 0.52)} BBH + Ours \mathbf{108.31\ _{(4.4\times)}} 16.1 53.72\ _{(\pm 0.55)} \mathbf{153.61\ _{(3.1\times)}} 13.2 58.52\ _{(\pm 0.52)} Baseline 20.68\ _{(1.0\times)} 43.8 37.08\ _{(\pm 0.43)} 10.06\ _{(1.0\times)} 33.9 45.01\ _{(\pm 0.44)} + dLLM-Cache 52.71\ _{(2.5\times)} 43.5 36.50\ _{(\pm 0.43)} 15.53\ _{(1.5\times)} 34.3 44.16\ _{(\pm 0.44)} + Fast-dLLM 81.25\ _{(3.9\times)} 44.1 37.28\ _{(\pm 0.43)} 50.58\ _{(5.0\times)} 34.2 46.62\ _{(\pm 0.44)} MMLU-pro + Ours \mathbf{124.06\ _{(6.0\times)}} 44.2 36.30\ _{(\pm 0.43)} \mathbf{62.13\ _{(6.2\times)}} 35.0 45.16\ _{(\pm 0.44)} Code Baseline 5.75\ _{(1.0\times)} 28.7 39.20 36.51\ _{(1.0\times)} 22.2 57.40 + dLLM-Cache 8.38\ _{(1.5\times)} 28.8 39.00 50.31\ _{(1.4\times)} 21.9 56.20 + Fast-dLLM 12.49\ _{(2.2\times)} 28.6 37.80 58.42\ _{(1.6\times)} 21.7 51.00 MBPP + Ours \mathbf{46.12\ _{(8.0\times)}} 28.7 39.00 \mathbf{114.85\ _{(3.1\times)}} 22.6 57.60 Baseline 37.48\ _{(1.0\times)} 14.4 42.07 28.00\ _{(1.0\times)} 14.0 58.54 + dLLM-Cache 40.29\ _{(1.1\times)} 14.3 39.63 21.15\ _{(0.8\times)} 14.1 45.12 + Fast-dLLM 81.90\ _{(2.2\times)} 14.3 41.71 40.34\ _{(1.4\times)} 13.8 53.05 HumanEval + Ours \mathbf{132.91\ _{(3.5\times)}} 14.3 42.07 \mathbf{42.49\ _{(1.5\times)}} 13.4 58.54 While the singular proxy identifier in Section 3.3 reduces per-token identification overhead, the layer’s computation is also governed by the update ratio \rho ( \ie , the fraction of tokens updated). Prior method ( 28 ) typically employs a uniform budget allocation, maintaining a static update ratio ( \eg , \rho=0.25 ) across all layers. However, our analysis of hidden state evolution reveals significant layer-wise heterogeneity that renders uniform allocation less efficient. As shown in Figure 2 , we evaluate the average ratio of highly drifting tokens per layer (defined as tokens whose cosine similarity with last decoding step falls below a threshold \tau=0.95 ). This analysis, conducted over 100 random samples from GSM8K ( 10 ) , MMLU-pro ( 44 ) , and MBPP ( 3 ) , demonstrates a consistent distribution across diverse tasks. Specifically, we observe that state drift varies drastically across the model depth: 1. Hidden states remain relatively stable as they primarily perform initial embedding transformations in the early layers; 2. Drift peaks as the model performs complex semantic transformations in the middle layers, requiring more frequent updates; 3. Dynamics stabilize again as approaching the final layer of the model. A fixed update threshold ( \eg , \rho=0.25 , represented by the dashed line in Figure 2 ) essentially over-allocates budget to the stable early layers and late layers. To exploit this heterogeneity, SPA-Cache replaces rigid allocation with an adaptive budget allocation scheme. Noticing that the drift curve roughly follows an asymmetric bell shape, we parameterize the layer-wise dynamic update ratio \rho(l) using a piecewise Gaussian function: \rho(l)=\begin{cases}\rho_{p}\cdot\exp{\left(\ln{\left(\frac{\rho_{1}}{\rho_{p}}\right)}\cdot\left(\frac{l-l_{p}}{l_{p}-1}\right)^{2}\right)}&\text{if }l\leq l_{p}\\ \rho_{p}\cdot\exp{\left(\ln{\left(\frac{\rho_{L}}{\rho_{p}}\right)}\cdot\left(\frac{l-l_{p}}{L-l_{p}}\right)^{2}\right)}&\text{if }l>l_{p}\end{cases}, (5) where, l denotes the current layer index. L is the total layer nubmer of the model. \rho_{p} is the peak update ratio at layer l_{p} . And \rho_{1},\rho_{L} represent the boundary update ratios for the first and last layers, respectively. This parameterization provides the flexibility to adapt to various DLMs with different skewness in their profiles. More analysis and configurations are detailed in Appendix C . The dynamic strategy ensures that computational resources are concentrated on high-variance layers where updates are most critical for preserving accuracy, while aggressively caching stable layers to maximize throughput. 4 Experiments 4.1 Experimental Setup Models and Benchmarks. We evaluate SPA-Cache across two representative Diffusion Language Models (DLMs): LLaDA-8B ( 30 ) and Dream-7B ( 48 ) . To assess generalizability, we conduct evaluations on seven diverse benchmarks covering mathematical reasoning (GSM8K ( 10 ) , MATH500 ( 19 ) ), scientific knowledge (GPQA ( 32 ) ), general-purpose QA (BBH ( 40 ) , MMLU-pro ( 44 ) ), and code generation (MBPP ( 3 ) , HumanEval ( 7 ) ). Baselines. We compare SPA-Cache against vanilla DLM decoding (no cache) and two state-of-the-art caching frameworks: dLLM-Cache ( 28 ) and Fast-dLLM ( 46 ) . Implementation Details. All experiments are executed on a single NVIDIA B200 GPU. For all experiments, we adopt the configuration from ( ) and set the allocation hyperparameter to \rho_{p}=0.25 . We report accuracy (for QA/Reasoning) and pass@1 (for coding tasks). To quantify efficiency gains, we measure average Tokens Per Second (TPS) and Time-To-First-Token (TTFT). Further implementation details and additional results on the post-trained LLaDA-1.5 ( 54 ) are provided in Appendix D and E , respectively. Table 3 : Integration of SPA-Cache with parallel strategy ( 46 ) . Our approach outperforms the Fast-dLLM dual cache in throughput while preserving comparable accuracy. Task Method TPS \uparrow Accuracy \uparrow Mathematics & Science Baseline 29.67 \ {}_{(1.0\times)} 78.62\ _{(\pm 1.13)} + Fast-dLLM 176.45 \ {}_{(5.9\times)} 77.02\ _{(\pm 1.16)} GSM8K + Ours 276.39 \ {}_{(9.3\times)} 76.81\ _{(\pm 1.18)} Baseline 20.55 \ {}_{(1.0\times)} 29.91\ _{(\pm 2.17)} + Fast-dLLM 44.40 \ {}_{(2.2\times)} 27.68\ _{(\pm 2.12)} GPQA + Ours 47.82 \ {}_{(2.3\times)} 27.01\ _{(\pm 2.10)} Baseline 33.35 \ {}_{(1.0\times)} 33.18\ _{(\pm 0.62)} + Fast-dLLM 86.69 \ {}_{(2.6\times)} 31.76\ _{(\pm 0.62)} MATH500 + Ours 289.71 \ {}_{(8.7\times)} 32.66\ _{(\pm 0.63)} General QA Baseline 24.85 \ {}_{(1.0\times)} 51.49\ _{(\pm 0.54)} + Fast-dLLM 301.33 \ {}_{(12.1\times)} 53.20\ _{(\pm 0.56)} BBH + Ours 693.96 \ {}_{(27.9\times)} 53.13\ _{(\pm 0.56)} Baseline 20.68 \ {}_{(1.0\times)} 37.08\ _{(\pm 0.43)} + Fast-dLLM 86.40 \ {}_{(4.2\times)} 37.34\ _{(\pm 0.43)} MMLU-pro + Ours 224.97 \ {}_{(10.9\times)} 36.85\ _{(\pm 0.42)} Code Baseline 5.75 \ {}_{(1.0\times)} 39.20 + Fast-dLLM 50.11 \ {}_{(8.7\times)} 37.80 MBPP + Ours 143.25 \ {}_{(24.9\times)} 38.20 Baseline 37.48 \ {}_{(1.0\times)} 42.07 + Fast-dLLM 117.67 \ {}_{(3.1\times)} 39.81 HumanEval + Ours 215.83 \ {}_{(5.8\times)} 40.24 4.2 Main Results Table 2 shows that SPA-Cache consistently outperforms existing caching methods across all benchmarks. By leveraging both the singular proxy identifier and adaptive budget allocation SPA-Cache achieves substantial throughput improvements and maintains comparable accuracy to vanilla decoding. Specifically, on the MMLU-pro benchmark, SPA-Cache yields a 6.0\times and 6.2\times speedup on LLaDA-8B and Dream-7B, respectively. The most significant gain is observed on MBPP with LLaDA-8B, where SPA-Cache attains an 8.0\times throughput acceleration. 4.3 Integration with Parallel Decoding. An advantage of DLMs is their inherent potentiality for parallel decoding ( 30 ) . SPA-Cache is orthogonal to these techniques. To demonstrate this synergy, we integrate SPA-Cache with the parallel decoding framework from 46 . As shown in Table 3 , combining SPA-Cache with parallelization yields further efficiency gains. While parallel decoding introduces a marginal distribution approximation error (resulting in a slight accuracy trade-off), the throughput improvements are remarkable. On the BBH benchmark, the combined approach achieves an unprecedented 28\times speedup over vanilla decoding. Crucially, SPA-Cache continues to outperform Fast-dLLM’s dual cache within the parallelized setting, showcasing its robustness. 4.4 Ablations Table 4 : Ablation on identifier and adaptive budget allocation. Identifier Peak \rho_{p} Avg \bar{\rho} TPS \uparrow Accuracy \uparrow None 100% 100% 29.01 78.62\ _{(\pm 1.13)} Value 25% 25% 164.88 \mathbf{78.59\ _{(\pm 1.13)}} Singular 128 25% 25% 179.43 78.23\ _{(\pm 1.12)} Singular 128 25% 16% \mathbf{189.13} 78.24\ _{(\pm 1.14)} Singular 128 16% 16% \mathbf{190.06} 75.65\ _{(\pm 1.17)} In this section, we conduct extensive ablation studies to systematically evaluate the individual contributions of our proposed components to the overall performance of SPA-Cache in practaice. Unless otherwise specified, all experiments in this section are performed using the LLaDA-8B model ( 30 ) on the GSM8K ( 10 ) benchmark. Efficacy of Proposed Components. We isolate the contributions of the singular proxy and adaptive allocation in Table 4 . Replacing the Value proxy identifier with our singular proxy identifier ( Section 3.3 ) increases TPS from 165 to 179 without degrading accuracy. Furthermore, introducing the adaptive budget allocation ( Section 3.4 ) allows the model to reduce the average update ratio from \bar{\rho}=25\% to \bar{\rho}=16\% , pushing the TPS to 189. Notably, when we ignore the layer-wise heterogeneity and force a uniform 16\% update ratio across all layers, accuracy drops from 78.23\% to 75.56\% . This confirms that our adaptive strategy successfully identifies and exploits stable layers to save computation without sacrificing model performance. Impact of Singular Proxy Rank. The dimension of the singular proxy ( r ) controls the trade-off between identification overhead and approximation quality. We quantitatively analyze this in Table 5 . We observe that while lower dimensions further improve TPS, accuracy begins to decline when r<128 . We therefore select r=128 as the sweet spot and use it as the default value across all experiments. Table 5 : Impact of Proxy Rank r on LLaDA-8B-Instruct ( 54 ) , GSM8K ( 10 ) . While the full-dimensional Value identifier uses d=4096 , our singular value proxy achieves comparable performance at r=512 . Notably, r=128 strike a good balance between throughput and accuracy. Identifier TPS \uparrow Accuracy \uparrow None 29.01 78.62\ _{(\pm 1.13)} Value 164.88 78.59\ _{(\pm 1.13)} Singular 512 172.56 78.57\ _{(\pm 1.13)} Singular 256 176.38 78.47\ _{(\pm 1.12)} Singular 128 179.43 78.23\ _{(\pm 1.12)} Singular 64 181.83 77.79\ _{(\pm 1.14)} Singular 32 182.66 76.71\ _{(\pm 1.15)} Figure 4 : Component-wise Latency Decomposition. Value proxy reduces Attention and FFN overhead, but it introduces additional computational cost during update identification. Our proposed singular proxy further reduces this identification cost. Analysis of Computational Overhead. To investigate the source of our efficiency gains, we profile the per-layer breakdown for LLaDA-8B in Figure 4 (setting prompt length 1024 , decoding length 256 , and \rho=5\% ). In the vanilla baseline, execution time is dominated by the linear projections and non-linearities in the Attention and FFN components. While the Value proxy caching methods successfully bypass these primary costs, they introduce a non-trivial identification overhead that emerges as a new computational bottleneck. In contrast, SPA-Cache’s singular proxy enables token selection in a highly compressed subspace, drastically reducing identification latency. This optimization allows the total execution time to approach the theoretical floor, maximizing the practical utility of the sparse update strategy. 5 Discussion Alternative Design Explorations. We investigated another alternatives that ultimately proved sub-optimal. In current DLM implementations with moderate sequence lengths ( \eg \leq 2048 ), the Attention is relatively lightweight compared to the FFN component. We hypothesized that using the Attention output as an identifier would provide a higher-fidelity signal for update selection. However, as shown in Table 1 , identification accuracy dropped significantly. Post-mortem analysis suggests this is due to the Anisotropy Masking Effect (see Appendix B ), where the Attention output exhibits a lower signal-to-noise ratio for hidden state dynamics. Limitations. While SPA-Cache demonstrates superior performance across various benchmarks, it possesses certain limitations. Like other DLM caching methods ( 28 ; 46 ; 29 ) , its efficacy relies on the temporal smoothness of hidden state transitions. In high-temperature sampling regimes ( \eg \tau\in[1.0,1.5] ), the increased stochasticity can lead to reduced accuracy. Additionally, deploying SPA-Cache in distributed or tensor-parallel settings may require advanced engineering specifications, increasing the implementation complexity. Further study. SPA-Cache primarily optimizes the decoding phase, leaving the prefilling latency (Time-to-First-Token, TTFT) largely unaffected. Furthermore, as current open-source DLMs do not yet support extended context windows, evaluating their performance on ultra-long sequences ( \eg , >32 k tokens) will be a critical area of investigation as foundation models evolve to incorporate such capabilities. 6 Conclusion This paper introduces SPA-Cache, a training-free and principled caching framework designed to address the inference bottlenecks of diffusion language models. Moving beyond purely empirical observations, we provide a theoretical foundation for hidden state update identification, justifying the use of Value states as an effective proxy. By projecting the identification process into a low-dimensional manifold through our singular proxy approach, we significantly reduce overhead of update identification. Furthermore, we capitalize on the discovered layer-wise heterogeneity, utilizing an adaptive budget allocation strategy to avoid redundant computations in stable layers. Extensive benchmarks confirm that SPA-Cache delivers substantial throughput gains without compromising generation fidelity, and its seamless integration with parallel decoding underscores its versatility. We believe SPA-Cache establishes a new efficiency baseline and will catalyze further research into the practical deployment of discrete diffusion models. Appendix A Proof of Theorems in Section 3 A.1 Proof of Theorem 3.1 Proof. We adopt two stability assumptions: 1. Stable Attention: The total variation of attention weights is bounded: \sum_{j}|\alpha_{ij}^{t+1}-\alpha_{ij}^{t}|\leq\delta_{A} . 2. Maximal Drift at i : The drift of the i -th Value vector upper-bounds the weighted drift of the context: \sum_{j}\alpha_{ij}^{t+1}\|{\bm{v}}_{j}^{t+1}-{\bm{v}}_{j}^{t}\|\leq\lambda\|{\bm{v}}_{i}^{t+1}-{\bm{v}}_{i}^{t}\| , where \lambda\geq 1 is a relax constant. We define the drift of the output vector as \Delta{\bm{h}}_{i}={\bm{h}}_{i}^{t+1}-{\bm{h}}_{i}^{t} . Using the triangle inequality, we separate the drift into Value shifts and Attention shifts: \|\Delta{\bm{h}}_{i}\|=\left\|\sum_{j}\alpha_{ij}^{t+1}{\bm{v}}_{j}^{t+1}-\sum_{j}\alpha_{ij}^{t}{\bm{v}}_{j}^{t}\right\|\leq\underbrace{\sum_{j}\alpha_{ij}^{t+1}\|{\bm{v}}_{j}^{t+1}-{\bm{v}}_{j}^{t}\|}_{\text{Weighted Value Drift }(\mathcal{D}_{v})}+\underbrace{\sum_{j}|\alpha_{ij}^{t+1}-\alpha_{ij}^{t}|\|{\bm{v}}_{j}^{t}\|}_{\text{Attention Shift }(\mathcal{E}_{\alpha})}. (6) For the value drift term \mathcal{D}_{v} : \mathcal{D}_{v}\leq\lambda\|{\bm{v}}_{i}^{t+1}-{\bm{v}}_{i}^{t}\|. (7) Let the norms be bounded such that \|{\bm{v}}\|\in[V_{\min},V_{\max}] and \|{\bm{h}}\|\in[H_{\min},H_{\max}] ( 33 ; 23 ) . For the attention shift term \mathcal{E}_{\alpha} , using the bound \delta_{A} and max norm V_{\max} : \mathcal{E}_{\alpha}\leq\delta_{A}V_{\max}. (8) Let \Delta_{V}=V_{\max}-V_{\min} be the maximum norm fluctuation. \displaystyle\|{\bm{v}}_{i}^{t+1}-{\bm{v}}_{i}^{t}\| \displaystyle=\sqrt{(\|{\bm{v}}_{i}^{t+1}\|-\|{\bm{v}}_{i}^{t}\|)^{2}+2\|{\bm{v}}_{i}^{t+1}\|\|{\bm{v}}_{i}^{t}\|(1-\mathcal{S}_{\cos}({\bm{v}}_{i}^{t},{\bm{v}}_{i}^{t+1}))} (9) \displaystyle\leq\sqrt{\Delta_{V}^{2}+2V_{\max}^{2}(1-\mathcal{S}_{\cos}({\bm{v}}_{i}^{t},{\bm{v}}_{i}^{t+1}))} (10) Using the subadditivity of the square root ( \sqrt{a+b}\leq\sqrt{a}+\sqrt{b} ): \|{\bm{v}}_{i}^{t+1}-{\bm{v}}_{i}^{t}\|\leq\Delta_{V}+V_{\max}\sqrt{2(1-\mathcal{S}_{\cos}({\bm{v}}_{i}^{t},{\bm{v}}_{i}^{t+1}))}. (11) Substituting this back into the bound for \|\Delta{\bm{h}}_{i}\| : \|\Delta{\bm{h}}_{i}\|\leq\lambda V_{\max}\sqrt{2(1-\mathcal{S}_{{\bm{v}}_{i}})}+(\lambda\Delta_{V}+\delta_{A}V_{\max}). (12) Let X=\lambda V_{\max}\sqrt{2(1-\mathcal{S}_{{\bm{v}}_{i}})} be the signal term and Y=\lambda\Delta_{V}+\delta_{A}V_{\max} be the noise term. From the geometry of cosine similarity: \mathcal{S}_{\cos}({\bm{h}}_{i}^{t},{\bm{h}}_{i}^{t+1})\geq 1-\frac{\|\Delta{\bm{h}}_{i}\|^{2}}{2H_{\min}^{2}}. (13) Rearranging for dissimilarity ( 1-\mathcal{S} ): 1-\mathcal{S}_{\cos}({\bm{h}}_{i}^{t},{\bm{h}}_{i}^{t+1})\leq\frac{(X+Y)^{2}}{2H_{\min}^{2}}\leq\frac{X^{2}+2XY+Y^{2}}{2H_{\min}^{2}}. (14) Substituting X^{2}=2\lambda^{2}V_{\max}^{2}(1-\mathcal{S}_{{\bm{v}}_{i}}) : 1-\mathcal{S}_{\cos}({\bm{h}}_{i}^{t},{\bm{h}}_{i}^{t+1})\leq\underbrace{\frac{\lambda^{2}V_{\max}^{2}}{H_{\min}^{2}}}_{C}(1-\mathcal{S}_{\cos}({\bm{v}}_{i}^{t},{\bm{v}}_{i}^{t+1}))+\underbrace{\frac{2XY+Y^{2}}{2H_{\min}^{2}}}_{\epsilon}. (15) This concludes the proof. The output dissimilarity is linearly bounded by the input dissimilarity of token i , scaled by the squared condition number C=(\lambda V_{\max}/H_{\min})^{2} . ∎ A.2 Proof of Theorem 3.2 Proof. A standard Transformer MLP consists of RMSNorm operations, several linear projections, activation functions \sigma(\cdot) . Each of these components is Lipschitz continuous ( 21 ) : • Linear transformations ( {\bm{W}} ) are Lipschitz with constant equal to their spectral norm \|{\bm{W}}\| . • Standard activations (ReLU, GELU, SiLU) are Lipschitz continuous with constant L_{\mathrm{act}}\approx 1 . • RMSNorm is Lipschitz continuous over a domain of inputs bounded away from zero. The composition of Lipschitz functions is Lipschitz. Thus, there exists a constant L such that for any inputs {\bm{h}}_{1},{\bm{h}}_{2} : \|f_{\mathrm{MLP}}({\bm{h}}_{1})-f_{\mathrm{MLP}}({\bm{h}}_{2})\|\leq L\|{\bm{h}}_{1}-{\bm{h}}_{2}\|. (16) Consider the squared Euclidean distance between the inputs {\bm{h}}_{1} and {\bm{h}}_{2} : \|{\bm{h}}_{1}-{\bm{h}}_{2}\|^{2}=\|{\bm{h}}_{1}\|^{2}+\|{\bm{h}}_{2}\|^{2}-2\|{\bm{h}}_{1}\|\|{\bm{h}}_{2}\|\mathcal{S}_{\cos}({\bm{h}}_{1},{\bm{h}}_{2}). (17) Assuming the hidden states within the Transformer operate within a bounded norm range ( 33 ; 23 ) , let H_{\min}\leq\|{\bm{h}}_{\cdot}\|\leq H_{\max} . The Euclidean distance can be exactly related to cosine similarity via: \|{\bm{h}}_{1}-{\bm{h}}_{2}\|^{2}=(\|{\bm{h}}_{1}\|-\|{\bm{h}}_{2}\|)^{2}+2\|{\bm{h}}_{1}\|\|{\bm{h}}_{2}\|(1-\mathcal{S}_{\cos}({\bm{h}}_{1},{\bm{h}}_{2})). (18) Bounding the magnitude difference by \Delta=H_{\max}-H_{\min} and the product by H_{\max}^{2} , we obtain a strict upper bound: \|{\bm{h}}_{1}-{\bm{h}}_{2}\|^{2}\leq\Delta^{2}+2H_{\max}^{2}(1-\mathcal{S}_{\cos}({\bm{h}}_{1},{\bm{h}}_{2})). (19) Using the subadditivity of the square root ( \sqrt{a+b}\leq\sqrt{a}+\sqrt{b} ): \|{\bm{h}}_{1}-{\bm{h}}_{2}\|\leq\Delta+\sqrt{2}H_{\max}\cdot\sqrt{1-\mathcal{S}_{\cos}({\bm{h}}_{1},{\bm{h}}_{2})}. (20) Substituting Equation 20 into the Lipschitz condition ( Equation 16 ), we set C=L\cdot\sqrt{2}H_{\max} and \epsilon=L\cdot\Delta : \|f_{\mathrm{MLP}}({\bm{h}}_{1})-f_{\mathrm{MLP}}({\bm{h}}_{2})\|\leq\epsilon+C\cdot\sqrt{1-\mathcal{S}_{\cos}({\bm{h}}_{1},{\bm{h}}_{2})}. (21) ∎ A.3 Proof of Theorem 3.4 A.3.1 Propositions and Lemma for the proof Proposition A.1 . (Generalized Bernoulli Theorem) For x\geq 0 , 1/\sqrt{1+x}\geq 1-x/2 . Proof. The proof is straightforward by Taylor expansion of 1/\sqrt{1+x} at x=0 : \frac{1}{\sqrt{1+x}}=1-\frac{1}{2}x+\underbrace{\frac{3}{8}x^{2}-\frac{5}{16}x^{3}+\ldots}_{\geq 0}\geq 1-\frac{1}{2}x. (22) ∎ Proposition A.2 . For any {\bm{h}}\in\mathbb{R}^{d} and the r -rank approximation {\bm{U}}_{r}\Lambda_{r}{\bm{V}}_{r}^{\top} of {\bm{W}} , the following inequality holds: \frac{\|{\bm{W}}{\bm{h}}-{\bm{U}}_{r}\Lambda_{r}{\bm{V}}_{r}^{\top}{\bm{h}}\|}{\|\Lambda_{r}{\bm{V}}_{r}^{\top}{\bm{h}}\|}\leq\frac{\lambda_{r+1}}{\lambda_{r}}\frac{\|{\bm{h}}\|}{\|{\bm{V}}_{r}^{\top}{\bm{h}}\|}. (23) Proof. For the numerator, we have \|{\bm{W}}{\bm{h}}-{\bm{U}}_{r}\Lambda_{r}{\bm{V}}_{r}^{\top}{\bm{h}}\|\leq\|{\bm{W}}-{\bm{U}}_{r}\Lambda_{r}{\bm{V}}_{r}^{\top}\|\|{\bm{h}}\|=\lambda_{r+1}\|{\bm{h}}\|. (24) For the denominator, we have \|\Lambda_{r}{\bm{V}}_{r}^{\top}{\bm{h}}\|\geq\lambda_{r}\|{\bm{V}}_{r}^{\top}{\bm{h}}\|. (25) Combining these two inequalities, \frac{\|{\bm{W}}{\bm{h}}-{\bm{U}}_{r}\Lambda_{r}{\bm{V}}_{r}^{\top}{\bm{h}}\|}{\|\Lambda_{r}{\bm{V}}_{r}^{\top}{\bm{h}}\|}\leq\frac{\lambda_{r+1}}{\lambda_{r}}\frac{\|{\bm{h}}\|}{\|{\bm{V}}_{r}^{\top}{\bm{h}}\|}. (26) ∎ Lemma A.3 . Let {\bm{W}}_{r}=\Lambda_{r}{\bm{V}}_{r}^{\top} , {\bm{o}}={\bm{W}}{\bm{h}} , and {\bm{o}}^{\prime}={\bm{W}}_{r}{\bm{h}} . Then the divergence of the cosine similarity between the full and low-dimensional projections can be bounded as follows: |\mathcal{S}_{\cos}({\bm{o}}_{1},{\bm{o}}_{2})-\mathcal{S}_{\cos}({\bm{o}}^{\prime}_{1},{\bm{o}}_{2}^{\prime})|\leq\frac{1}{2}\left(\frac{\lambda_{r+1}}{\lambda_{r}}\right)^{2}\left(\frac{\|{\bm{h}}_{1}\|}{\|{\bm{V}}_{r}^{\top}{\bm{h}}_{1}\|}+\frac{\|{\bm{h}}_{2}\|}{\|{\bm{V}}_{r}^{\top}{\bm{h}}_{2}\|}\right)^{2}. (27) Proof. We start with residuals of the low-rank approximation {\bm{R}}\coloneq{\bm{W}}-{\bm{U}}_{r}\Lambda_{r}{\bm{V}}_{r}^{\top} . By the definition of spectral norm, we have \|{\bm{R}}\|=\lambda_{r+1} . And {\bm{R}} and {\bm{U}}_{r} are orthogonal (mathematically, {\bm{R}}^{\top}{\bm{U}}_{r}=0 ) because the residual {\bm{R}} lives in the span of the discarded singular vectors \mathrm{span}({\bm{U}})\setminus\mathrm{span}({\bm{U}}_{r}) . Denote {\bm{A}}\coloneq{\bm{W}}-{\bm{R}}={\bm{U}}_{r}\Lambda_{r}{\bm{V}}_{r}^{\top}={\bm{U}}_{r}{\bm{W}}_{r} . As the orthogonal matrix {\bm{U}}_{r} preserves the inner product and norm, \mathcal{S}_{\cos}({\bm{o}}^{\prime}_{1},{\bm{o}}_{2}^{\prime})=\mathcal{S}_{\cos}({\bm{W}}_{r}{\bm{h}}_{1},{\bm{W}}_{r}{\bm{h}}_{2})=\mathcal{S}_{\cos}({\bm{A}}{\bm{h}}_{1},{\bm{A}}{\bm{h}}_{2}) , we use {\bm{A}} rather than {\bm{W}}_{r} in the following proof for clarity. Since {\bm{R}} and {\bm{A}} are orthogonal: {\bm{R}}^{\top}{\bm{A}}={\bm{R}}^{\top}{\bm{U}}_{r}\Lambda_{r}{\bm{V}}_{r}^{\top}=0\cdot\Lambda_{r}{\bm{V}}_{r}^{\top}=0 , the norm of {\bm{W}}{\bm{h}}_{1} can be decomposed as follows: \|{\bm{o}}_{1}\|^{2}=\|{\bm{W}}{\bm{h}}_{1}\|^{2}=\|{\bm{A}}{\bm{h}}_{1}+{\bm{R}}{\bm{h}}_{1}\|^{2}=\|{\bm{A}}{\bm{h}}_{1}\|^{2}+\|{\bm{R}}{\bm{h}}_{1}\|^{2}+\underbrace{2\langle{\bm{A}}{\bm{h}}_{1},{\bm{R}}{\bm{h}}_{1}\rangle}_{=0\ \text{as}\ {\bm{R}}^{\top}{\bm{A}}=0}. (28) Similarly, the inner products between the adjacent features can be decomposed as follows: \displaystyle\langle{\bm{o}}_{1},{\bm{o}}_{2}\rangle \displaystyle=\langle{\bm{W}}{\bm{h}}_{1},{\bm{W}}{\bm{h}}_{2}\rangle=\langle({\bm{A}}+{\bm{R}}){\bm{h}}_{1},({\bm{A}}+{\bm{R}}){\bm{h}}_{2}\rangle \displaystyle=\langle{\bm{A}}{\bm{h}}_{1},{\bm{A}}{\bm{h}}_{2}\rangle+\langle{\bm{R}}{\bm{h}}_{1},{\bm{R}}{\bm{h}}_{2}\rangle+\underbrace{\langle{\bm{A}}{\bm{h}}_{1},{\bm{R}}{\bm{h}}_{2}\rangle+\langle{\bm{R}}{\bm{h}}_{1},{\bm{A}}{\bm{h}}_{2}\rangle}_{=0\ \text{as}\ {\bm{R}}^{\top}{\bm{A}}=0}. (29) Denote a_{1}\coloneq\|{\bm{A}}{\bm{h}}_{1}\| , h_{1}\coloneq\|{\bm{W}}{\bm{h}}_{1}\| , and r_{1}\coloneq\|{\bm{R}}{\bm{h}}_{1}\| for brevity, \displaystyle\mathcal{S}_{\cos}({\bm{o}}_{1},{\bm{o}}_{2})-\mathcal{S}_{\cos}({\bm{o}}^{\prime}_{1},{\bm{o}}_{2}^{\prime}) \displaystyle=\frac{\langle{\bm{W}}{\bm{h}}_{1},{\bm{W}}{\bm{h}}_{2}\rangle}{\|{\bm{W}}{\bm{h}}_{1}\|\|{\bm{W}}{\bm{h}}_{2}\|}-\frac{\langle{\bm{A}}{\bm{h}}_{1},{\bm{A}}{\bm{h}}_{2}\rangle}{\|{\bm{A}}{\bm{h}}_{1}\|\|{\bm{A}}{\bm{h}}_{2}\|} \displaystyle=\frac{\langle{\bm{A}}{\bm{h}}_{1},{\bm{A}}{\bm{h}}_{2}\rangle+\langle{\bm{R}}{\bm{h}}_{1},{\bm{R}}{\bm{h}}_{2}\rangle}{\|{\bm{A}}{\bm{h}}_{1}\|\|{\bm{A}}{\bm{h}}_{2}\|}\cdot\frac{\|{\bm{A}}{\bm{h}}_{1}\|\|{\bm{A}}{\bm{h}}_{2}\|}{\|{\bm{W}}{\bm{h}}_{1}\|\|{\bm{W}}{\bm{h}}_{2}\|}-\frac{\langle{\bm{A}}{\bm{h}}_{1},{\bm{A}}{\bm{h}}_{2}\rangle}{\|{\bm{A}}{\bm{h}}_{1}\|\|{\bm{A}}{\bm{h}}_{2}\|} \displaystyle\mathllap{\text{{\color[rgb]{0.3906,0.5625,0.3086} [by \lx@cref{creftype\lx@tilde refnum}{eq: inner-prod}]}}} \displaystyle=\underbrace{\frac{\langle{\bm{A}}{\bm{h}}_{1},{\bm{A}}{\bm{h}}_{2}\rangle}{\|{\bm{A}}{\bm{h}}_{1}\|\|{\bm{A}}{\bm{h}}_{2}\|}}_{=\mathcal{S}_{\cos}({\bm{o}}^{\prime}_{1},{\bm{o}}_{2}^{\prime})\in[-1,1]}\underbrace{\left(\frac{\|{\bm{A}}{\bm{h}}_{1}\|\|{\bm{A}}{\bm{h}}_{2}\|}{\|{\bm{W}}{\bm{h}}_{1}\|\|{\bm{W}}{\bm{h}}_{2}\|}-1\right)}_{=\frac{a_{1}a_{2}}{h_{1}h_{2}}-1}+\frac{\langle{\bm{R}}{\bm{h}}_{1},{\bm{R}}{\bm{h}}_{2}\rangle}{\|{\bm{W}}{\bm{h}}_{1}\|\|{\bm{W}}{\bm{h}}_{2}\|}. (30) For the absolute value of the first term, we have \left|\mathcal{S}_{\cos}({\bm{o}}^{\prime}_{1},{\bm{o}}_{2}^{\prime})\right|\left|\frac{a_{1}a_{2}}{h_{1}h_{2}}-1\right|\leq\frac{1}{2}\left(\frac{r_{1}}{a_{1}}\right)^{2}+\frac{1}{2}\left(\frac{r_{2}}{a_{2}}\right)^{2}. (31) Here the inequality is derivated from Proposition A.1 by appling x=\left(\frac{r_{1}}{a_{1}}\right)^{2} : \frac{a_{1}}{h_{1}}=\frac{1}{\sqrt{1+\left(\frac{r_{1}}{a_{1}}\right)^{2}}}\geq 1-\frac{1}{2}\left(\frac{r_{1}}{a_{1}}\right)^{2}\text{ and similarly, }\frac{a_{2}}{h_{2}}\geq 1-\frac{1}{2}\left(\frac{r_{2}}{a_{2}}\right)^{2}, (32) \frac{a_{1}a_{2}}{h_{1}h_{2}}\geq 1-\frac{1}{2}\left(\frac{r_{1}}{a_{1}}\right)^{2}-\frac{1}{2}\left(\frac{r_{2}}{a_{2}}\right)^{2}+\frac{1}{4}\left(\frac{r_{1}r_{2}}{a_{1}a_{2}}\right)^{2}\geq 1-\frac{1}{2}\left(\frac{r_{1}}{a_{1}}\right)^{2}-\frac{1}{2}\left(\frac{r_{2}}{a_{2}}\right)^{2}. (33) So, \left|\frac{a_{1}a_{2}}{h_{1}h_{2}}-1\right|=1-\frac{a_{1}a_{2}}{h_{1}h_{2}}\leq\frac{1}{2}\left(\frac{r_{1}}{a_{1}}\right)^{2}+\frac{1}{2}\left(\frac{r_{2}}{a_{2}}\right)^{2}. (34) For the absolute value of the second term, we can use the Cauchy-Schwarz inequality: \left|\frac{\langle{\bm{R}}{\bm{h}}_{1},{\bm{R}}{\bm{h}}_{2}\rangle}{\|{\bm{W}}{\bm{h}}_{1}\|\|{\bm{W}}{\bm{h}}_{2}\|}\right|\leq\frac{\|{\bm{R}}{\bm{h}}_{1}\|\|{\bm{R}}{\bm{h}}_{2}\|}{\|{\bm{W}}{\bm{h}}_{1}\|\|{\bm{W}}{\bm{h}}_{2}\|}=\frac{r_{1}r_{2}}{h_{1}h_{2}}\leq\frac{r_{1}r_{2}}{a_{1}a_{2}} (35) Combining Equations 30 , 31 and 35 , we have |\mathcal{S}_{\cos}({\bm{o}}_{1},{\bm{o}}_{2})-\mathcal{S}_{\cos}({\bm{o}}^{\prime}_{1},{\bm{o}}_{2}^{\prime})|\leq\frac{1}{2}\left(\frac{r_{1}}{a_{1}}\right)^{2}+\frac{1}{2}\left(\frac{r_{2}}{a_{2}}\right)^{2}+\frac{r_{1}r_{2}}{a_{1}a_{2}}=\frac{1}{2}\left(\frac{r_{1}}{a_{1}}+\frac{r_{2}}{a_{2}}\right)^{2}. (36) By Proposition A.2 , \frac{r_{1}}{a_{1}}\leq\frac{\lambda_{r+1}}{\lambda_{r}}\frac{\|{\bm{h}}_{1}\|}{\|{\bm{V}}_{r}^{\top}{\bm{h}}_{1}\|},\ \text{and}\ \frac{r_{2}}{a_{2}}\leq\frac{\lambda_{r+1}}{\lambda_{r}}\frac{\|{\bm{h}}_{2}\|}{\|{\bm{V}}_{r}^{\top}{\bm{h}}_{2}\|}. (37) Substituting Equation 37 into Equation 36 , the final bound is |\mathcal{S}_{\cos}({\bm{o}}_{1},{\bm{o}}_{2})-\mathcal{S}_{\cos}({\bm{o}}^{\prime}_{1},{\bm{o}}_{2}^{\prime})|\leq\frac{1}{2}\left(\frac{\lambda_{r+1}}{\lambda_{r}}\right)^{2}\left(\frac{\|{\bm{h}}_{1}\|}{\|{\bm{V}}_{r}^{\top}{\bm{h}}_{1}\|}+\frac{\|{\bm{h}}_{2}\|}{\|{\bm{V}}_{r}^{\top}{\bm{h}}_{2}\|}\right)^{2}. (38) ∎ A.3.2 Proof of Theorem 3.4 Proof. With Lemma A.3 , it suffices to show that for any {\bm{h}}_{1},{\bm{h}}_{2}\in\mathrm{span}({\bm{V}}_{r}) , we have \|{\bm{h}}_{1}\|=\|{\bm{V}}_{r}^{\top}{\bm{h}}_{1}\| and \|{\bm{h}}_{2}\|=\|{\bm{V}}_{r}^{\top}{\bm{h}}_{2}\| , which establishes the conclusion of Theorem 3.4 . This is straightforward because {\bm{V}}_{r} is an orthonormal basis of \mathrm{span}({\bm{V}}_{r}) , so for any {\bm{h}}\in\mathrm{span}({\bm{V}}_{r}) , we have \|{\bm{h}}\|=\|{\bm{V}}_{r}^{\top}{\bm{h}}\| . ∎ Remark A.4 . (Connection with MaxLogits) It is widely observed that the spectral norm of the attention projection matrix, \|{\bm{W}}\|_{2}=\lambda_{1} , tends to become large during the training of large language models. While this behavior can lead to numerical instability ( 36 ) , it also incidentally benifits the error bound of our proposed singular proxy identifier. Appendix B Failure Analysis As discussed in Section 3.2 , using the attention output as the update identifier has the worst accuracy among all configurations in Table 1 . Intuitively, the attention output should be eligbile. We then dive deeper to investigate its failure mechanism. The core issue lies in that the anisotropy of the attention outputs have been masked, reducing the sensitivity of the similarity metric to subtle semantic drifts. Figure 5 : Empirical Evidence of Anisotropy Problem in Attention Outputs. The density plots compare the cosine similarity distributions of value states (blue) versus attention outputs (orange). While value states exhibit an isotropic distribution centered near zero (orthogonality), the attention outputs show a significant positive shift in mean similarity. Specifically, the anisotropy masks individual token characteristics, thereby reducing the sensitivity of the similarity metric to subtle semantic drifts ( 14 ; 15 ; 41 ) . It is widely observed that hidden states in Transformers, specifically the Value states {\bm{v}}_{i} in our context, can be decomposed into a common vector {\bm{c}} and an independent semantic signal {\bm{s}}_{i} : {\bm{v}}_{i}={\bm{c}}+{\bm{s}}_{i},\quad\text{where}\quad\mathbb{E}[{\bm{s}}_{i}]=\mathbf{0}\text{ and }\|{\bm{c}}\|>0. (39) While this phenomenon is present, the common vector {\bm{c}} remains relatively non-dominant ( {\|{\bm{s}}_{i}\|>\|{\bm{c}}\|} ), allowing the states to maintain sufficient orthogonality ( \mathbb{E}_{i\neq j}[\mathcal{S}_{\cos}({\bm{v}}_{i},{\bm{v}}_{j})]\approx 0 ). This preservation of semantic variance is visualized in Figure 5 . However, this representation collapses when states are aggregated into the attention output {\bm{h}}_{i}=\sum_{j}\alpha_{ij}{\bm{v}}_{j} . Given the stochastic constraint \sum_{j}\alpha_{ij}=1 , the transformation is expressed as: {\bm{h}}_{i}=\sum_{j}\alpha_{ij}({\bm{c}}+{\bm{s}}_{j})={\bm{c}}+\sum_{j}\alpha_{ij}{\bm{s}}_{j}. (40) Here, the attention output {\bm{h}}_{i} becomes heavily dominated by the common direction {\bm{c}} . Because the semantic signals {\bm{s}}_{j} are independent and zero-mean, their weighted sum undergoes a form of “signal cancellation,” leading to \|{\bm{c}}\|\gg\|\sum_{j}\alpha_{ij}{\bm{s}}_{j}\| . Consequently, the similarity distribution \mathcal{S}_{\cos}({\bm{h}}_{i},{\bm{h}}_{j}) shifts toward a narrow cone where cosine similarities approach 1. This anisotropy renders the identifier insensitive to the subtle semantic drifts required for accurate update tracking. Appendix C Details on Adaptive Budget Allocation Figure 6 : Distribution of drift for LLaDA-1.5 ( 54 ) and Dream-v0-Instruct-7B ( 48 ) . Table 6 : Hyperparameters for fitted piecewise Gaussian function. Model l_{p} \rho_{p} \rho_{1} \rho_{L} LLaDA-8B-Instruct 24 25\% 3\% 13\% LLaDA-1.5 25 25\% 3\% 13\% Dream-v0-Instruct-7B 14 30\% 5\% 25\% In Section 3.4 , we characterized the layer-wise heterogeneity of hidden state dynamics for LLaDA-8B ( 30 ) . In this section, we extend our analysis to a broader range of models. Specifically, the distribution profiles for LLaDA-1.5B ( 54 ) and Dream-7B ( 48 ) are illustrated in Figure 6 . While these models exhibit a general trend consistent with LLaDA-8B, the parameters for the fitted piecewise Gaussian distributions vary across architectures. We provide the specific hyperparameters employed for these models in Table 6 . It is important to emphasize that these hyperparameters remain fixed across all evaluated benchmarks, demonstrating the robust generalization capabilities of our proposed method. Appendix D Experimental Details Table 7 : Experiment settings across benchmarks. Task LLaDA-8B-Instruct & LLaDA-1.5 Dream-v0-Instruct-7B N-shot Gen. Length Block Length N-shot Gen. Length Block Length GSM8K 4 256 8 8 256 256 GPQA 5 128 64 5 256 256 MATH500 4 256 32 4 256 256 BBH 3 256 256 3 256 256 MMLU-pro 5 256 256 5 256 256 MBPP 3 512 32 3 256 256 HumanEval 0 512 32 0 256 256 In our experiments, we follow the decoding hyperparameters established by 28 . For completeness, these detailed configurations are provided in Table 7 . For the Fast-dLLM baseline ( 46 ) , which requires the block size to be smaller than the generation length to achieve effective acceleration, we set the block size to 32 uniformly across all models and benchmarks. Other method-specific hyperparameters, such as the refresh interval in dLLM-Cache ( 28 ) , are kept consistent with their original configurations. Regarding batch sizes, we use a batch size of 16 for all LLaDA-8B tasks. For the Dream-7B model, the batch size is set to 4. This specific reduction was necessitated by Out-of-Memory (OOM) issues encountered with Fast-dLLM during long-prompt benchmarks ( \eg MMLU-pro). While other evaluated methods could support larger batch sizes for higher throughput, we maintain a consistent batch size of 4 for Dream-7B to ensure a fair comparison across all baselines. Regarding the selection of the proxy rank r , we set r=128 for the LLaDA models ( 30 ; 54 ) , where the hidden dimension is d=4096 . This choice is consistent with our ablation studies detailed in Section 4.4 . For the Dream model ( 48 ) , which utilizes Grouped-Query Attention (GQA) ( 1 ) , the Value projection dimension is significantly smaller ( d=512 ). Consequently, we adopt a more aggressive proxy rank of r=32 to maintain a comparable acceleration. Appendix E Additional Experiment Results Table 8 : Comparision of different caching methods on LLaDA-1.5 ( 54 ) . Task Method TPS \uparrow TTFT (ms) \downarrow Accuracy \uparrow Mem. (GB) \downarrow Mathematics & Science Baseline 28.9\ _{(\times 1.0)} 28.04 81.96\ _{(\pm 1.1)} 62.3 + dLLM-Cache 68.5\ _{(\times 2.4)} 28.03 81.43\ _{(\pm 1.1)} 63.8 + Fast-dLLM 58.0\ _{(\times 2.0)} 28.09 82.18\ _{(\pm 1.1)} 64.1 GSM8K + Ours 189.9\ _{(\times 6.6)} 31.13 81.50\ _{(\pm 1.1)} 57.5 Baseline 21.0\ _{(\times 1.0)} 44.90 27.46\ _{(\pm 2.1)} 102.4 + dLLM-Cache 29.2\ _{(\times 1.4)} 44.72 27.90\ _{(\pm 2.1)} 77.0 + Fast-dLLM 39.4\ _{(\times 1.9)} 44.26 27.01\ _{(\pm 2.1)} 91.4 GPQA + Ours 49.2\ _{(\times 2.3)} 47.68 27.90\ _{(\pm 2.1)} 96.6 Baseline 35.5\ _{(\times 1.0)} 23.23 34.40\ _{(\pm 0.6)} 63.4 + dLLM-Cache 74.2\ _{(\times 2.1)} 23.28 33.92\ _{(\pm 0.6)} 57.9 + Fast-dLLM 83.9\ _{(\times 2.4)} 23.19 33.54\ _{(\pm 0.6)} 66.5 MATH500 + Ours 201.8\ _{(\times 5.7)} 23.13 34.56\ _{(\pm 0.6)} 51.6 General QA Baseline 22.3\ _{(\times 1.0)} 16.22 56.61\ _{(\pm 0.6)} 37.0 + dLLM-Cache 36.0\ _{(\times 1.6)} 16.35 57.00\ _{(\pm 0.6)} 45.0 + Fast-dLLM 103.9\ _{(\times 4.7)} 16.22 54.66\ _{(\pm 0.6)} 36.3 BBH + Ours 131.5\ _{(\times 5.9)} 16.27 57.43\ _{(\pm 0.6)} 34.3 Baseline 21.4\ _{(\times 1.0)} 43.99 38.51\ _{(\pm 0.4)} 95.3 + dLLM-Cache 51.7\ _{(\times 2.4)} 42.12 38.30\ _{(\pm 0.4)} 87.5 + Fast-dLLM 84.3\ _{(\times 4.0)} 43.41 38.31\ _{(\pm 0.4)} 79.3 MMLU-pro + Ours 233.2\ _{(\times 10.9)} 43.90 38.36\ _{(\pm 0.4)} 79.1 Code Baseline 6.7\ _{(\times 1.0)} 28.61 40.80 70.7 + dLLM-Cache 9.0\ _{(\times 1.3)} 28.22 40.80 74.2 + Fast-dLLM 13.3\ _{(\times 2.0)} 28.25 37.20 74.1 MBPP + Ours 69.4\ _{(\times 10.4)} 29.44 40.00 69.5 Baseline 46.8\ _{(\times 1.0)} 14.30 45.83 51.2 + dLLM-Cache 43.1\ _{(\times 0.9)} 14.28 38.41 55.7 + Fast-dLLM 92.8\ _{(\times 2.0)} 14.23 41.71 50.3 HumanEval + Ours 143.7\ _{(\times 3.1)} 14.23 46.88 48.9 We further evaluate our approach on LLaDA-1.5B ( 54 ) , with results detailed in Table 8 . The performance trends closely mirror those observed for the larger LLaDA-8B and Dream-7B models ( Table 2 ), underscoring the architectural robustness of our method. Notably, SPA-Cache consistently establishes a new state-of-the-art across all benchmarks, delivering substantial throughput gains while preserving the original model’s generation quality with negligible degradation.