EAGLE-Pangu: Accelerator-Safe Tree Speculative Decoding on Ascend NPUs Chang Han, Yijie Hu, Jingling Liu Abstract Autoregressive decoding remains a primary bottleneck in large language model (LLM) serving, motivating speculative decoding methods that reduce expensive teacher-model invocations by verifying multiple candidate tokens per step. Tree-structured speculation further increases parallelism, but is often brittle when ported across heterogeneous backends and accelerator stacks, where attention masking, KV-cache layouts, and indexing semantics are not interchangeable. We present EAGLE-Pangu , a reproducible system that ports EAGLE-3-style tree speculative decoding to a Pangu teacher backend on Ascend NPUs. EAGLE-Pangu contributes (i) an explicit branch/commit cache manager built on the Cache API, (ii) accelerator-safe tree tensorization that removes undefined negative indices by construction and validates structural invariants, and (iii) a fused-kernel-compatible teacher verification path with a debuggable eager fallback. On 240 turns from MT-Bench and HumanEval-style prompts, EAGLE-Pangu improves end-to-end decoding throughput by 1.27 × 1.27\times on average (up to 2.46 × 2.46\times at p99) over teacher-only greedy decoding in the fused-kernel performance path. We also provide a fused-kernel-free reference path with structured traces and invariant checks to support reproducible debugging and ablation across execution modes and tree budgets. 1 Introduction The dominant cost in LLM serving arises from auto-regressive decoding, where each new token requires an additional forward pass through a large “teacher” model. This sequential dependency fundamentally limits throughput and increases latency under realistic deployment constraints (e.g., long contexts, high concurrency, and strict tail-latency budgets). Speculative decoding mitigates this bottleneck by using a smaller “draft” model to propose candidate tokens that are then verified by the teacher, thereby reducing the number of teacher steps required for a fixed-length generation [ 3 ] . Tree-structured speculative decoding further generalizes this idea by validating multiple candidate continuations per step, increasing parallelism and potentially improving throughput beyond linear draft proposals [ 4 , 1 ] . Despite its conceptual simplicity, tree speculative decoding is often fragile in practice. Porting a tree-decoding implementation across model backends and accelerator stacks is not merely an exercise in “re-implementing kernels”: (i) the teacher model may expose non-standard KV-cache layouts and specialized attention masking interfaces, especially when supporting tree-mode execution; (ii) fused attention kernels impose stricter requirements on mask shapes, alignment, and boundary conditions than eager implementations; and (iii) indexing and gather semantics can differ across device runtimes, where negative or out-of-bounds indices may be unsupported, inconsistently handled, or silently miscomputed. In tree decoding, such discrepancies are amplified because candidate evaluation relies on structured indexing (e.g., retrieving node states along many speculative paths) and on precise masking to prevent cross-branch information leakage. Consequently, a naive port may exhibit degraded quality, sporadic failures, or irreproducible performance, even if it “runs.” In this work, we focus on making tree speculative decoding portable, correct, and reproducible on a Pangu teacher backend deployed on Ascend NPUs. We follow the EAGLE-3-style tree speculative decoding framework and its posterior acceptance rule [ 4 ] , and we do not claim a new decoding algorithm. Instead, we identify the key system-level failure modes that arise in this setting and propose a set of design principles and abstractions that preserve the intended decoding semantics while satisfying accelerator constraints. Our evaluation emphasizes (a) end-to-end decoding throughput, (b) answer quality under matched decoding configurations, and (c) stability and debuggability via structured traces and execution-mode controls. Concretely, we make the following contributions: • Branchable KV-cache abstraction. We introduce a cache interface that separates accepted-prefix state from per-branch speculative state, enabling correct cache cloning and acceptance updates while decoupling tree decoding from backend-specific KV representations. • Accelerator-safe tree tensor semantics. We propose a correctness-preserving indexing scheme that replaces undefined negative padding indices with a valid dummy index and applies a safe mapping prior to gather operations, complemented by invariant checks to detect shape and boundary violations. • Tree-masked teacher execution with fused attention. We formalize and integrate 4D tree attention masks into the teacher execution path, supporting fused attention kernels for throughput while retaining an eager fallback for verification and debugging. • Reproducible distributed pipeline (supporting). We provide a distributed preprocessing and caching mechanism that avoids redundant work and mitigates synchronization failures in large-scale runs. • Data-driven vocabulary subset mapping (supporting). We implement a reusable draft-vocabulary subset construction and caching workflow to support controlled speed–quality trade-offs in draft modeling. We benchmark EAGLE-Pangu on MT-Bench-style conversational prompts using consistent prompting templates and matched decoding settings for fair comparison. We report tokens-per-second and end-to-end wall-clock speedups relative to standard greedy generation, and we assess quality using the corresponding benchmark scoring protocol. To isolate the impact of each design component, we conduct targeted ablations over (i) cache management strategies, (ii) safe indexing and invariant enforcement, and (iii) fused versus eager attention execution, alongside analyses of tree hyperparameters and draft-vocabulary subset sizes. The remainder of the paper is organized as follows: Section 2 reviews speculative decoding and tree verification semantics; Section 3 presents our cache abstraction, accelerator-safe indexing, and fused teacher verification; Section 4 describes the implementation and evaluation harness; Section 5 reports experimental results and analyses; finally, Section 6 concludes and discusses limitations. 2 Background This section reviews speculative decoding and its tree-structured variants, and establishes the notation used throughout the paper. We also highlight the execution-level semantics—KV caching, attention masking, and tensor indexing—that become first-order concerns when porting tree decoding across heterogeneous backends and accelerator stacks. 2.1 Auto-regressive decoding and KV caching Let x=(x_{1},\ldots,x_{T}) denote a token sequence. An auto-regressive LLM defines a distribution p_{\theta}(x)\;=\;\prod_{t=1}^{T}p_{\theta}(x_{t}\mid x_{