ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM Serving Conference: 4th International Workshop on Testing Distributed Internet of Things Systems; April 27–30, 2026; Edinburgh, Scotland Uk 4th International Workshop on Testing Distributed Internet of Things Systems (TDIS ’26), April 27–30, 2026, Edinburgh, Scotland Uk DOI: 10.1145/3802513.3803483 ISBN: 979-8-4007-2608-8/26/04 CCS: Computing methodologies Distributed computing methodologies CCS: Computing methodologies Distributed artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computer systems organization Distributed architectures CCS: General and reference General conference proceedings

Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, Dimitrios S. Nikolopoulos · 2026

Read ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM Serving Conference: 4th International Workshop on Testing Distributed Internet of Things Systems; April 27–30, 2026; Edinburgh, Scotland Uk 4th International Workshop on Testing Distributed Internet of Things Systems (TDIS ’26), April 27–30, 2026, Edinburgh, Scotland Uk DOI: 10.1145/3802513.3803483 ISBN: 979-8-4007-2608-8/26/04 CCS: Computing methodologies Distributed computing methodologies CCS: Computing methodologies Distributed artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computer systems organization Distributed architectures CCS: General and reference General conference proceedings by Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, Dimitrios S. Nikolopoulos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09722.txt.

Introduction

Li and five co-authors present ConfigSpec, a method for choosing settings for edge-cloud speculative serving. Use means juggling draft-model variants, quantization levels, draft lengths, and edge devices. It maps devices and draft-target fit, scoring drafting throughput, accept rate, and power to rate goodput, costs, and energy use. Across three edge platforms and two model groups, optima conflict. Goodput favors the smallest, fastest drafter at lengths from 2 to 10, while cost and energy use both converge on length 2. Cost picks the largest drafter and energy the smallest.

Catalog record

CREATORXiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, Dimitrios S. Nikolopoulos
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30389
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09722

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge–Cloud Speculative LLM Serving Conference: 4th International Workshop on Testing Distributed Internet of Things Systems; April 27–30, 2026; Edinburgh, Scotland Uk 4th International Workshop on Testing Distributed Internet of Things Systems (TDIS ’26), April 27–30, 2026, Edinburgh, Scotland Uk DOI: 10.1145/3802513.3803483 ISBN: 979-8-4007-2608-8/26/04 CCS: Computing methodologies Distributed computing methodologies CCS: Computing methodologies Distributed artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computer systems organization Distributed architectures CCS: General and reference General conference proceedings Xiangchen Li, Saeid Ghafouri, Jiakun Fan, Babar Ali, Hans Vandierendonck, Dimitrios S. Nikolopoulos Abstract . Speculative decoding enables collaborative Large Language Model (LLM) inference across cloud and edge by separating lightweight token drafting from heavyweight verification. While prior systems show performance and cost benefits, practical deployment requires navigating a large configuration space spanning draft model variants, quantisation levels, speculative lengths, and heterogeneous edge devices. This paper presents ConfigSpec, a configuration-selection framework for distributed speculative LLM serving. ConfigSpec profiles edge devices and draft–target alignment, and models drafting throughput, acceptance rate, and power to evaluate goodput, verification cost efficiency, and energy efficiency across the joint configuration space. Our analysis across three edge platforms and two LLM families reveals structurally conflicting optima. Firstly, goodput is maximised by the smallest, fastest draft model at device-dependent speculative lengths ( K ∗ = 2 K^{*}{=}2 –10). Secondly, both cost and energy efficiency converge to K = 2 K{=}2 due to a dominant bonus-token effect—with cost favouring the largest drafter for its high acceptance rate and energy favouring the smallest for its low power draw . These conflicts confirm that no single fixed configuration can simultaneously optimise all objectives, underscoring the need for profiling-based configuration selection in disaggregated edge–cloud LLM inference. † † cc-license: by 1. Introduction Deploying Large Language Models (LLMs) closer to end users has become increasingly attractive due to the need to reduce end-to-end latency, limit bandwidth usage ( 14 ) , and preserve data locality and privacy ( 7 ) . Many interactive, streaming, and Internet of Things (IoT) applications in the fields of healthcare, industrial IoT, surveillance, etc., benefit from near-source inference execution, where responsiveness and data sensitivity are critical, enabling LLM inference on edge and mobile platforms ( 17 ) . For on-device inference, a wide range of model efficiency techniques have been explored, including quantization ( 15 ) , pruning ( 5 ) , and architectural simplification, which aim to reduce computation, memory footprint, and energy consumption on resource-constrained hardware. While these techniques can substantially lower inference cost and enable partial deployment of LLM models on edge devices, they face fundamental limits, as aggressive compression often leads to accuracy degradation and diminishing returns ( 2 ) . Consequently, fully executing high-capacity LLMs on edge devices alone remains impractical for many workloads, motivating hybrid cloud–edge execution models in which IoT devices offload sensory insights to edge, where edge devices perform lightweight or partial inference while delegating expensive computation to centralized infrastructure ( 9 ) . Speculative decoding is a decoding paradigm for autoregressive language models in which a lightweight draft model proposes multiple tokens ahead of time, and a higher-capacity target model subsequently verifies them against the target distribution. The target model accepts correct prefixes and discards the remaining, which eliminates divergence likelihood ( 8 ) . Crucially, speculative decoding preserves the output distribution of the target model despite using an auxiliary draft model ( 8 ; 1 ) . This process decouples generation into a drafting phase that prioritizes speed and a verification phase that enforces correctness ( 8 ; 12 ) . By enabling multiple tokens to be proposed and validated in a single step, speculative decoding exposes opportunities to overlap computation, amortize verification cost, and restructure inference across heterogeneous resources. Speculative decoding can substantially improve throughput, system capacity, and cost efficiency in heterogeneous edge environments ( 11 ; 10 ; 16 ) . By distributing generation across devices with different computational capabilities, these systems achieve better utilization of both edge and server resources while maintaining output quality. However, speculative decoding exposes a fundamental challenge in practice that is orthogonal to mechanism design, namely configuration selection. Designers must decide which draft model variants to deploy on what edge devices, and how to set the speculative decoding length, choices that directly impact performance, resource efficiency, monetary cost, and energy consumption across heterogeneous hardware. In realistic deployments, these configurations interact in non-trivial ways with hardware constraints, workload characteristics, and service-level objectives. Larger draft models typically improve token acceptance rates, but consume more compute and memory on the device, potentially limiting concurrency and degrading energy efficiency. No single configuration dominates across devices, workloads, and optimization objectives. Selecting configurations for distributed speculative LLM serving is challenging because system behavior depends on the interaction between draft model family, model size, quantization level, edge hardware, speculative decoding length, and target model alignment. These factors jointly determine verified-token throughput, verification cost efficiency, and energy efficiency, and their effects are often non-intuitive. For example, increasing draft model size improves acceptance rate but may reduce goodput and increase energy per verified token; similarly, platform differences can dominate model-size differences. As a result, configuration decisions cannot be inferred from model accuracy or hardware capability alone, and require systematic profiling and cross-metric evaluation. In this paper, we present ConfigSpec, a framework that combines systematic device profiling, draft–target alignment measurement, and analytical performance modeling to evaluate configuration quality. ConfigSpec characterizes each draft model and edge platform using measurable quantities—drafting throughput, acceptance rate against a target model, and device power draw—and analytically factors cloud verification effects into a set of parameters, mapping these profiles to deployment-relevant metrics including goodput, verification cost efficiency, and energy efficiency. By grounding configuration evaluation in measurable primitives while preserving the structural properties of speculative decoding, ConfigSpec enables fast, repeatable comparison of model families, quantization levels, and edge platforms prior to large-scale deployment. In summary, this paper makes the following contributions: • We identify configurations—specifically the joint choice of draft model, speculative decoding length, and edge platform—as a primary determinant of performance, cost, and energy in distributed edge–cloud speculative LLM serving. • We design ConfigSpec, a profiling-based configuration evaluation framework that measures drafting throughput, draft–target acceptance rate, and device power, and systematically maps them to system goodput, cost, and device energy . • Exhaustive enumeration of the joint (M,Q,K) configuration space reveals structurally conflicting optima: goodput favours smallest draft at device-dependent K^{*}{=}2 – 10 , while both cost and energy converge to K{=}2 via a bonus-token effect—yet cost selects the largest drafter and energy the smallest . These conflicts produce trade-offs of up to 2.9\times in goodput, 2.2\times in cost, and 7.8\times in energy between objective-optimal configurations on same device, confirming that no single fixed setting can simultaneously optimise all metrics. 2. Related Work Related works are categorized into three groups: speculative decoding algorithms, speculative decoding systems for edge and distributed serving, and edge–cloud collaboration for LLM inference. Speculative Decoding Algorithms. Speculative decoding was originally introduced as a lossless acceleration technique for autoregressive generation, enabling multiple tokens to be proposed in parallel by a lightweight draft model and verified by a larger target model. Leviathan et al. ( 8 ) formalize speculative decoding and prove that it preserves

Read the complete text