Local-Splitter: A Measurement Study of Seven Tactics for Reducing Cloud LLM Token Usage on Coding-Agent Workloads

Justice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum, [4pt] Sperix Labs VIA Cybersecurity Lab, KNUST Quantum and Assistive Technologies Lab, KNUST · 2026

Read Local-Splitter: A Measurement Study of Seven Tactics for Reducing Cloud LLM Token Usage on Coding-Agent Workloads by Justice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum, [4pt] Sperix Labs VIA Cybersecurity Lab, KNUST Quantum and Assistive Technologies Lab, KNUST (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12301.txt.

Introduction

Agyemang and four colleagues measure seven tactics for cutting cloud language-model token use when a small local model triages work before a frontier cloud model. Local routing plus prompt compression saves 45 to 79 percent of cloud tokens on edit-heavy and explanation-heavy coding work, while the full tactic set with draft-review saves 51 percent on retrieval-heavy work. The best subset depends on workload, the study's most actionable finding for teams deploying coding agents.

Catalog record

CREATORJustice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum, [4pt] Sperix Labs VIA Cybersecurity Lab, KNUST Quantum and Assistive Technologies Lab, KNUST
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35138
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12301

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Local-Splitter: A Measurement Study of Seven Tactics for Reducing Cloud LLM Token Usage on Coding-Agent Workloads Justice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum, [4pt] Sperix Labs VIA Cybersecurity Lab, KNUST Quantum and Assistive Technologies Lab, KNUST Abstract We present a systematic measurement study of seven tactics for reducing cloud LLM token usage when a small local model can act as a triage layer in front of a frontier cloud model. The tactics are: (1) local routing, (2) prompt compression, (3) semantic caching, (4) local drafting with cloud review, (5) minimal-diff edits, (6) structured intent extraction, and (7) batching with vendor prompt caching. We implement all seven in an open-source shim that speaks both MCP and the OpenAI-compatible HTTP surface, supporting any local model via Ollama and any cloud model via an OpenAI-compatible endpoint. We evaluate each tactic individually, in pairs, and in a greedy-additive subset across four coding-agent workload classes (edit-heavy, explanation-heavy, general chat, RAG-heavy). We measure tokens saved, dollar cost, latency, and routing accuracy. Our headline finding is that T1 (local routing) combined with T2 (prompt compression) achieves 45–79% cloud token savings on edit-heavy and explanation-heavy workloads, while on RAG-heavy workloads the full tactic set including T4 (draft-review) achieves 51% savings. We observe that the optimal tactic subset is workload-dependent, which we believe is the most actionable finding for practitioners deploying coding agents today. 1 Introduction Cloud LLM API costs have become a significant line-item in the operational budget of developer-tooling companies [ 7 ] . Coding agents [ 11 , 14 ] now routinely send tens of thousands of tokens to a frontier model per task, most of which are repetition, boilerplate, or context that could be cheaper to produce locally. Meanwhile, local-inference-ready small models of the 1–7 B parameter class have advanced to the point where they can correctly answer a meaningful fraction of developer queries on consumer hardware [ 9 , 15 , 1 ] . This paper asks: given a local small model and a frontier cloud model, what is the best way to split work between them to minimise total cloud token usage while keeping response quality near the baseline? We survey seven concrete tactics that each attack a distinct source of token waste, implement all seven, and evaluate them individually and in combination on four realistic workload classes. Our contributions are: • A taxonomy of seven orthogonal tactics for reducing cloud token usage via a local triage model (§ 3 ). • A reference implementation that is vendor-agnostic at both ends — any Ollama-compatible local model and any OpenAI-compatible cloud endpoint — and exposes both an MCP interface and an OpenAI-compatible HTTP proxy (§ 4 ). • An empirical evaluation measuring per-tactic and per-combination savings, latency, and quality deltas across four workload classes and multiple model pairs (§ 5 , § 6 ). • The observation that optimal tactic subset is workload-dependent , with concrete recommendations by workload class (§ 7 ). • A release of code, workloads, and evaluation harness for reproduction and extension. 2 Related Work Speculative decoding. Leviathan et al. [ 12 ] introduced speculative decoding as a token-level acceleration technique in which a small draft model proposes several tokens and a large verifier accepts or rejects them. Medusa [ 6 ] extends this with multiple decoding heads. Our tactic T4 adopts the same structural idea at the application layer: the local model drafts a complete response, and the cloud model is asked to review or patch it rather than generate from scratch. Prompt compression. LLMLingua [ 10 ] and its successor LLMLingua-2 [ 19 ] compress long input contexts by identifying and removing low-importance tokens. Our tactic T2 is a semantic-compression variant that runs the local model as a text-to-text compressor rather than a token-level filter. Semantic caching. GPTCache [ 5 ] caches responses keyed by embedding similarity. We replicate this as tactic T3 and extend it with rigorous quality measurement that the original did not report. Routing and cascades. FrugalGPT [ 7 ] and RouteLLM [ 17 ] present model-cascade architectures where requests are routed to progressively larger models based on confidence. Our tactic T1 is a single-stage local/cloud router, and we compare our empirical savings to those reported in the cascade literature. Vendor prompt caching. Anthropic’s cache_control: ephemeral [ 2 ] and OpenAI’s automatic prompt caching [ 18 ] enable substantial discounts when stable prefixes are re-sent. These are strictly cloud-side features; our tactic T7 integrates them into the splitter’s outbound requests. What we add. Individual tactics have been studied in isolation, often with different benchmarks. To our knowledge, this paper is the first to measure all seven on a common benchmark, to measure their pairwise interactions, and to quantify the workload-dependence of the optimal subset. 3 Tactics 3.1 T1 — Local routing A small local model reads each incoming request and emits a binary classification: trivial or complex . Trivial requests are answered by the local model directly and never reach the cloud. Complex requests flow to the next pipeline stage unchanged. This works because a substantial fraction of coding-agent traffic is structurally trivial: short completions, single-word renames, “what does this file do” queries, and boilerplate generation. A 3B-parameter local model gives acceptable answers for roughly a third to half of these requests; sending them to a frontier model is wasteful. The classifier uses a few-shot prompt that defines trivial as any request a junior engineer could answer in under ten seconds (short completion, rename, typo fix, lookup, restatement) and complex as anything requiring multi-step reasoning, ambiguous requirements, or multi-file refactoring. The classifier runs at temperature 0 with a 3-token output budget. On parse failure, the request defaults to complex . Risks. False positives (a trivial classification for a genuinely complex request) degrade answer quality. We mitigate this with a confidence margin: if the classifier’s logprob for trivial falls below a configurable threshold, the request is escalated to the cloud regardless. 3.2 T2 — Prompt compression Before a request reaches the cloud, the local model rewrites the context (system prompt, chat history, retrieved documents, file contents) to a shorter form that preserves semantic meaning. The cloud model then processes a trimmed prompt and is charged fewer input tokens. Context windows in agent prompts are large and often highly repetitive [ 4 ] . A typical coding-agent system prompt spans 3–8K tokens of boilerplate that can be summarised to roughly 400 tokens without losing the load-bearing instructions. We implement two compression modes. Static compression runs once at session start on the system prompt and caches the result. Dynamic compression runs per-call on the chat history and retrieved documents, which change each turn. The compression prompt instructs the local model to remove filler and repetition while preserving file paths, variable names, error messages, and numeric values verbatim. Risks. Information loss is the primary concern: a compressed prompt may drop a detail the cloud model would have needed. We benchmark against a held-out quality set and roll back compression if quality drops by more than a configurable threshold. 3.3 T3 — Semantic caching Every outbound request is embedded by a local embedding model, and responses are stored in a vector

Read the complete text