HiveMind : OS-Inspired Scheduling for Concurrent LLM Agent Workloads
Justice Owusu Agyemang, Jerry John Kponyo, Obed Kwasi Somuah, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum · 2026
Read HiveMind : OS-Inspired Scheduling for Concurrent LLM Agent Workloads by Justice Owusu Agyemang, Jerry John Kponyo, Obed Kwasi Somuah, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17111.txt.
Introduction
Agyemang and five colleagues tackle crashes when many language-model coding agents share one rate-limited API endpoint. In one incident, 3 of 11 parallel agents died from connection resets and HTTP 502 errors, a 27 percent failure rate despite capacity to serve all 11 in sequence. Their HIVEMIND proxy applies operating-system-style scheduling, admission control, rate tracking, backpressure, token budgets, and priority queues, cutting failures to between 0 and 18 percent across seven test scenarios.
Catalog record
| CREATOR | Justice Owusu Agyemang, Jerry John Kponyo, Obed Kwasi Somuah, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34318 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17111
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HiveMind : OS-Inspired Scheduling for Concurrent LLM Agent Workloads Justice Owusu Agyemang, Jerry John Kponyo, Obed Kwasi Somuah, Elliot Amponsah, Godfred Manu Addo Boakye, Kwame Opuni-Boachie Obour Agyekum Abstract When multiple LLM coding agents share a rate-limited API endpoint, they exhibit resource contention patterns analogous to unscheduled OS processes competing for CPU, memory, and I/O. In a motivating incident, 3 of 11 parallel agents died from connection resets and HTTP 502 errors—a 27% failure rate—despite the API having sufficient aggregate capacity to serve all 11 sequentially. We present HiveMind , a transparent HTTP proxy that applies five OS-inspired scheduling primitives—admission control, rate-limit tracking, AIMD backpressure with circuit breaking, token budget management, and priority queuing—to eliminate the failure modes caused by uncoordinated parallel execution. The proxy requires zero modifications to existing agent code and supports Anthropic, OpenAI, and local model APIs via auto-detected provider profiles. Our evaluation across seven scenarios (5–50 concurrent agents) shows that uncoordinated agents fail at 72–100% rates under contention, while HiveMind reduces failures to 0–18% and eliminates 48–100% of wasted compute. An ablation study reveals that transparent retry—not admission control—is the single most critical primitive, but the primitives are most effective in combination. Real-world validation against Ollama confirms that HiveMind adds under 3 ms of proxy overhead per request. The system is open-source under the MIT license. 1 Introduction The emergence of tool-augmented large language models has shifted software-engineering assistants from suggestion engines to autonomous agents that read, write, and execute code on a developer’s behalf [ 2 , 18 , 3 , 10 ] . When users spawn multiple such agents in parallel—a natural pattern for tasks like generating test suites, writing proof-of-concept exploits, or refactoring across modules—the agents compete for shared resources: API rate limits (requests and tokens per minute), network connections (concurrent connection limits per endpoint), context windows (fixed per model), and API-key quotas (billing and access limits). This resource contention leads to agent failures. The pattern is structurally identical to the contention that motivated operating-system schedulers: multiple processes competing for CPU, memory, and I/O without coordination leads to thrashing, starvation, and deadlock [ 19 , 6 ] . Yet current agent orchestration frameworks—LangChain [ 4 ] , CrewAI [ 15 ] , AutoGen [ 22 ] , Semantic Kernel [ 14 ] —treat the LLM API as an unlimited resource, providing composition mechanisms (chains, crews, multi-agent conversations) but not resource management. They are, in OS terms, running a multi-process system without a scheduler. Motivating observation. On April 15, 2026, we spawned 11 concurrent Claude Code agents to generate proof-of-concept scripts for security findings. All 11 shared one Anthropic API key through a single network proxy. Three agents died: two from ECONNRESET and one from HTTP 502. Each dead agent had consumed approximately 45 000 tokens before failing—a total waste of {\sim} 135 000 tokens and {\sim} 15 minutes of wall time. The eight surviving agents completed successfully because they happened to stagger their requests enough to avoid the bottleneck. Key insight: if the 11 agents had been staggered by just 5 seconds each, all 11 would have succeeded. The problem is not capacity—it is coordination. Table 1: Results of 11 uncoordinated concurrent agents (April 15, 2026). Outcome Count % Completed successfully 8 73 Died ( ECONNRESET ) 2 18 Died (HTTP 502) 1 9 Tokens wasted (dead agents) {\sim} 135 K Contribution. We present HiveMind , a scheduling system that applies OS scheduling principles to concurrent LLM agent workloads. The contributions are: 1. A formal analogy mapping OS resource-management concepts (admission control, congestion control, budgeting, priority scheduling) to the LLM agent domain (Table 2 ). 2. A transparent HTTP proxy that implements five scheduling primitives—admission control via condition variables, provider-aware rate-limit tracking, AIMD backpressure with circuit breaking, per-agent token budgets, and priority queuing with dependency DAGs—requiring zero modifications to existing agent code. 3. An evaluation across seven scenarios showing 72–100% failure reduction, and an ablation study revealing that transparent retry is the single most critical primitive. 4. An open-source implementation supporting Anthropic, OpenAI, Azure OpenAI, Google AI, and local models (Ollama, MLX) via auto-detected provider profiles. The remainder of this paper is organised as follows. Section 2 reviews the relevant background. Section 3 describes the proxy architecture and five scheduling primitives. Section 4 covers key implementation decisions. Section 5 reports evaluation results, ablation, and real-world validation. Section 6 surveys related work. Section 7 discusses tradeoffs and limitations, and Section 8 concludes. 2 Background 2.1 LLM Coding Agents A growing class of developer tools embed an LLM in an edit–test–commit loop. Claude Code [ 2 ] , Cursor [ 3 ] , GitHub Copilot [ 10 ] , OpenAI Codex CLI [ 18 ] , and Devin [ 7 ] each grant the model access to the local filesystem, a shell, and often a language server. The SWE-bench benchmark [ 13 ] and the SWE-agent framework [ 23 ] have further demonstrated that agents can resolve real GitHub issues end-to-end, making reliable API access a critical capability. Each agent is a long-running, stateful process that makes repeated API calls over a multi-turn conversation. A single agent session may consume 50 000–500 000 tokens across dozens of API calls, with each call dependent on the previous response. When an API call fails mid-session, the agent typically cannot recover: it has consumed tokens, modified files, and accumulated context that is lost on restart. 2.2 OS Scheduling Principles The resource contention patterns exhibited by concurrent LLM agents are structurally identical to those solved by operating system schedulers [ 19 , 20 ] : • Admission control. Limiting the number of concurrent processes to prevent thrashing. Dijkstra’s semaphore [ 9 ] is the classical mechanism. • Congestion control. TCP’s Additive Increase / Multiplicative Decrease (AIMD) algorithm [ 12 , 5 ] adjusts sending rate based on observed congestion signals (packet loss, increased RTT). • Circuit breaking. The circuit breaker pattern [ 16 ] stops sending requests to a failing service, allowing it to recover before resuming load. • Resource budgeting. Per-process memory limits, CPU quotas, and the OOM killer prevent any single process from monopolising shared resources. • Priority scheduling. Shortest-job-first and priority queues [ 19 ] ensure that high-value or short tasks are serviced before long or low-priority ones. 2.3 The OS–LLM Agent Analogy We formalise the mapping between OS concepts and LLM agent orchestration in Table 2 . This analogy is not merely illustrative—it is structurally precise. Each OS mechanism addresses a specific resource contention failure mode that has a direct counterpart in the LLM agent domain. Table 2: Structural mapping between OS resource management and LLM agent scheduling. Each row identifies an OS mechanism, its HiveMind counterpart, and the failure mode it addresses. OS Concept HiveMind Equivalent Resource Failure Mode Process LLM agent – Stateful, long-running, resource-consuming CPU time slice API request slot RPM/TPM Starvation under contention Memory Context window Fixed per model Cannot be shared or paged I/O bandwidth Network connections Conn. limits ECONNRESET , HTTP 502 Process scheduler Admission gate + queue Concurrency slots Thrashing, stampede Virtual memory Checkpointing Disk Context loss on eviction OOM killer Token budget enforcer Token pool Runaway agent monopolises API TCP congestion ctrl. AIMD backpressure Latency signal Throughput