Personal AI, On Personal Devices

Jon Saad-Falcon, Avanika Narayan, Robby Manihani, Tanvir Bhathal, Herumb Shandilya, Hakki Orhun Akengin, Gabriel Bo, Andrew Park, Matthew Hart, Caia Costello, Chuan Li, Christopher Ré, Azalia Mirhoseini · 2026

Read Personal AI, On Personal Devices by Jon Saad-Falcon, Avanika Narayan, Robby Manihani, Tanvir Bhathal, Herumb Shandilya, Hakki Orhun Akengin, Gabriel Bo, Andrew Park, Matthew Hart, Caia Costello, Chuan Li, Christopher Ré, Azalia Mirhoseini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17172.txt.

Introduction

Saad-Falcon and colleagues present OpenJarvis, a personal AI stack built to run entirely on a user's own devices instead of sending sensitive queries to cloud models. Its design splits the system into five editable primitives, and a search step lets cloud models propose improvements that only stick if they do not hurt accuracy. On-device specs then match or beat cloud accuracy on half of eight benchmarks.

Catalog record

CREATORJon Saad-Falcon, Avanika Narayan, Robby Manihani, Tanvir Bhathal, Herumb Shandilya, Hakki Orhun Akengin, Gabriel Bo, Andrew Park, Matthew Hart, Caia Costello, Chuan Li, Christopher Ré, Azalia Mirhoseini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102390
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17172

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Personal AI, On Personal Devices Jon Saad-Falcon, Avanika Narayan, Robby Manihani, Tanvir Bhathal, Herumb Shandilya, Hakki Orhun Akengin, Gabriel Bo, Andrew Park, Matthew Hart, Caia Costello, Chuan Li, Christopher Ré, Azalia Mirhoseini Abstract 1 Introduction Personal AI stacks, like OpenClaw [ 80 ] and Hermes Agent [ 60 ] , are now central to daily writing, research, coding, and scheduling. Yet today’s personal AI is rarely private or local. Most systems route each query to a cloud-hosted frontier model, often including sensitive personal data. This design demands thousands of dollars per year in recurring API and subscription spend [ 96 , 78 ] . It exposes private data to third-party servers, requires network connectivity to function, and leaves users without ownership of the models they depend on. It also consumes orders of magnitude more energy per token than local execution [ 75 ] . Meanwhile, consumer accelerators [ 5 , 70 , 34 ] now run 1B–128B open-weight models with FP8 quantization [ 21 ] , and open-weight families such as Qwen3.5 [ 71 ] and Gemma4 [ 28 ] trail frontier cloud models by 6–12 months on personal AI tasks rather than years [ 13 , 75 ] . On-device models nonetheless remain confined to trivial tasks like tone adjustment and text completion [ 6 ] . In this work we ask: can the core of a personal AI stack (i.e., model inference, agent execution, memory, and learning) run on-device while remaining competitive with cloud-only stacks? We study this question across eight personal AI benchmarks spanning writing, research, coding, and scheduling, including PinchBench [ 40 ] and GAIA [ 50 ] (Section 4.1 ). Figure 1: Overview of OpenJarvis . (Left) Five composable primitives (Intelligence, Engine, Agents, Tools & Memory, Learning) are composed through a declarative spec that can be shared, evaluated, and optimized end-to-end (Section 3.1 ). (Middle) Joint accuracy-efficiency evaluation across the evaluated local specs (green) and 3 cloud baselines (blue) reveals that on-device configurations approach within 3.2 pp of the best cloud model at roughly 800 \times lower marginal API cost per query (Section 4.3 ). (Right) LLM-guided spec search uses frontier cloud models as reflective proposers: the teacher diagnoses failures, proposes edits across the full spec , and a held-out gate accepts only non-regressing improvements, closing the remaining local–cloud accuracy gap by 13–32 pp with student training on-device (Section 4.4 ). As a first attempt, we ask whether one can simply swap the cloud model for a local one inside an existing stack. Replacing Claude Opus 4.6 with Qwen3.5-9B , while keeping the rest of OpenClaw or Hermes Agent fixed, drops accuracy by 25–39 percentage points across PinchBench and GAIA (Section 4.2 , Table 1 ). We identify two reasons for the local-cloud performance gap: • Substitution breaks monolithic stacks. Agent prompts, tool descriptions, memory configuration, and runtime settings are fused into the framework, co-designed for the intended cloud model. Substituting a local model breaks all of these at once (Section 4.2 ). • Single-primitive optimization plateaus. Of the bundled components, only prompts can be tuned without source-level changes. The rest are baked into the framework. Applying state-of-the-art prompt optimizers (GEPA [ 1 ] and DSPy [ 39 ] ) to the swapped-in local model, while holding everything else fixed, closes only 5 pp of the cloud–local gap on their own. Optimizing one primitive at a time cannot deliver the coordinated changes across model, prompts, tools, and runtime that misalignment demands. Motivated by these limitations, we introduce OpenJarvis , an architecture that makes personal AI stacks end-to-end optimizable. OpenJarvis has two core components. First, the spec : a typed configuration object that decomposes the personal AI system as five editable primitives: Intelligence , the language model architecture and its weights; Engine , the inference runtime; Agents , the reasoning loop; Tools & Memory , data integration and persistent storage; and Learning , the optimizer that updates the system from traces (Section 3.1 ). The spec exposes every component of the stack as a degree of freedom in one optimizable object. Second, LLM-guided spec search: a local–cloud search algorithm that jointly optimizes all four editable primitives of the spec: Intelligence, Engine, Agents, and Tools & Memory. LLM-guided spec search runs entirely on-device at inference time but uses frontier cloud models at search time to improve the local spec as much as possible. This gives us the best of both worlds: cloud-model capability transferred into a spec that runs locally. At each search step, a frontier model reads traces from the current spec and proposes coordinated edits (i.e., rewrite a tool description, adjust a runtime setting) across the four primitives. Each edit is accepted only if it improves performance on a held-out evaluation, and accepted edits become the new spec for the next step. Across eight benchmarks for personal AI, optimized specs match or exceed cloud accuracy on four, land within 3.2 pp of the best cloud baseline on average, and reduce marginal API cost by {\sim}800\times and end-to-end latency by 4\times (Section 4.4 ). LLM-guided spec search closes 13–32 pp of the cloud–local gap at 7–11 \times lower optimization cost than single-primitive baselines. We perform a detailed analysis of LLM-guided spec search’s search space. Our analysis highlights three axes that govern the local–cloud gap: (a) Editable surface. How does expanding the move space from prompts alone to all four editable primitives affect gap closure? In Figure 8 and Appendix C.5 , we show that expanding the editable set adds 5.5–16.5 pp. (b) Proposer choice. Does diagnosing failures before proposing edits matter, or would evolutionary spec search at the same move space suffice? In Figure 8 and Appendix C.7 , we show that the diagnose-and-propose loop in LLM-guided spec search adds 10.0 pp on average over evolutionary spec search at the same four-primitive move space. (c) Search budget. How does optimization cost trade off against gap closure? In Figure 7 , we show that LLM-guided spec search matches single-primitive baselines at 7–11 \times lower optimization cost. To summarize, our main contributions are as follows: • Propose OpenJarvis , an architecture that decomposes a personal AI system into five typed primitives composed into an editable spec, making the stack end-to-end optimizable and measurable on accuracy, cost, and latency (Section 3 ). • Propose LLM-guided spec search, a local–cloud collaboration that uses frontier models to propose spec edits at search time and runs the resulting spec entirely on-device at inference time, closing 13–32 pp of the cloud–local gap at 7–11 \times lower optimization cost than single-primitive baselines (Section 4 ). • Conduct an in-depth analysis across eight personal AI benchmarks, showing that on-device specs are Pareto-optimal on marginal API cost and latency with \sim 800 \times lower marginal API cost and 4 \times lower end-to-end latency, and isolating the contributions of the proposer and the move space to the overall gain (Section 4.3 ). 2 Related Work Unifying abstractions have historically accelerated ML progress. Imperative deep-learning frameworks [ 67 ] and type-based declarative ML toolboxes [ 55 ] unified model construction, while data systems for monitoring deployed ML products [ 72 ] and LM pipeline compilers [ 39 ] unified iterative improvement. The personal AI ecosystem is undergoing a

Read the complete text