AME: An Efficient Heterogeneous Agentic Memory Engine for Smartphones

Xinkui Zhao, Qingyu Ma, Yifan Zhang, Hengxuan Lou, Guanjie Cheng, Shuiguang Deng, Jianwei Yin · 2025

Read AME: An Efficient Heterogeneous Agentic Memory Engine for Smartphones by Xinkui Zhao, Qingyu Ma, Yifan Zhang, Hengxuan Lou, Guanjie Cheng, Shuiguang Deng, Jianwei Yin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19192.txt.

Introduction

Xinkui Zhao and eight co-authors build MUSE, a search engine for multimedia that runs on the phone itself. On-device retrieval powers cross-modal search and personal AI agents, but phones are hard hosts: the embeddings that represent photos, video and audio run to over a thousand dimensions. Media streams grow continuously through lifelogs and ambient recording, and memory, energy, and latency budgets are strict. Server retrieval systems assume static data and do not fit this moving target.

MUSE is co-designed for mobile chips, so interactive queries can run alongside background ingestion and index upkeep. It tackles the clash between a user's instant search and the constant indexing of new media on shared hardware. The work is a systems contribution aimed at private, on-device personal media assistants.

Catalog record

CREATORXinkui Zhao, Qingyu Ma, Yifan Zhang, Hengxuan Lou, Guanjie Cheng, Shuiguang Deng, Jianwei Yin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19192

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AME: An Efficient Heterogeneous Agentic Memory Engine for Smartphones Xinkui Zhao, Qingyu Ma, Yifan Zhang, Hengxuan Lou, Guanjie Cheng, Shuiguang Deng, Jianwei Yin Abstract . On-device agents on smartphones increasingly require continuously evolving memory to support personalized, context-aware, and long-term behaviors. To meet both privacy and responsiveness demands, user data is embedded as vectors and stored in a vector database for fast similarity search. However, most existing vector databases target server-class environments. When ported directly to smartphones, two gaps emerge: (G1) a mismatch between mobile SoC constraints and vector-database assumptions, including tight bandwidth budgets, limited on-chip memory, and stricter data type and layout constraints; and (G2) a workload mismatch, because on-device usage resembles a continuously learning memory, in which queries must coexist with frequent inserts, deletions, and ongoing index maintenance. To address these challenges, we propose AME , an on-device A gentic M emory E ngine co-designed with modern smartphone SoCs. AME introduces two key techniques: (1) a hardware-aware, high-efficiency matrix pipeline that maximizes compute-unit utilization and exploits multi-level on-chip storage to sustain high throughput; and (2) a hardware- and workload-aware scheduling scheme that coordinates querying, insertion, and index rebuilding to minimize latency. We implement AME on Snapdragon 8-series SoCs and evaluate it on HotpotQA ( 24 ) . In our experiments, AME improves query throughput by up to 1.4 × 1.4\times at matched recall, achieves up to 7 × 7\times faster index construction, and delivers up to 6 × 6\times higher insertion throughput under concurrent query workloads. 1. Introduction AI agents are rapidly evolving from single-turn assistants into persistent, goal-directed systems, including household robots ( 6 ) , mobile voice assistants ( 15 ) , and task planners ( 9 ) . To personalize behavior and improve sample efficiency, modern agents rely on agentic memory that accumulates over time ( 1 ; 21 ; 19 ; 26 ) . They record user history, voice interactions, video input, summaries of prior interactions to ground future decisions, as illustrated in Figure 1 . This information is embedded as vectors and stored in a vector database to enable efficient similarity search. At inference time, the agent retrieves the most relevant memories as additional context, conditioning its decisions on user-specific signals. In this way, agentic memory supports personalized behavior, including recalling long-horizon context, adapting tone to the user, and generating more concise prompts ( 16 ; 12 ) . A recent trend is to bring such agents onto smartphones ( 23 ; 25 ; 4 ) . Smartphones are a natural home for these agents: they carry the richest, most up-to-date personal context and are the primary interface for everyday tasks ( 20 ) . Keeping the agent’s vector memory on-device ensures low latency and privacy by avoiding network access to sensitive data. However, directly deploying server-oriented vector databases on smartphones is ineffective and exposes two gaps: Figure 1 . Motivation for the proposed AME : agentic memory stored in a vector database requires adaptation for efficient use on smartphones. G1: SoC–Database Mismatch. Server-side designs assume abundant bandwidth, large memory, and mature accelerator ecosystems. For example, IVF-based methods rely on wide, fast memory to probe many lists in parallel and on ample DRAM for centroids and postings, while HNSW ( 8 ) assumes large memory for graph neighbors, cache-friendly pointer chasing, and thread-rich CPUs/GPUs. In contrast, mobile SoCs operate under tight power and bandwidth budgets, have limited on-chip memory, and lack a unified matrix runtime to distribute workloads across CPUs, GPUs, and NPUs. Although recent SoCs integrate increasingly capable heterogeneous units ( 23 ; 18 ; 2 ) , they remain underutilized in vector workloads, where distance computations and type/layout conversions often fall back to the CPU. G2: Workload Mismatch. Unlike servers that often build indexes offline and serve relatively stable corpora, on-device usage is inherently dynamic. Smartphones continuously accumulate and modify personal context from interactions and sensors; there is no clear boundary between “idle” and “in use”. As a result, the vector memory must support frequent inserts and deletes while serving queries, and enable fast rebuild without disrupting interaction—demanding low tail latency and minimal interference under hybrid workloads. To address these gaps, we present AME , a collaborative heterogeneous agentic memory engine for smartphones. By exploiting heterogeneous compute units and the unified system memory fabric of modern SoCs, AME dynamically coordinates semantic memory operations with model inference, minimizing redundant data movement while maintaining low latency. Our key insight is that mobile processors provide a unified memory fabric shared by CPUs, GPUs, and NPUs, which, when effectively utilized, can substantially reduce communication overhead between memory management and model execution. Building on this observation, AME adopts a full-stack co-design across hardware and vector database, guided by two main techniques: (1) Hardware-Aware Heterogeneous Vector Computation Pipeline. AME refactors core vector similarity operations into accelerator-native matrix computations and connects CPUs, GPUs, and NPUs through a shared-memory-based pipeline. A unified, file-descriptor-based memory framework exposes common buffers to all compute units without explicit copies, allowing each backend to operate on shared data with minimal movement. Within this framework, AME leverages the NPU for heavy GEMM workloads by performing in situ FP32 \leftrightarrow FP16 conversion and layout transformation in HVX, and by overlapping execution with transfers. (2) Hardware- and Workload-Aware Vector Index Design. To address workload–hardware mismatch on mobile devices, we co-design the vector index and its execution paths with both the NPU and the memory hierarchy in mind. On the index side, we make IVF hardware-aware by aligning clustering and embedding dimensions with the NPU’s GEMM tile shapes, so that index build, insertion, and centroid updates map to dense, well-utilized matrix multiplications instead of irregular scalar code. On the execution side, we make the mapping workload-aware : AME distinguishes latency-critical single-query lookups, throughput-oriented batched queries, and background maintenance (inserts and rebuilds), and routes them through different CPU, GPU, and NPU paths using a small set of profiling-guided templates. This paper makes the following contributions: • To the best of our knowledge, AME is the first on-device agentic memory engine that exploits smartphone heterogeneity. On Snapdragon 8-series SoCs, AME improves query throughput by up to 1.4\times at matched recall, achieves up to 7\times faster index construction, and delivers up to 6\times higher insertion throughput under concurrent query workloads. • We introduce an NPU-side data adaptation layer that performs data type conversion, in-place layout transforms/transposes in HVX, and batched GEMM via shared-memory mapping, eliminating CPU-side preprocessing and reducing CPU–NPU overhead. • We co-design a hardware- and workload-aware vector index with template-driven heterogeneous execution and a memory-efficient scheduler, mapping queries, updates, and rebuilds to suitable computing units. 2. Background 2.1. Rise of On-Device Agentic Memory Recent advances in large language models (LLMs) have enabled intelligent assistants with strong reasoning capabilities ( 14 ; 5 ; 26 ) . As these assistants migrate to on-device execution, they require an internal agentic memory , namely a continuously updated store of user-specific signals such as recent interactions, local content, and stable preferences. Agentic memory extends beyond retrieval and must also support fast, continual updates. On mobile hardware, however, it is challenging to provide both low-latency queries and frequent updates under tight compute and memory budgets. Figure 2 . The architecture of the Snapdragon 8

Read the complete text