AlayaDB: The Data Foundation for Efficient and Effective Long-context LLM Inference
Yangshen Deng*, Zhengxin You*, Long Xiang*, Qilong Li, Peiqi Yuan, Zhaoyang Hong, Yitao Zheng, Wanting Li, Runzhong Li, Haotian Liu, Kyriakos Mouratidis, Man Lung Yiu, Huan Li, Qiaomu Shen, Rui Mao, Bo Tang 🖂 Corresponding email: [email protected] · 2025
Read AlayaDB: The Data Foundation for Efficient and Effective Long-context LLM Inference by Yangshen Deng*, Zhengxin You*, Long Xiang*, Qilong Li, Peiqi Yuan, Zhaoyang Hong, Yitao Zheng, Wanting Li, Runzhong Li, Haotian Liu, Kyriakos Mouratidis, Man Lung Yiu, Huan Li, Qiaomu Shen, Rui Mao, Bo Tang 🖂 Corresponding email: [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10326.txt.
Introduction
Yangshen Deng and co-authors present a database built for long-context language model inference, in a 2025 paper. When a model reads a very long prompt, it stores a large cache of keys and values, and computing attention over that cache is costly. Their system, AlayaDB, pulls that cache and its computation out of the model server into a dedicated vector database.
For firms that sell model access as a service, the design uses less hardware and gives higher output quality across workloads with unlike service targets. The core idea is to treat attention and cache handling as a database query, with a native optimiser for that query. Three industry use cases and broad tests on inference benchmarks are reported. Comparisons include cache splitting across machines and sparse attention by retrieval, against which the authors claim clear gains.
Catalog record
| CREATOR | Yangshen Deng*, Zhengxin You*, Long Xiang*, Qilong Li, Peiqi Yuan, Zhaoyang Hong, Yitao Zheng, Wanting Li, Runzhong Li, Haotian Liu, Kyriakos Mouratidis, Man Lung Yiu, Huan Li, Qiaomu Shen, Rui Mao, Bo Tang 🖂 Corresponding email: [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10326
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AlayaDB: The Data Foundation for Efficient and Effective Long-context LLM Inference Yangshen Deng*, Zhengxin You*, Long Xiang*, Qilong Li, Peiqi Yuan, Zhaoyang Hong, Yitao Zheng, Wanting Li, Runzhong Li, Haotian Liu, Kyriakos Mouratidis, Man Lung Yiu, Huan Li, Qiaomu Shen, Rui Mao, Bo Tang 🖂 Corresponding email: [email protected] Abstract . AlayaDB is a cutting-edge vector database system natively architected for efficient and effective long-context inference for Large Language Models (LLMs) at AlayaDB AI . Specifically, it decouples the KV cache and attention computation from the LLM inference systems, and encapsulates them into a novel vector database system. For the Model as a Service providers (MaaS), AlayaDB consumes fewer hardware resources and offers higher generation quality for various workloads with different kinds of Service Level Objectives (SLOs), when compared with the existing alternative solutions (e.g., KV cache disaggregation, retrieval-based sparse attention). The crux of AlayaDB is that it abstracts the attention computation and cache management for LLM inference into a query processing procedure, and optimizes the performance via a native query optimizer. In this work, we demonstrate the effectiveness of AlayaDB via (i) two use cases from our industry partners, and (ii) extensive experimental results on LLM inference benchmarks. CCS: Information systems Data management systems CCS: Computing methodologies Artificial intelligence † † footnotetext: * These authors contributed equally to this work. 🖂 Corresponding author: Prof. Bo Tang. 1. Introduction Large Language Models (LLMs) have been widely used in various real-world applications such as personal assistants ( cha, 2024 ; kim, 2024 ; gem, 2024 ; dee, 2024a ; Hui et al., 2024 ) , search engines ( bin, 2024 ; per, 2024 ; goo, 2024 ; ama, 2024 ) , code generators ( dee, 2024b ; cop, 2024 ; cur, 2024 ; Hui et al., 2024 ) and document analyzers ( Dhingra, 2023 ; Lin et al., 2024 ) . Efficient and effective LLM inference is an open problem in the industry ( lla, 2024a ; Pichai and Hassabis, 2024 ; Cai et al., 2024a ; AI et al., 2025 ) , especially for long-context (e.g., millions of tokens) inference. In particular, the performance of LLM inference systems is evaluated by three metrics: (1) inference latency , the end-to-end time cost for user tasks, (2) generation quality , the capabilities of LLM in various workloads, and (3) GPU memory consumption , the used hardware resources for the user tasks. Many solutions have been proposed to optimize these metrics in long-context LLM inference. They can be classified into three categories: (i) coupled architecture; (ii) KV cache disaggregation; and (iii) retrieval-based sparse attention. vLLM ( Kwon et al., 2023 ) , SGLang ( Zheng et al., 2024 ) and HuggingFace transformers ( Wolf et al., 2020 ) are the most widely-used LLM inference systems in (i) coupled architecture. LLM model computation and KV cache management are tightly coupled in these systems. These systems achieve high generation quality as they use a full attention mechanism. Mooncake ( Qin et al., 2024 ) and LMCache ( lmc, 2024 ; Liu et al., 2024b ) are representative LLM inference systems in (ii) KV cache disaggregation. They store the KV cache of contexts in external storage and reuse them among different LLM inference instances. Thus, the inference latency of these systems is improved as it reuses the KV cache and reduces the expensive computations (e.g., inner product and softmax). Recently, retrieval-based sparse attention solutions have been proposed (e.g., InfLLM ( Xiao et al., 2024c ) and RetrievalAttention ( Liu et al., 2024a ) ) to alleviate the large GPU memory consumption of these systems in both (i) and (ii). The core idea behind them is the sparse attention mechanism, i.e., only a subset of critical key and value tokens are selected to perform the attention computation. Unfortunately, existing systems cannot simultaneously optimize the three aforementioned performance metrics, as we will elaborate in Section 3 . At AlayaDB.AI , we designed an LLM-native vector database AlayaDB to overcome the limitations of existing LLM inference systems/solutions and enable efficient and effective long-context inference in LLM era. Specifically, for Model as a Service (MaaS) ( Gan et al., 2023 ) providers, the SLOs of different kinds of workloads indicate their requirements for the inference latency. Thus, the core challenge of AlayaDB is solving a bi-objective optimization problem, i.e., meet the SLOs of different workloads by consuming less GPU memory and offering higher generation quality simultaneously. The core idea of AlayaDB is to decouple both KV cache and attention computation and to encapsulate them into a monolithic vector database. The major benefits of the novel disaggregation level are three-fold. \bullet Lightweight LLM Inference System. The cache management and attention computation can be separated from the LLM inference engine, which lightens its burden. \bullet Interface Simplification. It simplifies the interface between LLM inference engine and KV cache service by only returning the attention result, instead of the KV cache content. \bullet Co-optimization Opportunity. It sheds light on co-optimizing attention computation and KV cache management in a monolithic vector database together. At a high level, AlayaDB’s role in LLM inference is comparable to the role of traditional databases ( Verbitski et al., 2017 ; Dageville et al., 2016 ; Stonebraker and Rowe, 1986 ; Lyu et al., 2021 ; Huang et al., 2020 ; Astrahan et al., 1976 ) in web applications. Specifically, the LLM application developers only need to pay attention to the logic of their applications while AlayaDB offers efficient long-context management from their developed LLM applications. This is analogous to web application developers focusing on the logic of their applications and leaving efficient data management to a traditional relational database. To achieve the above vision, there are three design goals of AlayaDB: (i) ease-to-use, (ii) high generation quality, and (iii) good efficiency. AlayaDB employs a novel system architecture and introduces end-to-end optimizations. Firstly, it provides simple-yet-powerful abstractions and APIs, which are compatible with the software ecosystem of LLMs. Secondly, it handles sparse attention computation as a vector search query. To improve the generation quality and reduce the memory consumption simultaneously, AlayaDB defines a novel query type, i.e., dynamic inner product range query (DIPR), which overcomes the limitations of the traditional top- k query. To accelerate query processing, AlayaDB includes a native query optimizer, which selects the best execution plan for efficient vector search. Last but not least, a suite of optimization techniques (from algorithm-side to index-side, from computation to storage) has been employed in AlayaDB. Compared to existing LLM inference systems, AlayaDB enjoys low latency, high generation quality, and low resource consumption at the same time from long-context inference. Our experience shows that AlayaDB greatly lowers the cost of hardware resources for handling long contexts and lightens the labor for optimizing the LLM infrastructure. AlayaDB has already been used to support several online LLM services including chatting apps and knowledge-base QA services in our industry partners. To sum up, the technical contributions of AlayaDB are as follows. \bullet Novel Decoupling Level for LLM Inference Systems: We classify existing LLM inference solutions into three categories and analyze their limitations to handle the challenges of long-context LLM inference. Then, we decouple the KV cache and attention computation from the LLM inference system