Alchemist : Towards the Design of Efficient Online Continual Learning System
Yuyang Huang, Yuhan Liu, Haryadi S. Gunawi, Beibin Li, Changho Hwang · 2025
Read Alchemist : Towards the Design of Efficient Online Continual Learning System by Yuyang Huang, Yuhan Liu, Haryadi S. Gunawi, Beibin Li, Changho Hwang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01066.txt.
Introduction
Large language models can keep learning from user feedback after deployment, but the usual setup wastes work. Training and serving run apart, so the trainer recomputes activations that serving already produced, burning 30 to 42 percent of training time. The authors present Alchemist, a system designed to reuse those serving activations for online continual learning. It records activations and key-value caches only during the prefill phase, which keeps the cost to serving small.
A second technique offloads stored activations smartly and hedges against slowdowns. Tests use ShareGPT-style inputs of varied length. Against a separate training cluster, the system raises training throughput up to 1.72 times, cuts training memory use by up to 47 percent, and doubles the number of training tokens supported. Serving latency barely moves. The authors offer Alchemist as the first design that makes online learning cheap enough to run beside live service.
Catalog record
| CREATOR | Yuyang Huang, Yuhan Liu, Haryadi S. Gunawi, Beibin Li, Changho Hwang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01066
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Alchemist : Towards the Design of Efficient Online Continual Learning System Yuyang Huang, Yuhan Liu, Haryadi S. Gunawi, Beibin Li, Changho Hwang Abstract . Continual learning has become a promising solution to refine large language models incrementally by leveraging user feedback. In particular, online continual learning — iteratively training the model with small batches of user feedback — has demonstrated notable performance improvements. However, the existing practice of separating training and serving processes forces the online trainer to recompute the intermediate results already done during serving. Such redundant computations can account for 30%–42% of total training time. In this paper, we propose Alchemist , to the best of our knowledge, the first online continual learning system that efficiently reuses serving activations to increase training throughput. Alchemist introduces two key techniques: (1) recording and storing activations and KV cache only during the prefill phase to minimize latency and memory overhead; and (2) smart activation offloading and hedging. Evaluations with inputs of varied token length sampled from ShareGPT dataset show that compared with a separate training cluster, Alchemist significantly increases training throughput by up to 1.72x, reduces up to 47% memory usage during training, and supports up to 2x more training tokens — all while maintaining negligible impact on serving latency. 1. Introduction Large Language Model (LLM) services, widely adopted in modern cloud ecosystems, power applications, such as chatbots, search engines, recommendation systems, and API offerings. The market is projected to grow from $6.5B (2024) to $140.8B (2033), as 92% of Fortune 500 companies integrate cloud-hosted LLMs for generative AI workflows ( TechTarget Editorial, 2023 ; Topics, 2024 ; Company, 2023 ) . To ensure these models remain accurate and continue to enhance user experiences, organizations must regularly retrain them to learn the latest data and refine performance. One of the most popular and promising training approaches is continual learning ( Gabriel, 2020 ; Zhang et al., 2023a ; Sun et al., 2020 ; Zhang et al., 2023b ; Cheng et al., 2024 ; Gabriel, 2020 ; Zan et al., 2022 ; Song et al., 2023 ; Cossu et al., 2022 ; Gururangan et al., 2020 ) , due to its capability of allowing models to be refined frequently, without training from scratch. It incrementally adapts the models with up-to-date real-world information and interactions with users or models that mimic user preferences ( Guo et al., 2024 ; Lee et al., 2024b ; DeepSeek-AI et al., 2025 ; Xiang et al., 2025 ) . For example, chatbots with web searching capability can learn from the newest searched context ( e.g. , latest news, financial data, et al. ) to provide the latest information in future responses. Or, code completion applications allow users to accept or reject suggested code, and this user feedback can be used to allow models to continuously learn their preference, providing better and more personalized code completion and debug suggestions. One of the most popular continual learning approach, online continual learning (§ 2.4 ), iteratively trains the model based on a small batch of user feedback or model-generated labels collected each time the model updated ( Guo et al., 2024 ; Qi et al., 2024a ; Gao et al., 2024b ; Shaikh et al., 2024 ; Xu et al., 2024 ; Carta et al., 2024 ; Gaven et al., 2024 ) . Figure 1. Lifecycle of modern AI services. In this context, we observe a significant gap between the modern need for iterative and frequent updates in LLM services and the conventional paradigm for LLM training. As illustrated in Figure 1 , existing systems typically separate training from serving to maintain low latency for serving. This separation occurs both temporally ( i.e. , data collected during serving is only used for training after a considerable delay) and spatially ( i.e. , training is carried out on machines that are distinct from those used for serving). However, such hard separation loses substantial opportunities to reduce computational costs. An important and interesting observation we made is that, with such design, online continual learning trainer needs to repeat the same computation that has already been conducted by the serving process, and this redundant computation often contributes to a significant portion of the entire computation for training. In reality, the trainer would spend 30%–42% of the total time for the computations that have already been done during serving, depending on the loss functions. This observation reveals a key insight of our approach: the intermediate results (i.e., activations) generated during serving can be directly re-used in the forward pass of training . Specifically, we reuse these activations from serving to eliminate redundant forward-pass computations. However, if the training and serving processes are separate on different machines as existing systems designed, to reuse the serving actions, they need to be sent to the training machines via the network, posing large network transmission overhead given the large size of activations. An intuitive and straightforward design is to remove the hard separation between serving and training and co-locate serving and training on the same machine, leveraging GPU multiplexing techniques ( e.g. , either temporal sharing or spatial sharing). So, once there is idle time in the serving process ( e.g. , during off-peak hours at night time), the serving activations can be used directly by the trainer, without any transmission delay across machines. A potential concern around this design is whether the serving workload offers sufficient idle resources. Nowadays, the serving resource is typically over-provisioned ( Kubernetes, 2025 ) to ensure end users’ experience ( e.g. latency service-level objectives (SLOs) guarantee). This indicates that a significant amount of idle resources is always available and can be harvested for training usage. Another concern is whether the newly collected serving data is safe to be immediately incorporated into model updates and whether the updated model can be immediately deployed to users. To address potential safety risks, multiple studies ( Kim and Lee, 2024 ; Inan et al., 2023 ; Morimura et al., 2024 ; Ousidhoum et al., 2021 ; Ji et al., 2024 ) have proposed safeguards to ensure only safe data can be trained by the model. Many other works ( Guo et al., 2024 ; Qi et al., 2024a ; Gao et al., 2024b ; Shaikh et al., 2024 ; Xu et al., 2024 ; Carta et al., 2024 ; Gaven et al., 2024 ; Wang et al., 2023b ) focus on ensuring that the updated models neither drift too far away from the pre-trained model, losing its generality, nor generate unethical or unlawful responses. However, there is still a system-level challenge that remains unsolved. Specifically, co-locating serving and training on the same machine may incur extra overhead in saving and reusing the serving activations. First of all, although reusing activations can reduce training latency, the serving process needs to record and store the activations produced by each layer and the forward computation graph, to be used in the backward pass for gradient computation (§ 4.1 ). This procedure incurs additional latency overhead to the forward passes during serving, which could potentially violate serving latency SLO. Concretely, such latency overhead can be up to 35% when storing and recording activations and computation graphs. Secondly, in a normal