Deliberation in Latent Space via Differentiable Cache Augmentation
Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, Arthur Szlam · 2024
Read Deliberation in Latent Space via Differentiable Cache Augmentation by Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, Arthur Szlam (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17747.txt.
Introduction
Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, and Arthur Szlam give a language model extra thinking power without changing the model at all. Standard ways of letting models think longer make them spell out reasoning steps as text first, which is slow and hard to train. Their approach bolts on a separate coprocessor that works directly on the model's key-value cache, the stored record of what it has read so far.
The coprocessor inserts extra latent embeddings into that cache, sharpening what the frozen model produces next. It is trained with the ordinary language modeling objective on standard text, end to end, while the main model stays untouched. Because it runs offline and asynchronously, the model still works normally when the coprocessor is absent. In experiments, augmented caches lowered perplexity on following text and improved results across reasoning-heavy tasks, with no task-specific training.
Catalog record
| CREATOR | Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, Arthur Szlam |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17747
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Deliberation in Latent Space via Differentiable Cache Augmentation Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, Arthur Szlam Abstract Techniques enabling large language models (LLMs) to “think more” by generating and attending to intermediate reasoning steps have shown promise in solving complex problems. However, the standard approaches generate sequences of discrete tokens immediately before responding, and so they can incur significant latency costs and be challenging to optimize. In this work, we demonstrate that a frozen LLM can be augmented with an offline coprocessor that operates on the model’s key-value (kv) cache. This coprocessor augments the cache with a set of latent embeddings designed to improve the fidelity of subsequent decoding. We train this coprocessor using the language modeling loss from the decoder on standard pretraining data, while keeping the decoder itself frozen. This approach enables the model to learn, in an end-to-end differentiable fashion, how to distill additional computation into its kv-cache. Because the decoder remains unchanged, the coprocessor can operate offline and asynchronously, and the language model can function normally if the coprocessor is unavailable or if a given cache is deemed not to require extra computation. We show experimentally that when a cache is augmented, the decoder achieves lower perplexity on numerous subsequent tokens. Furthermore, even without any task-specific training, our experiments demonstrate that cache augmentation consistently reduces perplexity and improves performance across a range of reasoning-intensive tasks. keywords Latent reasoning, Cache augmentation, LLM 1 Introduction Recent research ( Wei et al. 2022 ; Kojima et al. 2022 ; Wu et al. 2024 ) has shown that enabling large language models (LLMs) to generate, or even search over, intermediate sequences of steps before producing a final answer can significantly improve performance on reasoning tasks. More broadly, providing LLMs with the ability to allocate compute adaptively during generation can lead to more effective generation within a fixed compute budget ( Schuster et al. 2022 ) . However, at a high level, many of these “extra thinking” approaches are similar in that their sequences of intermediate outputs are discrete, making them difficult to train in an end-to-end fashion, and in that their extra “thinking” (i.e. computation) is performed just-in-time, as part of the output generating process. In this work, we introduce a fundamentally different approach, inspired by the literature on kv-cache compression ( Mu et al. 2024 ; Ge et al. 2024 ) . Our approach takes a step towards LLMs that can deliberate on their memories (encoded in the kv-cache), and distill these deliberations into a form usable for subsequent tasks. Specifically, our method processes the transformer’s cache and augments it with a set of soft tokens produced in a single forward pass–not sequentially. This extra processing is performed by a separate model, which we refer to as a “coprocessor”, while the base transformer remains frozen. Once the kv-cache is augmented with the coprocessor’s output (which we term “latent embeddings”), decoding proceeds as normal until the coprocessor is called again. This approach offers the following key advantages: End-to-end Differentiability : Our framework enables end-to-end backpropagation during coprocessor training, facilitating efficient optimization without the need for reinforcement learning techniques. We leverage the standard language-modeling loss on pre-training data, making the method scalable. Asynchronous Operation : Because cache augmentation improves results many tokens beyond the augmentation point, and because the base transformer remains frozen during coprocessor training, asynchronous coprocessor operation becomes feasible. This contrasts with existing methods where additional computation occurs sequentially, and online. Our approach opens the door to models that can strategically bank computation by deliberating and refining their internal memory, independent of composing a response to a query. We evaluate our method using Gemma-2 ( Team-Gemma et al. 2024 ) models pretrained on a diverse dataset mixture. Our experiments demonstrate that without any fine-tuning on specific downstream tasks, our approach consistently improves performance across a range of reasoning-intensive tasks. We observed that increasing the number of injected latent embeddings generally leads to better performance. For example, we observe a 10.05% improvement on GSM8K and a 4.70% improvement on MMLU, when augmenting the Gemma-2 2B model with 64 latent embeddings. These results highlight the potential of enhancing LLMs with kv-cache coprocessing for augmenting model capabilities. 2 Methodology We enhance a frozen LLM by training a coprocessor that inputs a key-value (kv) cache, and augments it with a set of soft tokens. This section details the architecture and training process of our approach. 2.1 Problem statement Given an input x and a desired target output y , and a pretrained, frozen LLM parameterized by \theta , we seek to learn a coprocessor, denoted by f . This coprocessor takes the kv-cache ( k_{\theta,x} , v_{\theta,x} ) generated by the frozen LLM when processing the input x as input, and outputs a sequence of latent representations z : \displaystyle f(k_{\theta,x},v_{\theta,x})\longrightarrow z (1) The objective of learning f is to produce latent embeddings z that, when combined with the input x , improve the frozen LLM’s ability to generate the correct target y . Specifically, we aim to maximize the expected log-likelihood of the target y given the input x and the learned latent embeddings z , as predicted by the frozen LLM: \displaystyle\max E_{x}[\log p_{\theta}(y|x,z)] (2) 2.2 Model architecture Figure 1: Overview of the proposed architecture. The input sequence is processed by a frozen LLM, generating a kv-cache. This cache is then passed to a coprocessor, along with trainable soft tokens. The coprocessor outputs latent embeddings which are used to augment the original kv-cache before being fed back into the LLM for output generation. Our proposed architecture enhances a frozen, pretrained LLM with a dedicated coprocessor module operating on the kv-cache. As illustrated in Figure 1 , the interaction between these components unfolds in three stages: • KV-cache Generation: The input sequence, x , is first processed by the frozen LLM to generate its corresponding kv-cache ( k_{\theta,x} , v_{\theta,x} ). This cache encapsulates the LLM’s internal representations of the input. Crucially, the LLM’s weights remain frozen throughout the entire process. • Augmentation: The kv-cache is then passed to the coprocessor module, which adopts the same model architecture as the pretrained LLM. The coprocessor also receives a sequence of distinct extra soft tokens with trainable embeddings. These tokens do not correspond to actual words or sub-words but serve as abstract prompts for the coprocessor. The coprocessor ingests the kv-cache and these tokens to produce a sequence of latent embeddings, z . • LLM Generation with Augmented Context: Finally, z is appended to the original kv-cache. This augmented cache is then fed back into the frozen LLM, providing it with enriched contextual information derived from the coprocessor. The LLM then proceeds to generate the output sequence, y , conditioned on both the original input x and the coprocessor’s output z . This allows the LLM to leverage the coprocessor’s latent inferences without requiring it to explicitly verbalize intermediate steps. Training focuses solely on optimizing the coprocessor and trainable embeddings’ weights. The coprocessor shares the same model architecture as the pretrained LLM, and its weights are initialized with the pretrained weights of the LLM. The loss is calculated on the final output y , and backpropagation is used to update only the