YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference

You Wu, Ziheng Chen, Yizhen Zhang, Haoyi Wu, Chengting Yu, Yuchi Xu, Wenbo Su, Bo Zheng, Kewei Tu · 2026

Read YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference by You Wu, Ziheng Chen, Yizhen Zhang, Haoyi Wu, Chengting Yu, Yuchi Xu, Wenbo Su, Bo Zheng, Kewei Tu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13556.txt.

Introduction

Wu and eight colleagues improve YOCO, a technique that saves memory in large language models by sharing key-value data across layers. Their YOCO++ adds a weighted residual link from each lower-half layer back to the bottom layer, raising model capacity without extra training or inference cost. At half the usual key-value cache size, it leads competing cross-layer compression methods and beats the standard Transformer in their tests.

Catalog record

CREATORYou Wu, Ziheng Chen, Yizhen Zhang, Haoyi Wu, Chengting Yu, Yuchi Xu, Wenbo Su, Bo Zheng, Kewei Tu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16887
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13556

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference You Wu, Ziheng Chen, Yizhen Zhang, Haoyi Wu, Chengting Yu, Yuchi Xu, Wenbo Su, Bo Zheng, Kewei Tu Abstract Cross-layer key-value (KV) compression has been found to be effective in efficient inference of large language models (LLMs). Although they reduce the memory consumption of the KV cache, such methods usually introduce non-negligible performance degradation. In this work, we aim to enhance the performance of YOCO, a cross-layer KV compression method that shares the KVs of the middle layer with the top-half layers. We propose YOCO++, an enhanced YOCO that incorporates a weighted residual connection between the KVs of each bottom-half layer and the bottom layer. Compared to YOCO, YOCO++ increases model capacity while maintaining the same training and inference efficiency. Our experiments show that YOCO++ achieves state-of-the-art performance among the cross-layer KV compression methods at a 50% KV cache compression rate, outperforming the standard Transformer. 1 Introduction The key-value (KV) cache is a major bottleneck for the inference of large language models (LLMs) ( 8 ) based on the Transformer ( 18 ) . Various KV cache compression methods ( 1 ; 27 ) have been proposed, among which cross-layer compression ( 20 ; 15 ; 3 ) is a widely studied category. These methods reduce the memory consumption of KV cache by retaining the KVs of only a subset of Transformer layers and reusing them in the remaining layers. Mainstream cross-layer compression methods directly share the KVs of a specific layer with other layers. For example, You Only Cache Once (YOCO) ( 15 ) shares the KVs of the middle layer with the top-half layers. Although the cross-layer KV sharing methods improve inference efficiency, they usually introduce non-negligible performance degradation. The main cause of this degradation is that the direct KV sharing from one layer to another weakens the model capacity. To alleviate this problem, one solution is to use a weighted combination of the KVs of multiple layers in a specific layer. FusedKV ( 9 ) applies this mechanism to YOCO by using the weighted combination of the KVs of the bottom and middle layers in the top-half layers, achieving a significant performance improvement but also introducing additional cache I/O overhead. In this work, we further explore the KV combination mechanism by using weighted combinations of the KVs of the current layer and other layers in layers that compute their own KVs, which can be viewed as incorporating residual connections between KVs ( 28 ) . We propose YOCO++, which enhances YOCO by introducing a weighted residual connection between the KVs of each bottom-half layer and the bottom layer. To avoid additional cache I/O overhead during decoding, we cache the combined KVs instead of the original ones. Our experiments show that YOCO++ maintains the same inference efficiency as YOCO, and achieves state-of-the-art performance cross-layer KV compression methods at a 50% KV cache compression rate. Our code is available at https://github.com/wuyou2002/YOCO-plus . 2 Related Work 2.1 Cross-Layer KV Cache Compression Cross-layer KV cache compression methods retain the KVs of a subset of Transformer layers and reuse them in the other layers. Layer-Condensed KV Cache (LCKV) ( 20 ) shares the KVs of the top layer with the other layers. You Only Cache Once (YOCO) ( 15 ) shares the KVs of the middle layer with the top-half layers. Cross-Layer Attention (CLA) ( 3 ) shares the KVs of the lower layer with the upper layer in each pair of adjacent layers. A systematic study ( 21 ) is conducted among different cross-layer KV sharing methods, including LCKV, YOCO and CLA. The study finds an effective new configuration called sandwich-middle, which shares the KVs of the middle layer with the other layers. SVFormer ( 28 ) shares the values of the bottom layer with the other layers. SkipV1Former ( 22 ) shares half of the value heads of the bottom layer with the other layers. FusedKV ( 9 ) uses weighted combinations of the KVs of the bottom and middle layers in the top-half layers, and FusedKV-Lite shares the keys of the middle layer and the values of the bottom layer with the top-half layers. 2.2 Enhanced Residual Connections in Transformer As an important component of the Transformer, residual connections ( 7 ) establish an information flow between the hidden states of different layers. In order to enhance the cross-layer information flow in the Transformer, DenseFormer ( 12 ) introduces the Depth-Weighted-Averaging mechanism, which computes a weighted combination of current and past hidden states. MUDDFormer ( 23 ) extends DenseFormer by changing the static, learnable residual weights into dynamic ones computed from the hidden states, and using a multiway design to decouple the four input streams of a Transformer layer, which are the query, key, value and residual. Attention Residuals ( 16 ) computes the dynamic residual weights using a softmax attention over preceding layer outputs, and introduce a block-level variant to reduce the memory footprint while maintaining performance. Hyper-Connections (HC) ( 29 ) expands residual connections in the width rather than the depth dimension by maintaining multiple residual streams, yielding performance gains but also violating the identity mapping property of residual connection and introduceing additional memory access overhead. Manifold-Constrained Hyper-Connections (mHC) ( 24 ) projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. ResFormer ( 28 ) enhances the propagation of initial information by introducing value residual connections in addition to the standard hidden residual connections. 3 Method 3.1 Preliminary: YOCO YOCO ( 15 ) is a decoder-decoder model with L layers consists of two parts, which is shown in Figure 1 (a). The bottom-half \frac{L}{2} layers are called the self-decoder, where the middle layer generates the global KV cache. The top-half \frac{L}{2} layers are called the cross-decoder, which reuses the global KV cache and does not generate KV cache by itself. The original YOCO uses efficient attention in the self-decoder to achieve higher efficiency, while we use standard attention in this work for fair comparison. The size of the KV cache is reduced by 50% because we only need to cache the KVs in the bottom-half layers. The prefilling latency is also reduced by 50% because we can exit early before computing the top-half layers. However, the cross-layer KV sharing mechanism introduce non-negligible performance degradation. 3.2 YOCO++ To enhance the performance of YOCO, we introduce a cross-layer KV combination mechanism, which uses a weighted combination of KVs of multiple layers in a specific layer. If we apply the combination mechanism to the top-half layers, the cache I/O of each layer increases proportionally with the number of KV layers it uses. To avoid this additional I/O overhead, we instead apply the combination mechanism to the bottom-half standard Transformer layers, which can be viewed as incorporating residual connections between KV. The question is, which layers’ KVs should we add to the KVs of each layer? In theory, adding the KVs of all previous layers to those of each layer would provide the strongest capacity. In practice, 28 finds that adding only the values of the bottom layer to those of each layer achieves

Read the complete text