Layer-wise Token Compression for Efficient Document Reranking Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809871 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Language models CCS: Information systems Information retrieval CCS: Information systems Retrieval models and ranking
Shengyao Zhuang, Zhichao Xu, Ivano Lauriola · 2026
Read Layer-wise Token Compression for Efficient Document Reranking Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809871 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Language models CCS: Information systems Information retrieval CCS: Information systems Retrieval models and ranking by Shengyao Zhuang, Zhichao Xu, Ivano Lauriola (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20683.txt.
Introduction
Shengyao Zhuang, Zhichao Xu, and Ivano Lauriola propose layer-wise token compression for document reranking. Cross-encoder rerankers are central to search but costly on long query-document pairs. Compressing tokens at the input layer works for bi-encoders yet fails for cross-encoders in their tests. Their LTC method instead pools tokens adaptively at middle transformer layers. That preserves ranking quality while lifting throughput by up to 25 percent on passage ranking tasks.
Catalog record
| CREATOR | Shengyao Zhuang, Zhichao Xu, Ivano Lauriola |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29349 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20683
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Layer-wise Token Compression for Efficient Document Reranking Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809871 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Language models CCS: Information systems Information retrieval CCS: Information systems Retrieval models and ranking Shengyao Zhuang, Zhichao Xu, Ivano Lauriola Abstract . Transformer-based document cross-encoder rerankers are a central component of modern information retrieval systems. Despite their success, these models suffer from high computational costs due to processing long query-document sequences at inference time. A known approach to improve efficiency is token compression, which consists of aggregating groups of tokens together in the initial embedding layer, reducing the effective number of tokens and making the computation faster. While token compression has proven to be successful for bi-encoder retrievers, we empirically observed that this approach may be ineffective for cross-encoder rerankers. In this paper, we propose Layer-wise Token Compression (LTC) , which applies adaptive token pooling at intermediate transformer layers. Through extensive ablation studies on MS MARCO passage and document ranking tasks, we demonstrate that compression at middle layers preserves ranking quality while increasing inference QPS by up to 25% for passage ranking and up to 116% for document ranking. We also extend LTC to listwise LLM rerankers and show that the same approach can be easily applied to long-context listwise reranking, where the QPS improvements are even greater. More surprisingly, when applying rerankers trained on short passages to long-document ranking tasks, models trained with compression outperform their uncompressed counterparts, suggesting that compression may act as a beneficial regularizer that encourages length-invariant representations. † † cc-license: by 1. Introduction Transformer-based document reranking models have become essential in modern information retrieval systems, including Transformer-based cross-encoder models ( Nogueira and Cho 2019 ; Yates et al. 2021 ; Xu et al. 2025a ; Xu et al. 2025b ) , generative LLM-based listwise rerankers ( Sun et al. 2023 ; Ma et al. 2023 ; Chen et al. 2025b ; Gangi Reddy et al. 2024 ) and, more recently, reasoning-based rerankers ( Weller et al. 2025 ; Zhuang et al. 2025 ; Yang et al. 2025b ) . Although these methods achieve remarkable effectiveness compared to traditional feature-based learning-to-rank approaches, they suffer from severe efficiency limitations. Specifically, the self-attention mechanism exhibits quadratic time and memory complexity with respect to the input sequence length, whether over query-document pairs in pointwise settings or query and multi-documents in listwise settings, resulting in high computational cost and latency in practical deployments. Token compression has emerged as a promising direction to improve the efficiency of these models ( Goyal et al. 2020 ; Bolya et al. 2023 ; Wei et al. 2025 ) . Broadly speaking, these approaches aim at reducing the number of tokens (e.g., by aggregation) consumed by the models, leading to a latency improvement. Recent work on dense retrieval models with bi-encoder architectures demonstrates that compressing token embeddings at the input layer can significantly reduce computational costs while preserving retrieval quality ( Zhang et al. 2025 ) . However, the applicability of such compression strategies to cross-encoder rerankers, where query and document tokens interact directly and efficiency constraints are most severe, remains limited ( Liu et al. 2025 ) to pointwise ranking. In this paper, we study token compression applied to Transformer-based cross-encoder rerankers in pointwise and listwise settings. We start from the Jasper embedding-level compression approach ( Zhang et al. 2025 ) , which applies 1D adaptive pooling to token embeddings in order to reduce the sequence length before the transformer layers. Although being effective for bi-encoder embedding models, we find that the same strategy does not generalize well when applied to cross-encoder rerankers, and it leads to a degradation in performance. We hypothesize that the issue is due to the fundamental difference in interaction modeling. In cross-encoders, relevance signals emerge from fine-grained query/document token interactions that occur from the earliest transformer layers onward. Applying token compression at the embedding layer removes these signals before interaction can take place, leading to substantial performance degradation. Bi-encoders, by contrast, encode queries and documents independently and rely on vector-level similarity rather than explicit token interactions, making them less sensitive to early-stage token compression. To address this limitation, we propose Layer-wise Token Compression (LTC) , which performs token pooling at an intermediate transformer layer. This design preserves early query/document token interactions in the lower layers, while reducing sequence length to accelerate the remaining computation. LTC yields a parametrizable efficiency/effectiveness trade-off: compressing earlier provides larger speedups but can harm ranking quality, whereas compressing later is safer but less beneficial. We systematically study this trade-off for pointwise cross-encoder rerankers and extend LTC to FIRST ( Gangi Reddy et al. 2024 ) style LLM-based listwise rerankers, where the much longer context can benefit even more from token compression. Overall, we find that intermediate-layer compression preserves ranking quality while substantially reducing computation, naturally extends to listwise rerankers via selective document-token compression, and yields models that generalize better to long-document inputs. 2. Related Work Improving the efficiency of Transformer-based models is a widely explored topic in the literature. Popular approaches include knowledge distillation ( Sanh et al. 2020 ; Hinton et al. 2015 ) , where a smaller model is trained to mimic a larger one; quantization ( Zafrir et al. 2019 ) , which reduces numerical precision to lower computation cost; and pruning ( Michel et al. 2019 ) , which removes redundant parameters or attention heads. Cross-encoder Reranking Models. Cross-encoder rerankers score query-document pairs by jointly encoding the concatenated input, yielding strong effectiveness at the cost of higher latency ( Nogueira and Cho 2019 ; Yates et al. 2021 ; Gao et al. 2021 ; Xu 2026 ; Zhuang and Zuccon 2021 ) . Beyond pointwise scoring, recent work explores LLM-based listwise reranking, where models produce a ranking over a candidate set directly ( Sun et al. 2023 ; Ma et al. 2023 ; Pradeep et al. 2023a ; Gangi Reddy et al. 2024 ; Chen et al. 2025b ; Zhuang et al. 2024 ; Chen et al. 2025a ; Chen et al. 2026 ) . These methods further increase input length (instruction + query + multiple documents), making efficiency a primary bottleneck. Token Compression. Recent methods accelerate transformers by reducing input sequence length, including token pruning based on attention statistics ( Goyal et al. 2020 ) and token merging that combines similar tokens ( Bolya et al. 2023 ) . For dense document retrieval, recent work demonstrates that embedding-level token compression can be highly effective for bi-encoder retrievers by compressing token embeddings before deep transformer computation ( Zhang et al. 2025 ) . However, because cross-encoders rely on early query/document interactions, directly applying the same embedding-level compression can be detrimental. Token reduction has also been studied in the context of reranking models ( Liu et al. 2025 ) . Our work differs in two ways: (1) we systematically characterize where compression should occur in cross-encoder rerankers, showing that intermediate-layer compression offers a favorable efficiency/effectiveness trade-off; and (2) we extend this approach to