Exploring RWKV for Sentence Embeddings: Layer-wise Analysis and Baseline Comparison for Semantic Similarity
Xinghan Pan · 2025
Read Exploring RWKV for Sentence Embeddings: Layer-wise Analysis and Baseline Comparison for Semantic Similarity by Xinghan Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14620.txt.
Introduction
Xinghan Pan asks if the RWKV language model can give useful sentence embeddings with no fine-tuning at all. RWKV is known for its linear attention method. That method scales in a way that differs from standard transformers. Pan asks what sense of likeness between sentences its hidden layers hold in a zero-shot setting.
The study looks at embeddings layer by layer from a trained RWKV model. Scores are gauged on the Microsoft Research Paraphrase Corpus with Spearman correlation. The base for contrast is a GloVe-based model. RWKV embeddings catch some sense of likeness, but they do worse than the simpler GloVe base. Pan also times how fast the model runs and how much graphics memory it uses. The results say RWKV's linear scaling gains do not yield embedding quality on their own.
Catalog record
| CREATOR | Xinghan Pan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14620
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring RWKV for Sentence Embeddings: Layer-wise Analysis and Baseline Comparison for Semantic Similarity Xinghan Pan Abstract This paper investigates the efficacy of RWKV, a novel language model architecture known for its linear attention mechanism, for generating sentence embeddings in a zero-shot setting. I conduct a layer-wise analysis to evaluate the semantic similarity captured by embeddings from different hidden layers of a pre-trained RWKV model. The performance is assessed on the Microsoft Research Paraphrase Corpus (MRPC) dataset using Spearman correlation, and compared against a GloVe-based baseline. My results indicate that while RWKV embeddings capture some semantic relatedness, they underperform compared to the GloVe baseline in terms of Spearman correlation. Furthermore, I observe a trend of decreasing semantic similarity performance with increasing layer depth in RWKV. I also analyze the inference time and GPU memory usage, highlighting the computational trade-offs associated with RWKV embeddings. The findings suggest that while RWKV offers potential advantages in terms of linear scaling, its zero-shot sentence embedding quality for semantic similarity tasks requires further investigation and potential task-specific fine-tuning to match or exceed simpler baselines. I discuss the limitations of my current study and propose directions for future work, including exploring advanced pooling strategies, comparison with state-of-the-art models, task-specific fine-tuning approaches, and more in-depth layer-wise analysis to enhance RWKV’s sentence embedding capabilities. To provide deeper insights, I also include a theoretical analysis of the RWKV architecture, discussing its linear attention, information propagation, pooling, and complexity from an information-theoretic perspective. 1 Introduction Sentence embeddings, vector representations of sentences that capture their semantic meaning, are crucial for various natural language processing (NLP) tasks, including semantic similarity detection, text classification, and information retrieval [ 6 , 14 ] . Traditionally, Transformer-based models, such as BERT [ 7 ] and Sentence-BERT [ 20 ] , have been dominant in generating high-quality sentence embeddings, achieving state-of-the-art results on numerous benchmarks [ 4 , 29 ] . However, these models often suffer from quadratic complexity in attention mechanisms, leading to high computational costs, especially for long sequences, limiting their scalability and efficiency in resource-constrained environments [ 24 ] . This motivates the exploration of efficient alternatives for sentence embedding generation. RWKV, a recently proposed language model architecture [ 19 ] , offers a promising alternative with its innovative RWKV (Receptance Weighted Key Value) attention mechanism. This architecture achieves linear time complexity, potentially offering significant advantages in terms of computational efficiency and scalability, making it attractive for deployment in real-world applications [ 19 ] . While RWKV has shown remarkable performance in language modeling tasks, achieving competitive results with Transformer-based models [ 19 ] , its application in generating sentence embeddings, particularly in zero-shot scenarios, remains relatively unexplored. This is especially pertinent given RWKV’s unique architecture, diverging from the prevalent Transformer paradigm, and warrants investigation into its capabilities for semantic representation. This paper aims to bridge this gap by investigating the effectiveness of pre-trained RWKV models for sentence embedding generation and semantic similarity assessment. I conduct a layer-wise analysis, extracting embeddings from different hidden layers of the RWKV model to determine the optimal layer for capturing semantic similarity. I evaluate the performance on the Microsoft Research Paraphrase Corpus (MRPC) dataset [ 8 ] , a benchmark for paraphrase detection, using Spearman correlation as the primary evaluation metric, a standard measure for semantic similarity tasks [ 20 , 4 ] . Furthermore, I compare the performance of RWKV embeddings against a simple yet effective GloVe-based baseline [ 18 ] , and analyze the inference time and GPU memory usage to assess the computational efficiency of RWKV in this context. By exploring RWKV for sentence embeddings, I aim to uncover its potential in this domain and highlight its unique trade-offs compared to established methods. This work provides an initial exploration into a novel application area for RWKV, distinct from existing applications like UI generation, image processing, or general language modeling, and contributes to the broader understanding of efficient sentence representation techniques. This study contributes to the growing body of research on efficient language models [ 23 , 13 , 28 ] and explores a novel application of the RWKV architecture for sentence embeddings. The findings provide initial insights into the zero-shot semantic similarity capabilities of RWKV and highlight potential avenues for future research and development in this direction, specifically focusing on how to best leverage RWKV’s architecture for semantic representation and bridge the performance gap with more established sentence embedding techniques. To further understand the observed empirical results, I also provide a theoretical analysis of the RWKV architecture, examining its core mechanisms and suggesting directions for improvement. 2 Related Work Sentence embeddings have been extensively studied in NLP, with early approaches relying on word averaging techniques using pre-trained word embeddings like Word2Vec [ 17 ] and GloVe [ 18 ] . These methods, while computationally efficient and easy to implement, often fail to capture complex semantic relationships, word order, and sentence structure, limiting their effectiveness in tasks requiring deeper semantic understanding [ 25 ] . The advent of deep learning, particularly recurrent neural networks (RNNs) and Transformers, revolutionized sentence embedding generation. Models like Skip-Thought [ 14 ] and FastSent [ 11 ] utilized RNNs to encode sentences into vector representations, capturing sentence-level semantics beyond simple word averaging. However, Transformer-based models, especially BERT [ 7 ] and its variants, have achieved state-of-the-art performance on various semantic tasks, setting new benchmarks in sentence representation learning [ 20 , 6 ] . Sentence-BERT (SBERT) [ 20 ] fine-tunes BERT with Siamese and triplet networks to produce semantically meaningful sentence embeddings that can be efficiently compared using cosine similarity, making it highly practical for semantic similarity search and clustering. Universal Sentence Encoder (USE) [ 4 ] also provides high-quality sentence embeddings. More recently, SimCSE [ 9 ] demonstrates effective contrastive learning for sentence embedding. These models, however, inherit the computational cost associated with Transformer architectures, particularly the quadratic complexity of the self-attention mechanism. Despite their success, Transformer models are computationally demanding due to their quadratic attention complexity, which becomes a bottleneck for processing long sequences and large-scale datasets [ 24 ] . This limitation has spurred research into more efficient architectures, including models with linear attention mechanisms and alternatives to attention [ 23 , 13 , 28 , 27 ] . RWKV [ 19 ] is a notable example, employing a novel RWKV attention that offers linear scaling with sequence length, addressing the efficiency concerns of traditional Transformers while maintaining competitive performance in language modeling. RWKV’s architecture, which blends RNN and Transformer principles, presents a unique approach to language modeling, offering a potentially more efficient alternative for various NLP tasks. While RWKV has demonstrated strong language modeling capabilities, its application to sentence embedding generation and semantic similarity tasks is still in its nascent stages. This paper aims to contribute to this emerging area by providing an empirical evaluation of RWKV for zero-shot semantic similarity, focusing on layer-wise analysis and comparison with a traditional baseline. Future work should extend this comparison to include state-of-the-art sentence embedding models to fully contextualize RWKV’s performance. This research explores a novel use case for this architecture beyond its original language modeling focus and contributes to the broader field of efficient sentence representation learning. 3 Methodology