Dewey Long Context Embedding Model: A Technical Report
Dun Zhang, Panxiang Zou, Yudong Zhou · 2025
Read Dewey Long Context Embedding Model: A Technical Report by Dun Zhang, Panxiang Zou, Yudong Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20376.txt.
Introduction
Dun Zhang, Panxiang Zou, and Yudong Zhou give a 2025 technical report on Dewey, a text embedding model. Embedding models turn text into numbers that search systems can compare. Long documents strain most models, which lose coherence past short lengths and hurt search quality.
Dewey is built to handle very long inputs and still keep local and global meaning. Its training aligns chunks of a document with the whole through a teaching method called distillation. The open model scores well on standard English and long-text tests. Teams that build search over large files may find it a useful base, and the release notes are shared in the report.
Catalog record
| CREATOR | Dun Zhang, Panxiang Zou, Yudong Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20376
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dewey Long Context Embedding Model: A Technical Report Dun Zhang, Panxiang Zou, Yudong Zhou Abstract This technical report presents the training methodology and evaluation results of the open-source dewey_en_beta embedding model. The increasing demand for retrieval-augmented generation (RAG) systems and the expanding context window capabilities of large language models (LLMs) have created critical challenges for conventional embedding models. Current approaches often struggle to maintain semantic coherence when processing documents exceeding typical sequence length limitations, significantly impacting retrieval performance in knowledge-intensive applications. This paper presents dewey_en_beta, a novel text embedding model that achieves excellent performance on MTEB (Eng, v2) Enevoldsen et al. 2025 and LongEmbed benchmark Zhu et al. 2024 while supporting 128K token sequences. Our technical contribution centers on chunk alignment training, an innovative methodology that enables the simultaneous generation of localized chunk embeddings and global document-level representations through distillation Zhang et al. 2025 . Information regarding the model release can be found at https://huggingface.co/infgrad/dewey_en_beta . 1 Introduction Text embedding models serve as fundamental components in contemporary information retrieval systems and retrieval-enhanced language models. While recent advancements in semantic representation learning have yielded considerable improvements, practical challenges remain in processing extended textual content. The sequence length capacity of embedding models presents an important technical consideration, as enhanced context processing could potentially improve document comprehension and facilitate more adaptable retrieval implementations. In this technical report, inspired by late chunking Günther et al. 2024 we present a preliminary investigation into chunk-alignment training strategies for text embedding models. This exploratory approach attempts to address the representation learning challenges associated with long-form documents through segment-level feature alignment. Our experimental framework demonstrates the feasibility of processing text sequences up to 128k tokens while preserving performance levels on standard evaluation benchmarks. The current implementation offers two operational modes: a conventional single-vector encoding scheme and an experimental multi-vector variant for extended context processing. The primary contribution lies in developing a chunk-alignment training method (achieved by knowledge distillation) that attempts to extend existing embedding architectures to longer textual sequences. It should be emphasized that this represents an initial exploration rather than a definitive solution, with the current results suggesting potential pathways for further investigation. Our comparative evaluations indicate relatively balanced performance across conventional benchmarks and preliminary long-context test sets, though we acknowledge significant room for improvement in both theoretical framework and practical implementation. 2 Training Methodology Selection of Base Model After investigating many models, we select ModernBERT-Large Warner et al. 2024 as the base model of dewey_en_beta. ModernBERT is a modernized bidirectional encoder-only Transformer model (BERT-style) pre-trained on 2 trillion tokens of English and code data with a native context length of up to 8,192 tokens. ModernBERT leverages recent architectural improvements: • Rotary Positional Embeddings (RoPE) Su et al. 2023 for long-context support. • Local-Global Alternating Attention Team et al. 2024 for efficiency on long inputs. • Unpadding and Flash Attention Portes et al. 2023 Zhang et al. 2024b Zeng et al. 2022 for efficient inference. To scale ModernBERT’s max length to 128k, we change the global_rope_theta to 73780400 according to Men et al. 2024 and https://spaces.ac.cn/archives/10122 . Chunk-Alignment Training Our model can generate three types of embeddings: 1. CLS embedding: A CLS embedding in a BERT-like model. In our training process, it will learn the teacher model’s embedding of the whole text. 2. Chunk embeddings: The mean embeddings of chunk token embeddings. In our training process, it will learn the teacher model’s embedding of each chunk. 3. Mean embedding: The mean embeddings of all token embeddings (excluding CLS, SEP, and prompt token embeddings). It is a special case of chunk embedding (i.e. the chunk is the whole text). For a more comprehensive introduction of our model and distillation framework, we make the following definitions: • teacher : a teacher embedding model with the function encode to encode texts to embeddings • cls\_embed,chunk\_embed_{i} : CLS embedding and chunk embedding of \text{i}_{\text{th}} chunk in student model(i.e. the model to be trained) • cls\_teacher\_embed,chunk\_teacher\_embed_{i} as the whole text teacher embedding and chunk teacher embeddings • s_{x} : The normalized vector representation of a text x produced by the student model. • t_{x} : The vector representation of the same text x , produced by a teacher model. • S_{X} : A matrix of normalized vector representations for a batch of text X produced by the student model. • T_{X} : A corresponding matrix of vector representations for the same batch of text X , generated by a teacher model. cls\_teacher\_embed and chunk\_teacher\_embed_{i} can be obtained by the following equations: cls\_teacher\_embed=teacher.encode(wholetext) chunk\_embed_{i}=teacher.encode(chunk_{i}) After getting cls and chunks embeddings of student and teacher model, we calculate cosine\_loss 1 and similarity\_loss 2 as the final training loss. \mathcal{L}_{cosine\_loss}=\sum_{x}1-s_{x}\cdot t_{x}. (1) \mathcal{L}_{similarity\_loss}=MSE(S_{X}S_{X}^{T},T_{X}T_{X}^{T})) (2) Figure 1: Model architecture Implementation Details We use Linq-Embed-Mistral Kim et al. 2024 as our teacher model. We get unsupervised texts from Infinity-Instruct of Artificial Intelligence BAAI Zhao et al. 2024 Zhang et al. 2024a and fineweb-edu Lozhkov et al. 2024 . We take two strategies to split text to chunks: 1. Split Text by Word 2. RecursiveCharacterTextSplitter in langchain https://python.langchain.com/docs/introduction/ We chose to use the RecursiveCharacterTextSplitter with 70% probability and the Split Text by Word with 30% probability. All the two strategies use a randomized chunk_size(from 64 to 500) and chunk_overlap(from 0.3*chunk\_size to 0.6*chunk\_size ). We are training our model with about 10 million data (about 100 million chunks). The batch size is set to 64, and the learning rate is 1e-4 with a 2000-step warmup and linear decay. We use the StableAdamW optimizer Wortsman et al. 2023 , which improves upon AdamW Loshchilov and Hutter 2019 by adding Adafactor-style update clipping as a per-parameter learning rate adjustment. We total train 2 epochs. We set weight decay to zero. The max length of training data is 2048. 3 Experimental Results English Text Embedding Benchmark MTEB(eng, v2) Enevoldsen et al. 2025 is a new English Massive Text Embedding Benchmark. This benchmark was created to account for the fact that many models have now been finetuned to tasks in the original MTEB, and contains tasks that are not as frequently used for model training. This way the new benchmark and leaderboard can give our users a more realistic expectation of models’ generalization performance. We evaluated our model’s performance on this benchmark. As shown in 1 , while our model supports a context length of 128k tokens, it still achieves competitive results, outperforming most models of comparable size and even some larger-scale models on this particular benchmark. Model Zero-shot Parameters Dimensions Max Tokens Mean (Task) Mean (TaskType) Classification Clustering Pair Classification Reranking Retrieval STS Summarization gemini-embedding-exp-03-07 95% Unknown 3072 8192 73.3 67.67 90.05 59.39 87.7 48.59 64.35 85.29 38.28 jasper_en_vision_language_v1 56% 1B 8960 131072 71.41 66.65 90.27 60.52 88.14 50 56.05 84.37 37.19 gte-Qwen2-7B-instruct NA 7B 3584 32768 70.72 65.77 88.52 58.97 85.9 50.47 58.09 82.69 35.74 stella_en_1.5B_v5 56% 1B 8960 131072 69.43 65.32 89.38 57.06 88.02 50.19 52.42 83.27 36.91 SFR-Embedding-2_R 85% 7B 4096 32768 69.82 65.31 90.54 59.39 88.09 48.99 53.75 80.86 35.54 Linq-Embed-Mistral 95% 7B 4096 32768 69.8 65.29 83 54.07 88.44 49.44 60.14 84.69 37.26 dewey_en_beta 95% 395M 2048 131072 0.68 63.30 81.83 51.75 86.82 46.35 56.32 84.21 35.79 gte-Qwen2-1.5B-instruct NA 1B 8960 32768 67.2 63.26 85.84 53.54 87.52