Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems
Stefano Cirillo, Domenico Desiato, Giuseppe Polese, Giandomenico Solimando · 2026
Read Benchmarking Google Embeddings 2 against Open-Source Models for Multilingual Dense Retrieval and RAG Systems by Stefano Cirillo, Domenico Desiato, Giuseppe Polese, Giandomenico Solimando (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23618.txt.
Description
Cirillo and colleagues benchmark Google Embeddings 2 against five open-source rivals for multilingual dense retrieval and retrieval-augmented generation. Google's Vertex-AI-hosted bi-encoder, with 2048-token context and task-type conditioning, ranks first on every task, scoring 0.638 average nDCG on BEIR and 0.282 on an Italian RAG bench. But at 231.6 milliseconds median latency it runs about 14 times slower than the fastest local models. Multilingual-E5-large nearly matches it on Italian at 31 milliseconds, while LaBSE trails badly at 0.188, and all models saturate at 32-token chunks.
Catalog record
| CREATOR | Stefano Cirillo, Domenico Desiato, Giuseppe Polese, Giandomenico Solimando |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44319 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23618