Multimodal Evaluation of Russian-language Architectures
Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov, Artem Safin, Maria TikhonovaAlexander Kharitonov , Yulia Lyakh , Petr Surovtsev , Denis Shevelev , Vildan Saburov , Vasily Konovalov , Elisei Rykov , Ivan Sviridov , Amina Miftakhova , Ilseyar Alimova, Alexander Panchenko, Alexander Kapitanov, Alena · 2025
Read Multimodal Evaluation of Russian-language Architectures by Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov, Artem Safin, Maria TikhonovaAlexander Kharitonov , Yulia Lyakh , Petr Surovtsev , Denis Shevelev , Vildan Saburov , Vasily Konovalov , Elisei Rykov , Ivan Sviridov , Amina Miftakhova , Ilseyar Alimova, Alexander Panchenko, Alexander Kapitanov, Alena (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15552.txt.
Description
Artem Chervyakov, Ulyana Isaeva, and sixteen co-authors introduce MERA Multi, the first open benchmark for multimodal models that work in Russian. Multimodal models handle text, images, audio, and video together, yet their skills and risks are still poorly mapped, and no such benchmark existed for Russian. The framework is instruction-based and built to test both general models and single-mode systems.
The benchmark offers a shared taxonomy of multimodal abilities and eighteen tasks built from scratch, with prompts and scoring unified across tasks. Content reflects Russian language and culture. Baseline results cover closed and open models, and watermarking of private sets guards against benchmark leakage. The method can be reused for other Slavic languages.
Catalog record
| CREATOR | Artem Chervyakov, Ulyana Isaeva, Anton Emelyanov, Artem Safin, Maria TikhonovaAlexander Kharitonov , Yulia Lyakh , Petr Surovtsev , Denis Shevelev , Vildan Saburov , Vasily Konovalov , Elisei Rykov , Ivan Sviridov , Amina Miftakhova , Ilseyar Alimova, Alexander Panchenko, Alexander Kapitanov, Alena |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 144 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-15552