PISA-Bench : The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models
2025
Read PISA-Bench : The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24792.txt.
Description
Patrick Haller, Fabio Barth, Jonas Golde, Georg Rehm, and Alan Akbik built a new bench to test vision and language models in many tongues at once. Past benches lean on made up content or stick to English, since hand checks of translations are slow and dear. Their bench draws on PISA tests, a school skill test used in more than eighty lands.
Each item keeps the text, image, and answer picks from the source tests, with a tag for the kind of ask. Items were turned into five more tongues, for six in all, laid side by side. Tests of top models showed small models in the non English tongues lag by a lot.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24792