PISA-Bench : The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models

2025

Read PISA-Bench : The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24792.txt.

Description

Patrick Haller, Fabio Barth, Jonas Golde, Georg Rehm, and Alan Akbik built a new bench to test vision and language models in many tongues at once. Past benches lean on made up content or stick to English, since hand checks of translations are slow and dear. Their bench draws on PISA tests, a school skill test used in more than eighty lands.

Each item keeps the text, image, and answer picks from the source tests, with a tag for the kind of ask. Items were turned into five more tongues, for six in all, laid side by side. Tests of top models showed small models in the non English tongues lag by a lot.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24792

Browse the full library · About the Tower of Babel