LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation

Ananya Raval, Aravind Narayanan, Vahid Reza Khazaie, Shaina Raza · 2025

Read LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation by Ananya Raval, Aravind Narayanan, Vahid Reza Khazaie, Shaina Raza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07274.txt.

Description

Ananya Raval, Aravind Narayanan, Vahid Khazaie, and Shaina Raza test fairness across languages in vision models, in work from 2025. Large multimodal models train on vast image text sets with thin language coverage. Thin coverage can bias answers and treat language groups unfairly in daily use.

Their benchmark, called LinguaMark, holds close to seven thousand image text pairs. Pairs span eleven languages and five social traits for balanced checks. Scores cover bias, answer relevance, and faithfulness to the shown image. Closed source models led overall scores in the reported runs. One open model matched well across languages and traits, which shows open paths can compete.

Catalog record

CREATORAnanya Raval, Aravind Narayanan, Vahid Reza Khazaie, Shaina Raza
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07274

Browse the full library · About the Tower of Babel