LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation
Ananya Raval, Aravind Narayanan, Vahid Reza Khazaie, Shaina Raza · 2025
Read LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation by Ananya Raval, Aravind Narayanan, Vahid Reza Khazaie, Shaina Raza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07274.txt.
Description
Ananya Raval, Aravind Narayanan, Vahid Khazaie, and Shaina Raza test fairness across languages in vision models, in work from 2025. Large multimodal models train on vast image text sets with thin language coverage. Thin coverage can bias answers and treat language groups unfairly in daily use.
Their benchmark, called LinguaMark, holds close to seven thousand image text pairs. Pairs span eleven languages and five social traits for balanced checks. Scores cover bias, answer relevance, and faithfulness to the shown image. Closed source models led overall scores in the reported runs. One open model matched well across languages and traits, which shows open paths can compete.
Catalog record
| CREATOR | Ananya Raval, Aravind Narayanan, Vahid Reza Khazaie, Shaina Raza |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07274