Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark

Sondos Mahmoud Bsharat ∗ * ,, Mukul Ranjan ∗ * ,, Aidar Myrzakhan ∗ * ,, Jiacheng Liu, Bowei Guo, Shengkun Tang, Zhuang Liu, Yuanzhi Li, Zhiqiang Shen † · 2025

Read Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark by Sondos Mahmoud Bsharat ∗ * ,, Mukul Ranjan ∗ * ,, Aidar Myrzakhan ∗ * ,, Jiacheng Liu, Bowei Guo, Shengkun Tang, Zhuang Liu, Yuanzhi Li, Zhiqiang Shen † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20786.txt.

Description

Sondos Mahmoud Bsharat and co-authors build a benchmark for language models on phones in this 2025 paper. Phone users query models in ways desktop users do not. Phones also cap model size through tight storage and compute budgets.

Their set, Mobile-MMLU, holds over sixteen thousand questions in eighty mobile-related fields. Items are multiple choice and ignore answer order, and they cover daily tasks like recipes and travel. A harder pro subset raises the bar further. Scores weigh speed, energy use, memory, and answer quality, not accuracy alone. Privacy and on-device work also count, since phones hold sensitive personal data close at hand.

Catalog record

CREATORSondos Mahmoud Bsharat ∗ * ,, Mukul Ranjan ∗ * ,, Aidar Myrzakhan ∗ * ,, Jiacheng Liu, Bowei Guo, Shengkun Tang, Zhuang Liu, Yuanzhi Li, Zhiqiang Shen †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-20786

Browse the full library · About the Tower of Babel