Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark
Sondos Mahmoud Bsharat ∗ * ,, Mukul Ranjan ∗ * ,, Aidar Myrzakhan ∗ * ,, Jiacheng Liu, Bowei Guo, Shengkun Tang, Zhuang Liu, Yuanzhi Li, Zhiqiang Shen † · 2025
Read Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark by Sondos Mahmoud Bsharat ∗ * ,, Mukul Ranjan ∗ * ,, Aidar Myrzakhan ∗ * ,, Jiacheng Liu, Bowei Guo, Shengkun Tang, Zhuang Liu, Yuanzhi Li, Zhiqiang Shen † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-20786.txt.
Description
Sondos Mahmoud Bsharat and co-authors build a benchmark for language models on phones in this 2025 paper. Phone users query models in ways desktop users do not. Phones also cap model size through tight storage and compute budgets.
Their set, Mobile-MMLU, holds over sixteen thousand questions in eighty mobile-related fields. Items are multiple choice and ignore answer order, and they cover daily tasks like recipes and travel. A harder pro subset raises the bar further. Scores weigh speed, energy use, memory, and answer quality, not accuracy alone. Privacy and on-device work also count, since phones hold sensitive personal data close at hand.
Catalog record
| CREATOR | Sondos Mahmoud Bsharat ∗ * ,, Mukul Ranjan ∗ * ,, Aidar Myrzakhan ∗ * ,, Jiacheng Liu, Bowei Guo, Shengkun Tang, Zhuang Liu, Yuanzhi Li, Zhiqiang Shen † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-20786