Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs
Wafa Alghallabi, Ritesh Thawkar, Sara Ghaboura, Ketan More, Omkar Thawakar, Hisham Cholakkal, Salman Khan, Rao Muhammad Anwer, Mohamed bin Zayed University of AI, Australian National University, Aalto University{wafa.alghallabi, sara.ghaboura, omkar.thawakar}@mbzuai.ac.ae https://mbzuai-oryx.github. · 2025
Read Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs by Wafa Alghallabi, Ritesh Thawkar, Sara Ghaboura, Ketan More, Omkar Thawakar, Hisham Cholakkal, Salman Khan, Rao Muhammad Anwer, Mohamed bin Zayed University of AI, Australian National University, Aalto University{wafa.alghallabi, sara.ghaboura, omkar.thawakar}@mbzuai.ac.ae https://mbzuai-oryx.github. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18152.txt.
Description
Wafa Alghallabi and co-authors introduce Fann or Flop, the first benchmark that tests how well large language models understand Arabic poetry, in a 2025 paper. Arabic verse is layered with metaphor and shaped by strict meters across a long history, making it a hard test of deep language grasp. The benchmark gathers poems from twelve historical eras and fourteen genres, from classical forms to free verse.
Each poem comes with explanations probing meaning, metaphor, meter, and cultural context. Leading models, strong on standard Arabic tests, mostly struggled with the poems. The authors release the benchmark and its scoring tools openly, aiming to push Arabic language models toward real interpretive skill.
Catalog record
| CREATOR | Wafa Alghallabi, Ritesh Thawkar, Sara Ghaboura, Ketan More, Omkar Thawakar, Hisham Cholakkal, Salman Khan, Rao Muhammad Anwer, Mohamed bin Zayed University of AI, Australian National University, Aalto University{wafa.alghallabi, sara.ghaboura, omkar.thawakar}@mbzuai.ac.ae https://mbzuai-oryx.github. |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18152