FilBench : Can LLMs Understand and Generate Filipino?
Lester James V. Miranda, Elyanah Aco, Conner Manuel, Jan Christian Blaise Cruz, Joseph Marvin Imperial · 2025
Read FilBench : Can LLMs Understand and Generate Filipino? by Lester James V. Miranda, Elyanah Aco, Conner Manuel, Jan Christian Blaise Cruz, Joseph Marvin Imperial (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-03523.txt.
Introduction
Lester James Miranda and co-authors build a benchmark to test language models in Filipino in a 2025 paper. Models shine on English tasks, but their skill in Filipino, Tagalog, and Cebuano was largely unmeasured. Their benchmark, FilBench, gathers tasks that reflect research priorities in the Philippines, including cultural knowledge, classic language tasks, reading, and generation.
They tested twenty-seven leading models on the suite. Even the best model scored only about seventy-two percent, leaving much headroom. Several models were weak at reading comprehension and translation. Models trained for Southeast Asian languages did not lead; their best entry scored about sixty-one percent. The authors argue that language-specific benchmarks like theirs are needed to drive progress and inclusion for Philippine languages.
Catalog record
| CREATOR | Lester James V. Miranda, Elyanah Aco, Conner Manuel, Jan Christian Blaise Cruz, Joseph Marvin Imperial |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-03523
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FilBench : Can LLMs Understand and Generate Filipino? Lester James V. Miranda, Elyanah Aco, Conner Manuel, Jan Christian Blaise Cruz, Joseph Marvin Imperial Abstract Despite the impressive performance of LLMs on English-based tasks, little is known about their capabilities in specific languages such as Filipino. In this work, we address this gap by introducing FilBench , a Filipino-centric benchmark designed to evaluate LLMs across a diverse set of tasks and capabilities in Filipino, Tagalog, and Cebuano. We carefully curate the tasks in FilBench to reflect the priorities and trends of NLP research in the Philippines such as Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation. By evaluating 27 state-of-the-art LLMs on FilBench , we find that several LLMs suffer from reading comprehension and translation capabilities. Our results indicate that FilBench is challenging, with the best model, GPT-4o, achieving only a score of 72.23%. Moreover, we also find that models trained specifically for Southeast Asian languages tend to underperform on FilBench , with the highest-performing model, SEA-LION v3 70B, achieving only a score of 61.07%. Our work demonstrates the value of curating language-specific LLM benchmarks to aid in driving progress on Filipino NLP and increasing the inclusion of Philippine languages in LLM development. Figure 1: Overview of FilBench . In order to comprehensively assess the full capabilities of LLMs on Philippine languages, we curate an evaluation suite consisting of 4 categories and 12 subtasks across Filipino, Tagalog, and Cebuano based on the research priorities of the Philippine NLP community (§ 3.1 ). 1 Introduction While large language models (LLMs) have shown impressive performance on a variety of English-based tasks and capabilities, their effectiveness remains largely unexplored for low-resource languages such as Filipino. This knowledge gap exists for two reasons. First, most low-resource languages, especially Filipino-centric benchmarks, developed before the ChatGPT-era ( Gururaja et al. 2023 , —2022) are ill-posed for current LLM evaluation despite their value in understanding language system capabilities. Second, existing multilingual LLM benchmarks either exclude Filipino entirely ( Liu et al. 2025 ; Huang et al. 2025 , inter alia ) or fail to provide sufficient task and/or language diversity ( Susanto et al. 2025 ) . Filipino is an important language to consider for LLM evaluation not only because of its unique linguistic properties such as its voice marking system ( Bardají et al. 2024 ) , but also due to its large speaker population with more than 28 million speakers in the Philippines and over 2 million speakers abroad ( Philippine Statistics Authority 2020 ) . In this work, we perform a comprehensive study of the strengths and limitations of LLM capabilities on Filipino-centric tasks. We introduce an evaluation suite called FilBench , consisting of 4 categories and 12 diverse sub-tasks that are formulated for LLM evaluation. The choice of tasks to include in FilBench is based on our study of research trends and priorities in Filipino NLP (§ 3.1 , § J ). Evaluating models on FilBench reveals significant gaps in LLM performance, for instance, in text generation capabilities. The contributions of this study are three-fold: • We close the resource gap by curating Filipino test sets across four broad task categories: Cultural Knowledge Assessment (CK), Classical NLP (CN), Reading Comprehension (RC), and Generation (GN). We transform these datasets into a unified task format aligned with standard LLM evaluation practices across literature. Our evaluation suite, FilBench , consists of test instances across 4 categories and 12 sub-tasks (§ 3 ). • We bridge the evaluation gap by evaluating 27 state-of-the-art LLMs on FilBench (§ 4 ). We find that the best model, GPT-4o, only achieve around 72.23% aggregated performance while the best Southeast-Asian model, SEA-LION v3 70B, only obtains a score of 61.07%. • We provide analyses and insights to the strengths and weaknesses of LLMs when presented with Filipino-centric tasks and test cases (§ 5 ). Notably, we find that text generation suffers the most, with the lowest scores across models due to failure modes such as hallucination and poor instruction-following. FilBench demonstrates the value of constructing language-specific benchmarks to reveal gaps in language model capabilities and benefit the wider speaker community. More importantly, we hope that this work aids in improving the state of Filipino NLP and increase the inclusion of Philippine languages in LLM development. 2 Background Languages in the Philippines. The Philippines is home to approximately 117 million language speakers across more than 185 distinct languages ( Eberhard et al. 2024 ; McFarland 2008 ; Metila et al. 2016 ) . One of its official languages is Filipino ( FIL ), which is a standardized form of Tagalog ( TGL ) and used mainly in Metro Manila. 1 1 1 The designation of Filipino and Tagalog as separate languages is often a point of contention, although they are linguistically similar ( Villafania 2007 ) . We follow the official view of the Komisyon ng Wikang Filipino (Commision on the Filipino Language) and treat them as separate. Aside from Filipino and Tagalog, Cebuano ( CEB ) is the second most widely spoken language in the Philippines with over 28 million speakers. It is part of the Visayan language family and is spoken mainly in regions of Cebu, Siquijor, and Bohol among many others ( Pilar et al. 2023 ) . As part of the same subgroup of Philippine languages, Tagalog and Cebuano share similar linguistic characteristics such as shared vocabulary and comparable word formulation processes and affixation rules, among others Bacalla 2019 ; Imperial and Kochmar 2023 . In our work, we focus on the these three languages because they cover the majority of Filipino speakers, representing approximately 61% of the country’s population ( Philippine Statistics Authority 2020 ) . Task Formulation in LLM Evaluation. In order to standardize how each test example is presented to an LLM, it must first be formatted into a consistent prompt structure or formulation . Multiple-choice formulation (MCF) is a common standard in evaluating LLMs across a vast array of tasks ( Gu et al. 2024 ; Fourrier et al. 2024 ) . In MCF, a question is posed with answers presented as labeled choices, where scoring is done by comparing the LLM’s choice to the gold label. For evaluating LLMs on generative tasks such as translation, one approach is to write an instruction prompting an LLM to translate a given text from a source language to a target language. Then, the generated output by the LLM is compared against the reference translation using various machine translation metrics ( Papineni et al. 2002 ; Lin 2004 ) . Category Sub-Task Dataset Languages # Instances Classical NLP (CN) Text Classification Dengue Filipino Livelo and Cheng 2018 FIL 4,015 BalitaNLP Buñag and Esquivel 2023 TGL 70,352 SIB-200 Adelani et al. 2024 CEB , FIL 99 Named-Entity Recognition CebuaNER Pilar et al. 2023 CEB 1,310 TLUnified-NER Miranda 2023 TGL 1,579 Universal NER Mayhew et al. 2024 CEB , TGL 105 Sentiment Analysis FiReCS Cosme and De Leon 2023 FIL 7,340 Cultural Knowledge Assessment (CK) Regional Knowledge INCLUDE Romanou et al. 2024 TGL 510 Factual Knowledge Global MMLU Singh et al. 2024 TGL 14,042 Cultural Values KALAHI Montalan et al. 2024 TGL 150 Word-sense Disambiguation StingrayBench Cahyawijaya et al. 2024