MARCA: A Checklist-Based Benchmark for Multilingual Web Search.
Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires, Celio Larcher, Hugo Abonizio, Marcos Piau, Roseval Malaquias Junior, Rodrigo Nogueira, Thiago Laitz · 2026
Read MARCA: A Checklist-Based Benchmark for Multilingual Web Search. by Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires, Celio Larcher, Hugo Abonizio, Marcos Piau, Roseval Malaquias Junior, Rodrigo Nogueira, Thiago Laitz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14448.txt.
Introduction
Almeida and eight colleagues present MARCA, a bilingual English-Portuguese benchmark for testing language models on web-based information seeking. Fifty-two hand-written multi-entity questions pair with hand-checked checklist rubrics that score answer completeness and correctness. Fourteen models run under two setups: a basic path with direct search and scraping, and an orchestrator path that splits tasks across delegated subagents. Orchestration often improves coverage, scores vary widely across models, and transfer between English and Portuguese proves uneven.
Catalog record
| CREATOR | Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires, Celio Larcher, Hugo Abonizio, Marcos Piau, Roseval Malaquias Junior, Rodrigo Nogueira, Thiago Laitz |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23306 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MARCA: A Checklist-Based Benchmark for Multilingual Web Search. Thales Sales Almeida, Giovana Kerche Bonás, Ramon Pires, Celio Larcher, Hugo Abonizio, Marcos Piau, Roseval Malaquias Junior, Rodrigo Nogueira, Thiago Laitz Abstract Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesize complete answers. While recent benchmarks evaluate web-browsing and agentic tool use, multilingual settings, and Portuguese in particular, remain underexplored. We present MARCA , a bilingual (English and Portuguese) benchmark for evaluating LLMs on web-based information seeking. MARCA consists of 52 manually authored multi-entity questions, paired with manually validated checklist-style rubrics that explicitly measure answer completeness and correctness. We evaluate 14 models under two interaction settings: a Basic framework with direct web search and scraping, and an Orchestrator framework that enables task decomposition via delegated subagents. To capture stochasticity, each question is executed multiple times and performance is reported with run-level uncertainty. Across models, we observe large performance differences, find that orchestration often improves coverage, and identify substantial variability in how models transfer from English to Portuguese. The benchmark is available at https://github.com/maritaca-ai/MARCA 1 Introduction Large language models (LLMs) are increasingly used as general-purpose sources of information: users ask for facts, comparisons, and up-to-date summaries, often expecting the system to both retrieve evidence and synthesize a reliable answer. This paradigm depends critically on models’ ability to search and read the web effectively and to consolidate information without omissions or hallucinations. Recent work has begun to evaluate such capabilities via web-browsing and agentic benchmarks [ 17 , 28 , 33 , 8 ] , but most evaluations remain centered on English. Portuguese, despite being one of the most widely spoken languages globally with over 250 million native speakers, has very limited benchmarks and studies that evaluate web search and evidence-grounded answering capabilities. We introduce MARCA ( M aritaca A I R esearch C hecklist ev A luation), a benchmark for evaluating LLMs’ ability to find and verify information on the web in a multilingual setting. MARCA focuses on multi-entity questions that require systems to gather evidence from multiple webpages and produce structured, complete answers. Each question is paired with a manually curated checklist rubric, enabling fine-grained evaluation of correctness and coverage. Beyond language, we study how interaction design shapes performance. We evaluate models in two frameworks: (i) a Basic setting where the model directly uses web_search and web_scrape , and (ii) an Orchestrator setting where the model can delegate sub-questions to subagents that perform web interactions. Our contributions are: • A new bilingual benchmark : parallel English and Portuguese versions of MARCA with 52 manually authored questions spanning 9 domains for evaluating web-search and evidence-grounded answering. • Checklist-based evaluation : manually authored rubrics that measure both correctness and completeness on questions involving multiple entities. • Framework comparison : an evaluation of Basic tool use versus Orchestrator-style delegation. 2 Related Work 2.1 Web Browsing and Search Benchmarks Evaluating LLMs as systems that actively search and synthesize information from the web has received growing attention. WebGPT [ 17 ] introduced a browser-assisted QA paradigm where models issue queries, navigate pages, and produce grounded answers with human feedback. More recently, BrowseComp [ 28 ] proposed a benchmark of 1,266 questions designed to require persistent, multi-step browsing to find hard-to-retrieve, entangled information. Extensions of this line include BrowseComp-ZH [ 32 ] , which targets the Chinese web and reveals that browsing ability can degrade substantially outside English, BrowseComp-Plus [ 6 ] , which introduces a fixed corpus for fairer and more reproducible evaluation of deep-research agents, and MM-BrowseComp [ 12 ] , which extends the paradigm to multimodal queries. Beyond single-answer retrieval, a broader line of work evaluates LLMs as tool-using agents. WebArena [ 33 ] and Mind2Web [ 8 ] test navigation of real or simulated web interfaces, while WebChoreArena [ 14 ] extends these to tedious, memory- and calculation-intensive tasks. AgentBench and ToolLLM assess broader tool-calling capabilities across diverse APIs. These efforts primarily measure task completion rather than the quality and completeness of the information retrieved. Search Arena [ 13 ] takes a complementary approach, collecting over 24,000 crowdsourced interactions with search-augmented LLMs to analyze human preferences in realistic information-seeking settings. LiveNewsBench [ 31 ] introduces a scalable, regularly updated benchmark that automatically generates fresh QA pairs from recent news articles, addressing the risk that static benchmarks conflate memorization with genuine search ability. On the deep-research front, LiveResearchBench [ 27 ] evaluates long-form, citation-grounded report generation using rubric-based criteria, while Deep Research Bench [ 9 ] provides a frozen web snapshot for reproducible evaluation of multi-step research agents. Despite this rapidly growing landscape, most of these benchmarks are English-centric. Multilingual retrieval resources such as mMARCO [ 5 ] and MIRACL [ 30 ] evaluate passage retrieval across languages, but none evaluate end-to-end web-interacting agents. 2.2 Regional and Linguistic Variation in LLM Performance Recent work has shown that LLM capabilities can vary considerably across cultural, regional, and linguistic contexts, motivating language-specific evaluation. The Blend benchmark [ 16 ] measures models’ understanding of daily activities across countries, showing that models are better attuned to regions with stronger digital representation. WorldBench [ 15 ] uses socioeconomic data to evaluate how well models recall information about different countries, finding higher error rates for underrepresented regions such as Africa and the Middle East. TiEBe [ 2 ] evaluates factual recall of notable events by country and observes a strong correlation between a country’s GDP and model performance. INCLUDE [ 23 ] constructed a large-scale evaluation suite of nearly 200,000 QA pairs from local exam sources across 44 languages, demonstrating significant performance differentials across languages and regional knowledge types, and crucially showing that translating English benchmarks is insufficient because it ignores the cultural and regional knowledge of the environments where multilingual systems would be deployed. ALM-Bench [ 26 ] extends this to the multimodal setting, evaluating language models across 100 languages with culturally grounded visual questions; accuracy drops from 88.4% (English) to 50.8% (Amharic) for GPT-4o, revealing a consistent performance gap between high- and low-resource languages. These disparities also manifest in agentic and web-search settings [ 4 , 23 ] . BrowseComp-ZH [ 32 ] found meaningful degradation in web-browsing performance when moving from English to Chinese, demonstrating that search benchmarks designed for English do not transfer straightforwardly to other information ecosystems. For Portuguese specifically, the evaluation landscape has expanded but remains fragmented. Structured exams such as the Brazilian National University Entrance Exam (ENEM) [ 21 ] , university entrance exams [ 3 , 24 ] , and bar exam evaluations [ 22 ] provide standardized assessments. More recently, PoETa v2 [ 1 ] introduced a comprehensive benchmark of 44 Portuguese tasks, providing the most extensive evaluation of Portuguese LLMs to date. Despite this progress, no existing Portuguese benchmark evaluates web-search and evidence-grounded answering capabilities, which is the setting where language-specific retrieval dynamics, content availability, and query formulation strategies can meaningfully affect performance. MARCA addresses this gap by providing a bilingual benchmark that allows performance differences to be directly attributed to language, enabling fine-grained diagnosis of when and why models fail in non-English web interactions. 3 Methodology This section describes the construction of