MAPS: A Multilingual Benchmark for Global Agent Performance and Security

Omer Hofman 1 , Oren Rachmil 1 , Shamik Bose 1 , Vikas Pahuja 1 , Jonathan Brokman 1 , Toshiya Shimizu 2 , Trisha Starostina 3 , Kelly Marchisio 3 , Seraphina Goldfarb-Tarrant 3 , Roman Vainshtein 1 1 Fujitsu Research of Europe 2 Fujitsu Limited 3 Cohere · 2025

Read MAPS: A Multilingual Benchmark for Global Agent Performance and Security by Omer Hofman 1 , Oren Rachmil 1 , Shamik Bose 1 , Vikas Pahuja 1 , Jonathan Brokman 1 , Toshiya Shimizu 2 , Trisha Starostina 3 , Kelly Marchisio 3 , Seraphina Goldfarb-Tarrant 3 , Roman Vainshtein 1 1 Fujitsu Research of Europe 2 Fujitsu Limited 3 Cohere (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15935.txt.

Introduction

AI agents built on large language models now use tools and memory, but models weaken in languages other than English, and agents inherit that weakness. Users of other languages may face unreliable or unsafe agent behavior. The authors propose MAPS, a multilingual benchmark suite for agent performance and security. It translates four established benchmarks, covering real-world tasks, code generation, math reasoning, and agent security, into eleven languages.

The suite holds 805 unique tasks and 9,660 language-specific instances. Testing shows both performance and security degrade when agents leave English, with the drop varying by task and growing with the amount of translated input. The authors offer MAPS as the first standardized frame for judging multilingual agents and as a push toward agents that serve all users equitably and safely.

Catalog record

CREATOROmer Hofman 1 , Oren Rachmil 1 , Shamik Bose 1 , Vikas Pahuja 1 , Jonathan Brokman 1 , Toshiya Shimizu 2 , Trisha Starostina 3 , Kelly Marchisio 3 , Seraphina Goldfarb-Tarrant 3 , Roman Vainshtein 1 1 Fujitsu Research of Europe 2 Fujitsu Limited 3 Cohere
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15935

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MAPS: A Multilingual Benchmark for Global Agent Performance and Security Omer Hofman 1 , Oren Rachmil 1 , Shamik Bose 1 , Vikas Pahuja 1 , Jonathan Brokman 1 , Toshiya Shimizu 2 , Trisha Starostina 3 , Kelly Marchisio 3 , Seraphina Goldfarb-Tarrant 3 , Roman Vainshtein 1 1 Fujitsu Research of Europe 2 Fujitsu Limited 3 Cohere Abstract Agentic AI systems, which build on Large Language Models (LLMs) and interact with tools and memory, have rapidly advanced in capability and scope. Yet, since LLMs have been shown to struggle in multilingual settings, typically resulting in lower performance and reduced safety, agentic systems risk inheriting these limitations. This raises concerns about the global accessibility of such systems, as users interacting in languages other than English may encounter unreliable or security-critical agent behavior. Despite growing interest in evaluating agentic AI, existing benchmarks focus exclusively on English, leaving multilingual settings unexplored. To address this gap, we propose MAPS, a multilingual benchmark suite designed to evaluate agentic AI systems across diverse languages and tasks. MAPS builds on four widely used agentic benchmarks — GAIA (real-world tasks), SWE-bench (code generation), MATH (mathematical reasoning), and the Agent Security Benchmark (security). We translate each dataset into ten diverse languages, resulting in 805 unique tasks and 8,855 total language-specific instances. Our benchmark suite enables a systematic analysis of how multilingual contexts affect agent performance and robustness. Empirically, we observe consistent degradation in both performance and security when transitioning from English to other languages, with severity varying by task and correlating with the amount of translated input. Building on these findings, we provide actionable recommendations to guide agentic AI systems development and assessment under multilingual settings. This work establishes a standardized evaluation framework, encouraging future research towards equitable, reliable, and globally accessible agentic AI. MAPS benchmark suite is publicly available. 1 1 1 https://huggingface.co/datasets/Fujitsu-FRE/Multilingual-Agentic-AI-Benchmark 1 Introduction LLM-based agentic AI systems combine multi-step reasoning with external tools and memory to solve open-ended tasks such as code generation, web navigation, planning, and transactional services like booking and ordering ( Acharya et al. 2025 ) . By doing so, they extend to complex, real-world problems beyond standard LLM benchmarks. Since such real-world applications serve speakers of diverse languages, maintaining consistent reliability in every language becomes critical. However, since agentic behavior is grounded in LLMs, which often perform inconsistently across languages ( Deng et al. 2023 ; Wang et al. 2023 ) , agents may inherit these multilingual limitations as well, affecting their functionality and trustworthiness. This presents a barrier to equitable access, as non-English users may face degraded responses, incorrect tool actions, or unsafe behaviors—failures that can lead to actual harm in the real world, including erroneous transactions, data corruption, and other security vulnerabilities ( Zhang et al. 2024 ) . To assess emerging agentic systems, various benchmarks have been proposed to evaluate agent performance across a range of tasks ( Mialon et al. 2023 ; Jimenez et al. 2023 ; Chang et al. 2024 ; Xu et al. 2024 ) . However, these benchmarks remain English-only. In contrast to multilingual LLM benchmarks ( Dang et al. 2024 ; Shi et al. 2022 ; Goyal et al. 2022 ) , no equivalent exists for agentic AI tasks—creating a blind spot in our understanding of cross-language performance, safety, and security. Figure 1: MAPS benchmark suite evaluates LLM-based agents across 11 languages and 4 agentic benchmarks covering performance and security. In this paper, we address this gap. We hypothesize that multilingual settings will reveal performance and security gaps in agentic systems that are not captured by the existing, English-only benchmarks. To investigate this, we introduce MAPS, a M ultilingual A gentic AI Benchmark Suite for P erformance and S ecurity. MAPS is based on four established agentic benchmarks: GAIA (real-world tasks) ( Mialon et al. 2023 ) , SWE-bench (code generation) ( Jimenez et al. 2023 ) , MATH (mathematical reasoning) ( Hendrycks et al. 2021 ) , and the Agent Security Benchmark (security) ( Zhang et al. 2024 ) . These benchmarks are extended to ten typologically diverse languages beyond English 2 2 2 MAPS offers evaluation in the following ten languages: German, Spanish, Portuguese (Brazil), Japanese, Russian, Italian, Arabic, Hebrew, Korean, and Hindi. , by employing a hybrid machine and LLM-based translation approach Ki and Carpuat 2024 with extended verification and enhancements. In total, MAPS includes 805 unique tasks, each available in 11 language versions—including the original English and 10 translated variants—for a total of 8,855 instances. An overview of the benchmark structure is shown in Figure 1 . To demonstrate the use of MAPS and test our hypothesis, we selected a leading open-source agent associated with each of the four original benchmarks and applied it to the corresponding multilingual extension. We observed notable declines in both task performance and security when moving from English to other languages, with the severity of these drops varying by task type and correlating with the proportion of non-English input, suggesting that multilingual performance interventions should be targeted based on input composition and task sensitivity. Beyond overall degradation, our findings reveal that multilingual inputs can amplify agentic vulnerabilities in safety-critical tasks, highlighting the need for multilingual risk assessment. These results empirically support our hypothesis and demonstrate the utility of MAPS as a tool for systematic, multilingual evaluation of agentic AI systems. The primary contributions of this paper are threefold: • To the best of our knowledge, we introduce the first multilingual benchmark suite for agentic AI, extending four widely used benchmarks into ten typologically diverse languages for systematic performance and security assessment. • The efficacy and quality of the proposed benchmark are demonstrated through a large-scale evaluation of leading agents as well as human expert verification. • We present the first quantifiable analysis and evidence that multilingual settings reveal critical performance, safety, and security gaps in agentic systems, along with actionable recommendations for improving their development. 2 Background and Related Work 2.1 Agentic AI Benchmarks With the rapid advancement of LLM-based agents, a diverse suite of benchmarks has been developed to assess their autonomy, tool use, planning, and memory integration ( Yao et al. 2024 ; Xu et al. 2024 ; Yehudai et al. 2025 ) . We organized these suites along three primary dimensions. Evaluation objective : performance-oriented benchmarks measure task completion, multi-step reasoning, and correct tool invocation (e.g., AgentBoard ( Chang et al. 2024 ) ), whereas security-focused suites probe robustness to adversarial inputs, jailbreaks, and unsafe behaviors (e.g., AgentHarm ( Andriushchenko et al. 2024 ) ). Agentic task scope : full-agentic evaluations present only problem statements and expected outcomes, requiring end-to-end planning and execution (e.g., GAIA ( Mialon et al. 2023 ) ), while semi-agentic frameworks supply scaffolding, such as code templates or mock APIs, to isolate the LLM’s reasoning and tool-selection core (e.g., AppWorld ( Trivedi et al. 2024 ) . Design and evaluation characteristics : most benchmarks span a limited set of domains (three to five use cases), typically including real-world information retrieval and navigation (e.g., AssistantBench ( Yoran et al. 2024 ) ), code generation (e.g., SWE-Bench ( Jimenez et al. 2023 ) ), reasoning and planning (e.g., MATH ( Hendrycks et al. 2021 ) , Travel Planner

Read the complete text