PingPong : A Natural Benchmark for Multi-Turn Code-Switching Dialogues

Mohammad Rifqi Farhansyah, Hanif Muhammad Zhafran, Farid Adilazuarda, Shamsuddeen Hassan Muhammad, Maryam Ibrahim Mukhtar, Nedjma Ousidhoum, Genta Indra Winata, Ayu Purwarianti, Alham Fikri Aji · 2026

Read PingPong : A Natural Benchmark for Multi-Turn Code-Switching Dialogues by Mohammad Rifqi Farhansyah, Hanif Muhammad Zhafran, Farid Adilazuarda, Shamsuddeen Hassan Muhammad, Maryam Ibrahim Mukhtar, Nedjma Ousidhoum, Genta Indra Winata, Ayu Purwarianti, Alham Fikri Aji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17277.txt.

Introduction

Mohammad Rifqi Farhansyah and co-authors present PingPong, a test set for code-switching talk. Code-switching means mixing languages in one chat. Most of the world does it each day. Yet few test sets show its true mess. PingPong holds natural multi-party dialogues that switch codes. It covers five language mixes. Some chats use three languages at once.

A fair test set shows where language models fail real users. Models that ace one-language tests can stumble here. PingPong gives builders a true target for mixed talk. It can guide training for chat tools in mixed-language homes and cities. The work fills a clear gap in how models are judged.

Catalog record

CREATORMohammad Rifqi Farhansyah, Hanif Muhammad Zhafran, Farid Adilazuarda, Shamsuddeen Hassan Muhammad, Maryam Ibrahim Mukhtar, Nedjma Ousidhoum, Genta Indra Winata, Ayu Purwarianti, Alham Fikri Aji
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17277

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PingPong : A Natural Benchmark for Multi-Turn Code-Switching Dialogues Mohammad Rifqi Farhansyah, Hanif Muhammad Zhafran, Farid Adilazuarda, Shamsuddeen Hassan Muhammad, Maryam Ibrahim Mukhtar, Nedjma Ousidhoum, Genta Indra Winata, Ayu Purwarianti, Alham Fikri Aji Abstract Code-switching is a widespread practice among the world’s multilingual majority, yet few benchmarks accurately reflect its complexity in everyday communication. We present PingPong , a benchmark for natural multi-party code-switching dialogues covering five language-combination variations, some of which are trilingual. Our dataset consists of human-authored conversations among 2 to 4 participants covering authentic, multi-threaded structures where replies frequently reference much earlier points in the dialogue. We demonstrate that our data is significantly more natural and structurally diverse than machine-generated alternatives, offering greater variation in message length, speaker dominance, and reply distance. Based on these dialogues, we define three downstream tasks: Question Answering , Dialogue Summarization , and Topic Classification . Evaluations of several state-of-the-art language models on PingPong reveal that performance remains limited on code-switched inputs, underscoring the urgent need for more robust NLP systems capable of addressing the intricacies of real-world multilingual discourse. 1 Introduction Figure 1: Illustration of dialogue samples from PingPong . The bottom panel highlights how each turn corresponds to previous turns, as labeled in our dataset. Dataset Open-Source CS Type #Comp. Lang. #Regions #Gen. Task Conv. SEAME 38 ✓ Bilingual 0 1 0 ✓ GLUECoS 30 ✓ Bilingual 0 2 1 ✗ CommonDost 46 ✓ Bilingual 0 2 1 ✓ MalayalamMixSentiment 18 ✓ Bilingual 1 1 0 ✗ TamilMixSentiment 19 ✗ Bilingual 0 1 0 ✗ hinglishNorm 39 ✓ Bilingual 0 1 1 ✓ LinCE 4 ✓ Bilingual 2 3 1 ✗ CSCS 11 ✓ Bilingual 1 1 0 ✗ CanVEC 44 ✓ Bilingual 1 1 1 ✗ ArzEn 26 ✗ Bilingual 1 1 1 ✗ PHINC 58 ✓ Bilingual 0 1 1 ✗ MIPE 24 ✗ Bilingual 0 1 1 ✗ HinGE 59 ✓ Bilingual 0 1 1 ✗ TCS 61 ✗ Bilingual 0 1 2 ✓ GupShup 41 ✓ Bilingual 0 1 1 ✓ DOSA 53 ✓ Bilingual 1 1 0 ✗ TweetTaglish 28 ✓ Bilingual 1 1 0 ✗ BaSCo 7 ✓ Bilingual 1 1 0 ✓ MHE 52 ✗ Trilingual 1 1 0 ✗ ASCEND 37 ✓ Bilingual 0 1 1 ✓ L3Cube-HingCorpus 43 ✓ Bilingual 0 1 1 ✗ CroCoSum 76 ✓ Bilingual 0 1 1 ✗ CS-PREF 34 ✓ Bilingual 1 2 1 ✗ CS-Sum 60 ✓ Bilingual 1 2 1 ✗ CodeMixQA 68 ✓ Bilingual 16 3 1 ✗ PingPong ✓ Bilingual Trilingual 4 3 2 ✓ Table 1: Comparison of code-switching benchmark datasets. Open-Source indicates public availability of the dataset. CS Type denotes the code-switching setting. # Comp. Lang. counts under-studied languages beyond high-resource ones (e.g., English, Hindi, Chinese). # Regions reports geographic coverage (e.g., South Asian-English, European-English). # Gen. Tasks indicates the number of generative tasks. Conv. denotes whether the dataset is conversational. Code-switching, the practice of alternating between two or more languages within a single conversation, is a pervasive linguistic phenomenon in contemporary multilingual societies 47 ; 42 ; 15 ; 9 . With the global number of multilingual speakers surpassing that of monolinguals, the frequency of code-switching in both informal and formal communication continues to rise 64 ; 25 ; 66 ; 69 . Despite recent progress in multilingual language models 73 ; 8 ; 62 ; 74 , their ability to handle code-switched dialogue remains underexplored and insufficiently evaluated 35 ; 71 ; 2 . Although several code-switching benchmarks have been introduced in the past, these resources are becoming outdated, and therefore, increasingly misaligned with the current era of LLMs and are less reflective of the everyday realities of code-switching 5 ; 31 . To address this gap, we present PingPong , a novel benchmark for code-switching in dialogues. Our benchmark consists of conversations in which fluent multilingual speakers naturally code-switch between languages while discussing predefined topics, resulting in multiple language-combination variations through multi-party conversational interactions.. PingPong comprises three downstream tasks, including Question Answering , Dialogue Summarization , and Multi-label Topic Classification . In addition, the benchmark provides broad language coverage, spanning both high-resource and low-resource language combinations, with certain language combination further distinguished by differences in writing scripts. In summary, our contributions are as follow: Figure 2: Overview of PingPong dataset construction. We recruit annotators who speak five language combinations and group them into small teams of 2 to 4 participants. Each group engages in multi-party dialogue creation, after which downstream annotations are performed. The final output is the curated PingPong dataset, focusing on natural code-switched dialogues for multiple tasks. • We introduce PingPong , a novel benchmark for evaluating natural multi-party code-switching in conversational settings, designed to capture the fluid dynamics of multilingual communication. • PingPong consists of authentic interactions among 2 to 4 speakers discussing everyday topics, with conversations ranging from brief exchanges of 17 turns to extended dialogues of up to 189 turns. Reflecting real conversational behavior, speakers may produce long-distance replies by responding to earlier messages while alternating turns. • PingPong covers five language combinations spanning high-resource, low-resource, and diverse-script languages, enabling evaluation across a broad range of multilingual scenarios. • Our evaluation framework spans three downstream tasks— Question Answering , Dialogue Summarization , and Topic Classification —providing a holistic assessment of language understanding and generation under code-switching conditions. • We conduct extensive experiments with both open-source and proprietary LLMs, revealing their limitations and highlighting persistent challenges in modeling code-switched, multi-party conversations. 2 Related Work Research on code-switching has been studied extensively by linguists for decades 56 . In NLP, code-switching has been explored across a wide range of tasks 67 , including language identification 6 ; 16 ; 45 ; 20 , named entity recognition 70 ; 3 ; 29 , part-of-speech tagging 57 ; 17 ; 12 , and sentiment analysis 78 ; 54 . In parallel, dataset development and shared evaluations have been actively promoted through initiatives such as CALCS 27 ; 10 and the FIRE shared tasks 33 ; 1 ; 40 . More recently, this line of work has begun to consolidate around shared benchmarks that enable systematic comparison across models, language combinations, and tasks. However, despite the growing body of work, several limitations remain across existing code-switching datasets (Table 1 ). From a sustainability perspective, a number of datasets are not publicly released 19 ; 26 ; 24 . In terms of linguistic diversity, most benchmarks focus exclusively on bilingual code-switching, with only a single dataset explicitly addressing trilingual scenarios 52 . Moreover, both the language and regional coverage remain narrow, as code-switching phenomena in many languages are still under-explored in NLP 18 ; 4 ; 11 ; 44 ; 26 ; 53 ; 28 ; 7 ; 52 ; 34 ; 60 ; 68 . In addition, most existing datasets fall short of capturing the conversational nature of real-world code-switching, which often emerges in spontaneous and interactive settings 38 ; 46 ; 39 ; 61 ; 41 ; 37 . Evaluation tasks are also frequently limited to traditional benchmarks that no longer reflect contemporary NLP challenges 30 ; 43 . To address these gaps, we introduce PingPong , a large-scale open-source evaluation suite that expands language and

Read the complete text