From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service
Haoyu He Jinyu Zhuang Haoran Chu, Shuhang Yu J& T AI Group, Hao Wang Kunpeng Han, J&T Express, School of Information Science, Technology, ShanghaiTech University {robert.zhuang,wayne.yu,lion.han}@jtexpress.com ,{hehy12024,chuhr2023,wanghao1}@shanghaitech.edu.cn · 2026
Read From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service by Haoyu He Jinyu Zhuang Haoran Chu, Shuhang Yu J& T AI Group, Hao Wang Kunpeng Han, J&T Express, School of Information Science, Technology, ShanghaiTech University {robert.zhuang,wayne.yu,lion.han}@jtexpress.com ,{hehy12024,chuhr2023,wanghao1}@shanghaitech.edu.cn (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-23172.txt.
Introduction
He and co-authors build a benchmark for multilingual intent classification in logistics customer service. Most benchmarks use machine-translated text, which is cleaner than real user queries. Their set holds about thirty thousand real, de-identified queries. They were drawn from a far larger log with quality checks and human review. A two-level label tree organizes the intents. Paired native and translated test sets expose the evaluation gap. Translated tests overstate performance, mainly on rare intents and cross-lingual transfer, they find.
Catalog record
| CREATOR | Haoyu He Jinyu Zhuang Haoran Chu, Shuhang Yu J& T AI Group, Hao Wang Kunpeng Han, J&T Express, School of Information Science, Technology, ShanghaiTech University {robert.zhuang,wayne.yu,lion.han}@jtexpress.com ,{hehy12024,chuhr2023,wanghao1}@shanghaitech.edu.cn |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24262 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.23172
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service Haoyu He Jinyu Zhuang Haoran Chu, Shuhang Yu J& T AI Group, Hao Wang Kunpeng Han, J&T Express, School of Information Science, Technology, ShanghaiTech University {robert.zhuang,wayne.yu,lion.han}@jtexpress.com ,{hehy12024,chuhr2023,wanghao1}@shanghaitech.edu.cn Abstract Multilingual intent classification is central to customer-service systems on global logistics platforms, where models must process noisy user queries across languages and hierarchical label spaces. Yet most existing multilingual benchmarks rely on machine-translated text, which is typically cleaner and more standardized than native customer requests and can therefore overestimate real-world robustness. We present a public benchmark for hierarchical multilingual intent classification constructed from real logistics customer-service logs. The dataset contains approximately 30K de-identified, stand-alone user queries curated from 600K historical records through filtering, LLM-assisted quality control, and human verification, and is organized into a two-level taxonomy with 13 parent and 17 leaf intents. English, Spanish, and Arabic are included as seen languages, while Indonesian, Chinese, and additional test-only languages support zero-shot evaluation. To directly measure the gap between synthetic and real evaluation, we provide paired native and machine-translated test sets and benchmark multilingual encoders, embedding models, and small language models under flat and hierarchical protocols. Results show that translated test sets substantially overestimate performance on noisy native queries, especially for long-tail intents and cross-lingual transfer, underscoring the need for more realistic multilingual intent benchmarks. 1 1 1 Current resources are available at https://anonymous.4open.science/r/MICCS , and will be officially released on Hugging Face following the OpenReview process. 1 Introduction Intent classification is a core problem in natural language understanding and underpins many interactive systems, including customer-service automation, virtual assistants, and tool-using language models ( 13 ; 3 ; 18 ) . In multilingual customer-service settings, the task is particularly challenging: semantically equivalent intents may be expressed quite differently across languages and dialects, while real user queries are often short, noisy, and domain-specific. On global logistics platforms, systems must route requests about tracking, delivery issues, address changes, refunds, and claims, often under hierarchical label taxonomies ( 19 ; 12 ; 17 ) . Although existing multilingual benchmarks have enabled substantial progress, they only partly reflect these deployment conditions. Widely used resources such as MTOP and MASSIVE provide broad language coverage, but much of their multilingual data comes from machine translation or controlled data collection rather than naturally occurring customer requests ( 15 ; 11 ) . Other customer-service datasets are more realistic in domain, but often focus on multi-turn dialogue, sentiment, search, or matching tasks rather than stand-alone intent-routing queries with hierarchical labels ( 4 ; 12 ; 22 ) . As a result, current benchmarks may underrepresent the lexical variability, shorthand, code-mixing, and long-tail label distributions that characterize production traffic ( 1 ; 9 ) . This mismatch also affects evaluation. Machine-translated test sets are typically cleaner and more standardized than native user text, and may therefore yield overly optimistic estimates of multilingual robustness ( 1 ; 9 ) . Measuring this synthetic-to-native gap requires matched evaluation settings in which the underlying intent is fixed while the surface form differs between native and translated queries. To address this gap, we introduce a public benchmark for hierarchical multilingual intent classification built from real logistics customer-service logs. The dataset contains approximately 30K de-identified labeled queries curated from roughly 600K historical records through rule-based filtering, LLM-assisted quality control, and human verification. It is organized into a two-level taxonomy with 13 parent intents and 17 leaf intents. English, Spanish, and Arabic are included as seen languages, while Indonesian, Chinese, and additional test-only languages support zero-shot evaluation. Because the benchmark is derived from native, stand-alone user queries used for routing, it preserves the noise and distributional properties of real customer-service traffic. To explicitly study the gap between synthetic and real multilingual evaluation, we additionally provide paired native and machine-translated test sets. We benchmark multilingual encoders, embedding models, and small language models under both flat and hierarchical protocols. Across model families, translated evaluation consistently yields higher performance than native evaluation, with especially large gaps for long-tail intents and cross-lingual transfer. These results suggest that translated benchmarks can substantially overstate robustness on real multilingual user traffic. In summary, our contributions are as follows: • We release a public real-world benchmark for hierarchical multilingual intent classification in logistics customer service, together with a practical curation pipeline that combines de-identification, filtering, LLM-assisted quality control, and human verification. • We provide paired native and machine-translated test sets for direct, controlled measurement of the synthetic-to-native evaluation gap. • We benchmark multilingual encoders, embedding models, and small language models under flat and hierarchical evaluation settings, with a focus on long-tail prediction, noisy native text, and cross-lingual generalization. • We show empirically that machine-translated evaluation can systematically overestimate model performance relative to native-query evaluation. 2 Dataset Construction We construct a multilingual intent benchmark from historical customer-service logs. The released data consist of de-identified user queries paired with a practical two-level intent taxonomy. Figure 1: Construction pipeline for the benchmark. 2.1 Construction Pipeline Source data. The raw pool comprises historical customer-service interactions. From this pool, we extract user utterances that express identifiable service intents. The released benchmark contains approximately 30K labeled queries. English, Spanish, and Arabic are used as seen languages, while Indonesian, Chinese, and additional held-out languages are reserved for zero-shot transfer evaluation. De-identification and cleaning. Before annotation and release, we remove personally identifiable information, such as tracking numbers and detailed addresses, using rule-based filters followed by manual checks. We then exclude bot-generated templates, exact and near duplicates, malformed queries, and utterances without clear intent. Deduplication combines string matching with embedding-based semantic similarity. This process reduces annotation noise while preserving naturally occurring linguistic variation. Taxonomy construction. We group the cleaned queries by operational function and user goal, and iteratively refine the label set to balance coverage, label separability, and per-class support. The final taxonomy is organized as a two-level hierarchy with 13 parent intents and 17 leaf intents. Details of the clustering configuration are provided in Appendix B and Figure 3 . Annotation and quality control. We use a semi-automatic annotation pipeline with human verification. Candidate samples are pre-screened with LLM-assisted filtering and consistency checks, and are then reviewed by two annotators for each language. A 5% subset is double-annotated for test-set construction; annotator agreement on this subset is 95%. Remaining disagreements are adjudicated by a third expert, and samples without consensus are removed. 3 Dataset Characteristics 3.1 Overview Appendix Table 3 summarizes the corpus statistics. The release contains approximately 30K queries, split into train/dev/test at a ratio of 20:1:1. A representative de-identified example is shown below: To improve label generation stability in generative classification, we prepend a fixed prefix (e.g., “i1”) to all label IDs ( 24 ) . 3.2 Native vs. Synthetic Evaluation A key feature of the benchmark is the distinction between native and synthetic multilingual evaluation. Native evaluation uses a 674-example human-verified subset of held-out multilingual traffic, with cross-language semantic duplicates removed. For synthetic evaluation, we construct paired machine-translated test sets from a reliability-filtered source subset and verify the translations manually. These translated sets preserve intent semantics while removing much of the noise present in native traffic. We therefore treat them as a controlled synthetic baseline rather than a direct