HSCodeComp : A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application

Yiqian Yang † , Tian Lan † , Qianghuai Jia ∗ , Li Zhu, Hui Jiang, Hang Zhu, Longyue Wang, Weihua Luo, Kaifu Zhang Alibaba International Digital Commerce ∗ * Corresponding Author: Qianghuai Jia ([email protected]) † \dagger Equal Contribution: Yiqian Yang, Tian Lan · 2025

Read HSCodeComp : A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application by Yiqian Yang † , Tian Lan † , Qianghuai Jia ∗ , Li Zhu, Hui Jiang, Hang Zhu, Longyue Wang, Weihua Luo, Kaifu Zhang Alibaba International Digital Commerce ∗ * Corresponding Author: Qianghuai Jia ([email protected]) † \dagger Equal Contribution: Yiqian Yang, Tian Lan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19631.txt.

Introduction

Yiqian Yang and eight co-authors introduce HSCodeComp, a benchmark that tests whether AI search agents can apply hierarchical rule systems, using customs tariff codes as the test case. Every traded product needs a ten-digit Harmonized System code, and assigning one means navigating layered rules with vague boundaries. The dataset holds six hundred thirty-two real products from large e-commerce platforms, coded by human experts.

The best agent tested reached only about forty-seven percent accuracy on the full ten-digit code, against ninety-five percent for the human experts. Adding more test-time computation did not close the gap. The results expose hierarchical rule application as a weak spot of current agents, with direct relevance to law, medicine, and other rule-bound fields.

Catalog record

CREATORYiqian Yang † , Tian Lan † , Qianghuai Jia ∗ , Li Zhu, Hui Jiang, Hang Zhu, Longyue Wang, Weihua Luo, Kaifu Zhang Alibaba International Digital Commerce ∗ * Corresponding Author: Qianghuai Jia ([email protected]) † \dagger Equal Contribution: Yiqian Yang, Tian Lan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19631

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HSCodeComp : A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application Yiqian Yang † , Tian Lan † , Qianghuai Jia ∗ , Li Zhu, Hui Jiang, Hang Zhu, Longyue Wang, Weihua Luo, Kaifu Zhang Alibaba International Digital Commerce ∗ * Corresponding Author: Qianghuai Jia ([email protected]) † \dagger Equal Contribution: Yiqian Yang, Tian Lan Abstract Abstract Effective deep search agents must not only access open-domain and domain-specific knowledge but also apply complex rules—such as legal clauses, medical manuals and tariff rules. These rules often feature vague boundaries and implicit logic relationships, making precise application challenging for agents. However, this critical capability is largely overlooked by current agent benchmarks. To fill this gap, we introduce HSCodeComp , the first realistic, expert-level e-commerce benchmark designed to evaluate deep search agents in hierarchical rule application. In this task, the deep reasoning process of agents is guided by these rules to predict 10-digit Harmonized System Code (HSCode) of products with noisy but realistic descriptions. These codes, established by the World Customs Organization, are vital for global supply chain efficiency. Built from real-world data collected from large-scale e-commerce platforms, our proposed HSCodeComp comprises 632 product entries spanning diverse product categories, with these HSCodes annotated by several human experts. Extensive experimental results on several state-of-the-art LLMs, open-source, and closed-source agents reveal a huge performance gap: best agent achieves only 46.8% 10-digit accuracy, far below human experts at 95.0%. Besides, detailed analysis demonstrates the challenges of hierarchical rule application, and test-time scaling fails to improve performance further. https://github.com/AIDC-AI/Marco-Search-Agent https://huggingface.co/datasets/AIDC-AI/HSCodeComp Figure 1 : Left: Recent benchmarks reveal the increasing knowledge complexity and capability requirements for agents. Right: 10-digit HSCode accuracy of state-of-the-art baseline largely lags behind human experts (46.8% < 95.0%), proving the challenges of hierarchical rule application. The closed-source agent ( \star ) is evaluated on the subset due to API unavailability. 1 Introduction Deep search agents have demonstrated significant value in solving complex real-world problems, where robust external knowledge utilization constitutes a critical capability [ Wu et al., 2025 , Tao et al., 2025 , Li et al., 2025b ] . To evaluate this capability, numerous established benchmarks are proposed to assess agents in utilizing open-domain data ( e.g., GAIA [ Mialon et al., 2023b ] and BrowseComp [ Wei et al., 2025 ] ) and domain-specific data ( e.g., WebMall [ Peeters et al., 2025a ] , FinSearchComp [ Hu et al., 2025a ] and MedBrowseComp [ Yu et al., 2025b ] ). Beyond open-domain and domain-specific data, agents also need to effectively apply rules that encode human expert knowledge, particularly in scenarios like law, medical and e-commerce [ Li et al., 2025a , Chen et al., 2025b , Yao et al., 2022 , Chollet et al., 2025 ] . For instance, legal case adjudication require interpreting abstract legal provisions, and accurate e-commerce product classification in depends on tariff rules [ Grainger, 2024 ] . Previous works have defined rule application as using specific logical rules with supporting facts to derive conclusions [ Wang et al., 2024 , Servantez et al., 2024 ] . In contrast, we define it as a core capability for deep search agents, where human-written rules are systematically applied to guide complex reasoning and decision-making [ Sadowski and Chudziak, 2025 ] . Building on this observation, we categorize knowledge data for deep search agents into three levels (Figure 1 , left), with increasing knowledge complexity: (1) Level 1: Open-domain Data - Tests understanding and deep reasoning abilities of agents on long-form web content. Established benchmarks include GAIA [ Mialon et al., 2023b ] and BrowseComp [ Wei et al., 2025 ] ; (2) Level 2: Structured Data - Assesses agents to precisely utilize structured data such as databases and knowledge graphs, as seen in domain-specific benchmarks like WebMall [ Peeters et al., 2025a ] , MedBrowseComp [ Chen et al., 2025b ] and FinSearchComp [ Hu et al., 2025a ] ; (3) Level 3: Rule Data - Evaluates agents to apply complex and abstract rules [ Chollet et al., 2025 ] . This level presents two key challenges: (a) making accurate decisions when rules contain vague natural language descriptions [ Sadowski and Chudziak, 2025 ] ; and (b) reasoning about logical dependencies among rules, such as exception clauses and cross-category relationships [ Guha et al., 2023 ] . Despite the importance of rule application in real-world scenarios, current agent benchmarks largely overlook its evaluation. To fill this gap, we introduce HSCodeComp (short for the H armonized S ystem Code (HSCode) Comp etition), the first realistic, expert-level e-commerce benchmark designed to evaluate agents in predicting complete 10-digit Harmonized System Code (HSCode) of the product, using hierarchical rules ( e.g., eWTP tariff rules 1 1 1 https://www.ewtp.com/web/smart/hscode ). HSCodes organize products through a hierarchical structure spanning over 5,000 distinct codes across multiple classification levels, representing the global standard for classifying traded international goods, established by the World Customs Organization and implemented across more than 200 countries for customs clearance and tariff determination [ Grainger, 2024 , Nath et al., 2025 ] . Built from the data of the large-scale e-commerce platforms, our proposed HSCodeComp comprises 632 carefully curated product entries, encompassing 27 unique HS chapters and 32 distinct first-level categories. These HSCodes have been rigorously annotated by multiple e-commerce domain experts, ensuring that HSCodeComp is expert-level. Accurately predicting the exact 10-digit HSCode presents significant challenges: agents must perform multi-hop hierarchical reasoning with complex tariff rules while processing noisy but realistic product descriptions that often contain abbreviations, language variations, or incomplete information. Extensive experiments on the state-of-the-art baselines, including 14 advanced foundation models, 6 advanced open-source agent systems and 3 closed-source agent systems, demonstrate that HSCode prediction task remains a substantial challenge for current AI approaches. As shown in the Figure 1 (right), even the best-performing system (SmolAgent [ Roucher et al., 2025 ] with GPT-5) achieves only 46.8% accuracy, substantially below the 95.0% accuracy attained by human experts. Further detailed analysis reveals that existing agent systems lack critical capabilities required for this complex hierarchical rule applications. Notably, test-time scaling approach—which has proven effective in other reasoning tasks [ Guo et al., 2025 , Liu et al., 2025 ] —fail to improve performance on HSCodeComp . These observations demonstrate the challenging nature of our proposed HSCodeComp , highlighting the need for more effective designs of agent systems. To facilitate future research, we will publicly release codes and the benchmark dataset of HSCodeComp . 2 Related Works 2.1 Previous Works in HSCode Prediction Previous works treat HSCode prediction as the e-commerce text classification task [ Grainger, 2024 ] , using pre-trained BERT models [ Liao et al., 2024 , Shubham et al., 2022 ] or Large Language Models (LLMs) prompting [ Hussain and Ahmed, 2023 ] . However, these approaches fail to leverage domain-specific knowledge, especially the rules written by human experts [ Hussain and Ahmed, 2023 , Judy, 2024 ] . Besides, existing HSCode benchmarks face two critical limitations [ Judy, 2024 , Lee et al., 2024 , Stassin et al., 2023 ] : (1) they are typically constructed from publicly accessible customs rulings, suffering from data leakage; (2) they are not released. In contrast, our released HSCodeComp is collected

Read the complete text