Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents
Minbyul Jeong · 2026
Read Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents by Minbyul Jeong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27595.txt.
Introduction
Most tests of AI web agents reward depth, hiding one answer behind a chain of clues. Breadth is rarely measured outside English. Breadth means listing each member of a set with details. Minbyul Jeong built Ko-WideSearch in 2026 to fill that gap. It is a Korean benchmark of set-listing tasks, produced by a build-and-verify pipeline.
Each task names a parent entity. Examples are a television season, a dynasty, or an election. The agent must list the membership and fill a table of attributes for each item. The benchmark holds 228 tables over 190 entities. They span sixteen categories and three tiers. Scoring covers items, columns, and rows. Jeong tested twenty web agents. The failure pattern was steady. Agents recovered the set but not the rows. Item scores sat near ninety-three, while full-row scores sat near fifty-four. More searching and more spending did not close the gap.
Catalog record
| CREATOR | Minbyul Jeong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27595
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents Minbyul Jeong Abstract Web-agent benchmarks overwhelmingly measure depth—pinning one obscure answer behind a chain of constraints—while breadth, exhaustively enumerating a closed set and filling each item’s attributes, is barely evaluated, especially outside English. Breadth is also hard to build: certifying that a gold set is complete and every cell correct is far costlier than checking a single answer. I introduce Ko-WideSearch , a Korean breadth-search benchmark built by an automated synthesize-and-verify pipeline. Each task names a set-parent entity—a TV season, a dynasty, a league, an administrative region, an election—and asks for its full membership plus a per-item attribute table, graded by Item-, Column-, and Row-F1. It spans 228 tables over 190 entities and sixteen categories across three difficulty tiers, set by two structural knobs I dial independently—table width and a 2-D composite key—so cross-product membership climbs from 0% to 100% across the tiers. A single normalization-aware comparator is shared between gold construction and grading, so stable date and count columns are not over-dropped on formatting alone. Across twenty web agents, the failure is consistent: agents recover the set but not the rows (e.g. Item-F1 92.8 against Row-F1 53.7), accuracy falls steadily as the knobs harden, and neither more search nor more spend closes the gap. Broken down by cell, the hard part is finding the right value, not formatting it: open-ended free-text cells fail most, while cells with a standard answer such as a date or a name usually come out right. Introduction Figure 1: The benchmark and its running example. Left: tasks span Korean categories (rows) and three difficulty tiers (columns); each cell is one real task, with a strip of its columns beneath—the membership-key column(s) shaded, one box marking a list, two a cross-product grid. Reading left to right, tasks gain columns to fill and their membership turns from a list into a grid. Right: the Government/ Hard cell shown in full—a hard-set example I reuse as the paper’s running example: every Korean metropolitan province’s elected head in the 7th and 8th local elections. Its rows are (province \times round) pairs (every province in every election, 17\times 2=34 rows), and per-row attributes such as each winner’s age are looked up off the results page (cross-source). Four of sixteen categories are shown. Ask a web agent to list every low-cost carrier flying in Korea as of June 2026, and for each one its parent company, the year it launched, its main hub, and how many aircraft it operates today. A capable agent has to do two things at once. It must recover the complete set—nine carriers, where dropping Aero K or inventing a tenth is immediately wrong—; and it must fill, for each row, a cell that lives on a different page, because the current fleet count is not printed beside the airline’s name. Neither half is a hard single-fact lookup. The difficulty is that the answer is a whole table, and it has to be exhaustive and internally complete. This is breadth —a different axis from the one today’s browsing-agent benchmarks measure. The dominant paradigm is depth: BrowseComp [ Wei et al. 2025 ] and its Chinese and Korean siblings BrowseComp-ZH [ Zhou et al. 2025 ] and K-BrowseComp [ Lee et al. 2026 ] hide a single short answer behind several constraints, and the agent traverses a multi-hop or parallel-constraint path to recover it. WideSearch [ Wong et al. 2025 ] showed that even frontier agents fail set-valued tasks, not because the information is hidden but because of scale and exhaustiveness: they drop rows, mis-fill cells, and lose track of the set boundary. Breadth, in other words, stresses a capability that depth benchmarks never exercise—holding and completing a structured set across many pages. Two gaps motivate this work. First, breadth-search evaluation barely exists outside English: Korean agentic benchmarks are scarce [ Lee et al. 2026 ] , yet browsing ability is uniquely language- and culture-bound—an agent must navigate Korean sources whose structure, terminology, and search conventions differ from English ones [ Kim et al. 2024a , Son et al. 2025 ] . Second, breadth poses a construction problem of its own: certifying that a gold set is complete and that every attribute cell is correct is far harder than checking one answer, and doing it at scale by hand is prohibitive—the original WideSearch was hand-built at 200 tables and high annotation cost. I address both with Ko-WideSearch , a Korean breadth-search benchmark built by an automated synthesize-and-verify pipeline. Each task names a set-parent entity—a TV season, a dynasty, a league, an administrative region, an election—and asks for the full membership plus an attribute table, graded by Item-, Column-, and Row-F1. I organize the benchmark—228 tables in all—into three difficulty tiers along two structural axes I dial independently: table width and a 2-D composite key (Figure 1 ). My contributions are fourfold. (1) I introduce Ko-WideSearch : 228 Korean breadth-search tables—4,262 gold rows and 14,560 attribute cells over 190 set-parent entities and sixteen categories—every table web-sourced and decontaminated. (2) I build it with an automated synthesize-and-verify pipeline that makes Korean breadth-search evaluation scalable while keeping the gold trustworthy: a build agent enumerates each gold table by exhaustive web search, three independent gates certify non-memorizability, completeness, and cross-source attribute verification, and two difficulty knobs manufacture WideSearch-grade hard breadth—2-D membership rises from 0% in Easy to 100% in Hard , and median width from three to seven columns. (3) I find that a naïve attribute cross-check over-drops stable columns on formatting alone and, because the same comparator grades model output, would otherwise mis-score agents, so I fix both with one normalization-aware comparator; separately, I web-ground the single-page-versus-cross-source sourcing label, which an LLM guesses only roughly 72% accurately against the live web. (4) I adopt a leakage-aware release: because an agent’s own search can surface posted gold and lift the answer, I follow the held-out precedent of GAIA [ Mialon et al. 2023 ] and BrowseComp [ Wei et al. 2025 ] and open-source the pipeline and scorer under MIT while distributing the evaluation data by request, keeping the set off the surface agents search so Korean breadth search can be evaluated and re-grown. Related Work Web agents, tool use, and browsing benchmarks. A browsing agent is a language model that acts—reasoning, calling tools, and reading back what it finds—a loop established by ReAct [ Yao et al. 2023 ] and by tool-augmented models that teach themselves to invoke APIs [ Schick et al. 2023 , Patil et al. 2024 , Qin et al. 2024 ] . Pointed at the open web, such agents have been studied as browser-assisted question answerers [ Nakano et al. 2021 ] and as navigators of real and simulated sites [ Yao et al. 2022 , Deng et al. 2023 , He et al. 2024 , Zhou et al. 2024 ] , and stress-tested by agent suites such as AgentBench [ Liu et al. 2024 ] , GAIA [ Mialon et al. 2023 ] , and AssistantBench [ Yoran et al. 2024 ] ; more recently, search has been folded into reasoning itself [ Li et al. 2025 ] . The hardest browsing benchmarks