PeopleSearchBench: A Multi-Dimensional Benchmark for Evaluating AI-Powered People Search Platforms

Wei Wang*, Tianyu Shi* †, Shuai Zhang* †, Boyang Xia, Zequn Xie, Chenyu Zeng, Qi Zhang, Lynn Ai, Yaqi Yu, Kaiming Zhang, Feiyue Tang LessieAI research team [email protected] [email protected] https://github.com/LessieAI/people-search-bench · 2026

Read PeopleSearchBench: A Multi-Dimensional Benchmark for Evaluating AI-Powered People Search Platforms by Wei Wang*, Tianyu Shi* †, Shuai Zhang* †, Boyang Xia, Zequn Xie, Chenyu Zeng, Qi Zhang, Lynn Ai, Yaqi Yu, Kaiming Zhang, Feiyue Tang LessieAI research team [email protected] [email protected] https://github.com/LessieAI/people-search-bench (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27476.txt.

Introduction

Shi and co-authors launch PeopleSearchBench, a test for people search tools. Such tools now serve hiring, sales, and expert hunts. The bench holds 119 queries in four task types. Queries span several languages. Each query splits into clear checkable criteria. Live web checks then judge each returned person. That method agrees with human raters at kappa 0.84. Four platforms were tested on three scoring axes. Multi-source agents beat single-domain tools by wide gaps.

Catalog record

CREATORWei Wang*, Tianyu Shi* †, Shuai Zhang* †, Boyang Xia, Zequn Xie, Chenyu Zeng, Qi Zhang, Lynn Ai, Yaqi Yu, Kaiming Zhang, Feiyue Tang LessieAI research team [email protected] [email protected] https://github.com/LessieAI/people-search-bench
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60314
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27476

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PeopleSearchBench: A Multi-Dimensional Benchmark for Evaluating AI-Powered People Search Platforms Wei Wang*, Tianyu Shi* †, Shuai Zhang* †, Boyang Xia, Zequn Xie, Chenyu Zeng, Qi Zhang, Lynn Ai, Yaqi Yu, Kaiming Zhang, Feiyue Tang LessieAI research team [email protected] [email protected] https://github.com/LessieAI/people-search-bench Abstract AI-powered people search platforms are increasingly used in recruiting, sales prospecting, and professional networking, yet there is still no standard, comprehensive benchmark for evaluating and comparing their performance. To address this gap, we present PeopleSearchBench , an open-source benchmark that evaluates four people search platforms on 119 real-world queries among four distinct scenarios: corporate recruiting, B2B sales prospecting, expert search with deterministic answers, and influencer/KOL discovery. A central contribution of this work is Criteria-Grounded Verification , an evaluation pipeline for factual relevance assessment. The pipeline extracts explicit, verifiable criteria from each query and checks whether each returned person satisfies them using live web search. This process produces binary relevance judgments grounded in factual verification, rather than the more subjective quality scores often used in holistic LLM-as-judge evaluation. We evaluate systems along three dimensions: Relevance Precision, measured by padded nDCG@10; Effective Coverage, measured by task completion and qualified result yield; and Information Utility, measured by the completeness and usefulness of the returned profiles. These dimensions are averaged with equal weight to produce an overall score. Across four people search platforms, our benchmark shows that Lessie, a specialized AI people search agent, achieves the strongest overall performance. The overall score of Lessie is 65.2, which is 18.5% higher than that of the second-ranked system. It is also the only system to achieve 100% task completion across all 119 queries. To support reproducibility and reliability, we also report confidence intervals, human validation of the verification pipeline (Cohen’s κ = 0.84 \kappa=0.84 ), ablation studies on key design choices, and full documentation of queries, prompts, and normalization procedures. All code, query definitions, and aggregated results are publicly available at https://github.com/LessieAI/people-search-bench . 0 0 footnotetext: † Corresponding authors. 1 Introduction People search—the task of finding individuals who match a specific combination of role, skills, location, and domain expertise—is a common workflow in recruiting, sales, and marketing. As AI-powered platforms increasingly automate this process, comparing their effectiveness has become important but remains difficult. Despite rapid adoption, there is still no widely accepted methodology for evaluating people-search systems in a rigorous and reproducible manner. Existing benchmarks for information retrieval [ 12 , 16 ] and question answering [ 9 ] do not adequately address this setting, where outputs are real individuals, valid answers are often non-exhaustive, and key profile attributes require independent verification. The challenge extends beyond the absence of a labeled dataset to the evaluation methodology itself. Standard benchmarks typically rely on pre-defined relevance labels, while holistic LLM-as-judge approaches often depend on subjective overall assessments [ 20 ] . Neither is fully adequate for people search. Most queries admit many correct answers, and practical utility depends not only on retrieving relevant individuals, but also on returning enough qualified candidates with verifiable and navigable profile information to support immediate downstream action. As a result, evaluation must account for multiple criteria and support factual verification against external evidence. We introduce PeopleSearchBench, an open-source benchmark containing 119 queries in four languages (English, Portuguese, Spanish, Dutch) grouped into four commercially relevant scenarios: Recruiting (30 queries), B2B Prospecting (32), Expert/Deterministic Search (28), and Influencer/KOL Discovery (29). Evaluation is conducted through our Criteria-Grounded Verification pipeline, which breaks down each query into explicit checkable criteria and verifies each result against those criteria through live web search. This produces binary factual judgments instead of subjective quality scores, making the evaluation process more reproducible and less prone to bias. We apply our evaluation framework to four platforms that represent distinct architectural approaches to people search: a specialized AI search agent, a structured search API, an AI-powered recruiting platform, and a general-purpose AI agent. The results reveal that Lessie, the specialized AI people search agent, achieves the highest overall score (65.2) with an 18.5% lead over the second-ranked platform, and is the only system that maintains 100% task completion across all 119 queries. Performance varies substantially across query types: recruiting queries are relatively competitive for platforms with access to large professional databases, while influencer discovery shows the widest performance gap between systems. The original conference version of this work presented the core benchmark design and main experimental results. In this extended technical report, we address the need for greater reproducibility and statistical rigor by adding several key contributions: (1) bootstrap confidence intervals and paired significance tests for all scores; (2) human validation of the verification pipeline on 200 person-query pairs (Cohen’s \kappa = 0.84); (3) the complete set of 119 queries with metadata and the normalization schema; (4) all evaluation prompts and execution protocols; (5) cost and latency analysis; (6) systematic error analysis with case studies; and (7) ablation studies on the qualified-result threshold, dimension weights, top-K, and partial credit. We believe these additions make this benchmark a valuable resource for the research community as AI-powered people search continues to evolve. 2 Related Work This section surveys four areas that inform our benchmark design and identifies the specific gap each leaves open for people search evaluation. Information retrieval benchmarks. The TREC benchmarks [ 13 ] established the template for modern information retrieval evaluation with test collections and pooling-based relevance judgments. More recently, BEIR [ 12 ] broadened the scope to include 18 heterogeneous datasets for zero-shot retrieval evaluation, and MTEB [ 11 ] extended this approach to embedding models across a wide range of tasks.However, all of these benchmarks evaluate document-level or passage-level retrieval where a result is judged as a single text unit. People search differs in that each result is a real individual with multiple independently verifiable attributes (role, employer, location, skills), and relevance cannot be reduced to a single topical-match judgment. LLM-based evaluation. 20 demonstrated that LLM judges can approximate human preferences in open-ended text generation tasks, and follow-up work has addressed limitations including positional bias [ 14 ] , multi-dimensional rubrics [ 8 ] , and score calibration [ 15 ] . A shared limitation of existing work is that judges still rely primarily on parametric knowledge to assess output quality. For people search, parametric knowledge is insufficient because a person’s current employer, title, and location change over time and must be verified against external sources. Our Criteria-Grounded Verification pipeline addresses this by decomposing evaluation into explicit factual checks grounded in live web search. Entity-centric search. Entity retrieval from knowledge bases [ 6 ] and enterprise corpora [ 2 ] was studied in the INEX and SemSearch tracks, which assume fixed entity collections with known attributes. 1 surveyed expertise retrieval within closed organizational corpora, and 5 described LinkedIn’s talent search, evaluated using platform-specific engagement data. Neither setting supports cross-platform comparison over open-web results. Moreover, these efforts predate LLM-powered autonomous search agents and do not provide evaluation protocols suited to their output formats and capabilities. Our benchmark addresses both gaps by using externally verifiable criteria applied uniformly across architecturally diverse platforms. Agentic AI evaluation. Recent agent benchmarks cover software engineering [ 7 ] , web interaction [ 21 ] , and general task completion [ 10 ] . These primarily evaluate binary

Read the complete text