COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding
Xiaozhe Li, Tianyi Lyu, Siyi Yang, Yizhao Yang, Yuxi Gong, Jinxuan Huang, Ligao Zhang, Zhuoyi Huang, Qingwen Liu, Tongji University, Stanford University, CurrentsAI Research · 2026
Read COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding by Xiaozhe Li, Tianyi Lyu, Siyi Yang, Yizhao Yang, Yuxi Gong, Jinxuan Huang, Ligao Zhang, Zhuoyi Huang, Qingwen Liu, Tongji University, Stanford University, CurrentsAI Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21329.txt.
Introduction
Li and co-authors introduce COIN-Bench, a benchmark for collective intent understanding in language models. Models follow single instructions well, but distilling shared intent from noisy public discussion is untested. Their live updating benchmark works in the consumer domain. It treats intent as a layered structure, from plain scenarios to deep causal reasoning. Scoring mixes rule based checks with a model judge, plus hierarchical structuring and retrieval based verification. Twenty leading models handle surface aggregation well. They still struggle with the depth that complex intent synthesis needs.
Catalog record
| CREATOR | Xiaozhe Li, Tianyi Lyu, Siyi Yang, Yizhao Yang, Yuxi Gong, Jinxuan Huang, Ligao Zhang, Zhuoyi Huang, Qingwen Liu, Tongji University, Stanford University, CurrentsAI Research |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66796 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21329
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
COINBench: Moving Beyond Individual Perspectives to Collective Intent Understanding Xiaozhe Li, Tianyi Lyu, Siyi Yang, Yizhao Yang, Yuxi Gong, Jinxuan Huang, Ligao Zhang, Zhuoyi Huang, Qingwen Liu, Tongji University, Stanford University, CurrentsAI Research Abstract Understanding human intent is a high-level cognitive challenge for Large Language Models (LLMs), requiring sophisticated reasoning over noisy, conflicting, and non-linear discourse. While LLMs excel at following individual instructions, their ability to distill Collective Intent —the process of extracting consensus, resolving contradictions, and inferring latent trends from multi-source public discussions—remains largely unexplored. To bridge this gap, we introduce COIN-Bench , a dynamic, real-world, live-updating benchmark specifically designed to evaluate LLMs on collective intent understanding within the consumer domain. Unlike traditional benchmarks that focus on transactional outcomes, COIN-Bench operationalizes intent as a hierarchical cognitive structure (from explicit scenarios to deep causal reasoning). We implement a robust evaluation pipeline that synergizes rule-based method with an LLM-as-the-Judge approach. This framework incorporates COIN-Tree for hierarchical cognitive structuring and retrieval-augmented verification ( COIN-RAG ) to ensure expert-level precision in analyzing raw, collective human discussions. An extensive evaluation of 20 state-of-the-art LLMs across four dimensions— depth, breadth, informativeness, and correctness —reveals that while current models handle surface-level aggregation, they struggle with the analytical depth required for complex intent synthesis. COIN-Bench establishes a new standard for advancing LLMs from passive instruction-followers to expert-level analytical agents capable of deciphering the collective voice of the real world. See our project page on COIN-Bench . Figure 1: Motivation for COIN-Bench . Existing benchmarks predominantly rely on individual perspectives and synthetic data, which fail to assess model performance in deciphering complex, real-world collective intent. We propose COIN-Bench , a large-scale, live-updating benchmark that evaluates collective intent understanding across four key dimensions: depth, breadth, accuracy, and informativeness. 1 Introduction The emergence of Large Language Models (LLMs) 24 ; 9 ; 10 has catalyzed a fundamental paradigm shift in artificial intelligence, moving beyond mere text generation toward sophisticated reasoning and real-world problem-solving 28 . Current state-of-the-art models exhibit remarkable proficiency in structured, task-oriented environments, as evidenced by their performance on benchmarks such as SWE-bench 14 for software engineering, SPIDER2.0 17 for SQL generation, and GAIA 22 for tool-augmented reasoning. However, these evaluations predominantly focus on the model as a reasoning agent interacting with a environment , leaving a pivotal dimension of intelligence largely unexplored: understanding collective human intent. This task requires the sophisticated ability to distill coherent consensus from the noisy, fragmented, and often conflicting perspectives inherent in a population. Real-world public discourse, such as community debates or consumer feedback, is characterized by "swarm-like" intelligence. Unlike linear instructions, these discussions involve an intricate interplay of divergent goals, emotional nuances, and implicit assumptions. To truly comprehend this environment, an LLM must go beyond parsing individual sentences; it must synthesize multi-source signals, resolve contradictions, and infer latent trends within an evolving discourse. This capability—acting as an expert-level analyst capable of navigating complex public opinion—represents a high-level cognitive frontier that current evaluation frameworks fail to capture. To rigorously investigate this capability, we identify the consumer domain as an ideal microcosm for Collective Intent . Consumer discussions are far more than transactional records; they embody grounded yet complex human reasoning, rich in the noise and redundancy inherent to collective intelligence. While prior benchmarks have explored social or emotional dimensions 25 ; 3 ; 23 , they often rely on hand-crafted or semi-synthetic datasets that lack the "raw noise" of real-world discourse. Furthermore, recent consumer-focused benchmarks like Shop-R1 35 primarily treat intent as a surface-level prediction of shopper actions (e.g., purchase vs. non-purchase). We argue that such transactional outcomes overlook the deeper Collective Intent —the multi-layered cognitive structure distilled from a diverse population. There is a pressing need for a benchmark that shifts the focus from simple action prediction to the hierarchical synthesis of collective human voices. To bridge this gap, we introduce COIN-Bench (Collective Intent Benchmark), a large-scale, dynamic, and live-updating framework designed to evaluate LLMs on their synthesis of real-world collective consumer intent. COIN-Bench encompasses over 200k opinions across 9 major domains and 54 sub-categories, spanning 1,400+ products. Crucially, we depart from passive evaluation (e.g., multiple-choice questions) in favor of an Active Probing Paradigm . We task LLMs with acting as "meta-analysts" to reconstruct chaotic public discourse into structured questionnaires. This paradigm is underpinned by three core cognitive pillars: • Consensus Extraction : Designing valid questionnaires requires models to implicitly perform denoising and clustering, identifying salient topics amidst the noise. • COIN-Tree for Hierarchical Resolution : Collective intent is inherently stratified. We map model outputs onto a hierarchical cognitive topology to strictly distinguish between surface-level aggregation (L1–L3) and deep causal reasoning or trend prediction (L4–L5). • COIN-RAG for Grounded Verification : To ensure correctness, we employ a retrieval-augmented verification pipeline that penalizes hallucinations by ensuring all inferred consensus is strictly grounded in the raw discussion context. Evaluating 20 diverse LLMs reveals a critical performance dichotomy: while reasoning-enhanced models (e.g., OpenAI o1/o3) demonstrate superior depth in penetrating complex cognitive layers, a significant gap remains in their ability to achieve broad and expert-level analytical synthesis. Our findings suggest that even the most advanced models struggle with the multifaceted nature of collective intent, highlighting substantial room for future development. In summary, our contributions are as follows: • A Large-Scale, Live Benchmark : We introduce COIN-Bench , featuring 200k+ real-world consumer discussions across 1,400+ products. Its dynamic nature ensures real-time updates and mitigates data contamination. • Robust Active Evaluation Pipeline : We propose the Active Probing Paradigm, which transforms traditional passive structural questioning into a process of model-led self-exploration and autonomous Q&A. By integrating the CoIn-Tree with the CoIn-RAG verification framework, this pipeline ensures high-precision assessment, fundamentally shifting the evaluation focus from passive response generation to the active, structured reconstruction of collective intent. • Extensive Empirical Analysis : We provide a comprehensive evaluation of 20 leading LLMs, revealing fundamental limitations in current models’ ability to synthesize deep, multi-source human intent. 2 BENCHMARK CONSTRUCTION 2.1 Data Curation Constructing a benchmark for collective intent requires a corpus that reflects the noisy, fragmented, and contradictory nature of real-world public discourse. We established an automated curation pipeline designed to retrieve high-density discussions while filtering out irrelevant noise. The process consists of three integrated stages: Hybrid Acquisition (Search & Retrieval). To maximize coverage, we employ a hybrid retrieval strategy combining semantic vector search with LLM-augmented API queries. For a given focal product (seed entity), the system first generates context-aware keywords using an LLM to expand the search space. It then retrieves discussion threads from diverse open-source web platforms. To ensure semantic relevance, we utilize vector embeddings to measure similarity between the seed topic and retrieved content, retaining only threads surpassing a strict similarity threshold. Denoising and Filtering. Raw web data is inherently unstructured. We apply a cascade of quality filters: (1) Rule-based Filtering removes low-information content (e.g., extremely short texts <20 chars); (2) Semantic Filtering employs an LLM-based verifier to exclude discussions that drift from the core topic; (3) Temporal Alignment prioritizes recent discussions to ensure the intent reflects current market realities. This pipeline ensures the final dataset preserves the complexity of swarm intelligence while maintaining high contextual fidelity. 2.2 Data Statistics Table 1: Comparison with existing related benchmarks. "Real-world" indicates