Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement
Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma, Kosuke Takahashi, Takahiro Omi, Kosuke Arima, Takuto Asakura, Chung-Chi Chen, Tatsuya Ishigaki · 2026
Read Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement by Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma, Kosuke Takahashi, Takahiro Omi, Kosuke Arima, Takuto Asakura, Chung-Chi Chen, Tatsuya Ishigaki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22517.txt.
Introduction
Hirota and eight colleagues ask whether an automatic judge of business ideas should mimic a pooled consensus or model each expert alone. They release PBIG-DATA, roughly 3,000 expert scores over 300 patent-based product ideas across six business dimensions. Experts disagree sharply on fine-grained scores but agree more on coarse choices, a pattern the authors read as structured diversity rather than noise. Comparing rubric-only, aggregate, and personalized judges, they find personalized judges track their target evaluator more closely across dimensions and model sizes.
Catalog record
| CREATOR | Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma, Kosuke Takahashi, Takahiro Omi, Kosuke Arima, Takuto Asakura, Chung-Chi Chen, Tatsuya Ishigaki |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36520 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22517
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement Wataru Hirota, Tomoki Taniguchi, Tomoko Ohkuma, Kosuke Takahashi, Takahiro Omi, Kosuke Arima, Takuto Asakura, Chung-Chi Chen, Tatsuya Ishigaki Abstract Evaluating LLM-generated business ideas is often harder to scale than generating them. Unlike standard NLP benchmarks, business idea evaluation relies on multi-dimensional criteria such as feasibility, novelty, differentiation, user need, and market size, and expert judgments often disagree. This paper studies a methodological question raised by such disagreement: should an automatic judge approximate an aggregate consensus, or model evaluators individually? We introduce PBIG-DATA, a dataset of approximately 3,000 individual scores across 300 patent-grounded product ideas, provided by domain experts on six business-oriented dimensions: specificity, technical validity, innovativeness, competitive advantage, need validity, and market size. Analyses show substantial expert disagreement on fine-grained ordinal scores, while agreement is higher under coarse selection, suggesting structured heterogeneity rather than random noise. We then compare three judge configurations: a rubric-only zero-shot judge, an aggregate judge conditioned on mixed evaluator histories, and a personalized judge conditioned on the target evaluator’s scoring history. Across dimensions and model sizes, personalized judges align more closely with the corresponding evaluator than aggregate judges, and evaluator agreement correlates with similarity of judge-generated reasoning only under personalized conditioning. These results indicate that pooled labels can be a fragile target in pluralistic evaluation settings and motivate evaluator-conditioned judge designs for business idea assessment. rmTeXGyreTermesX [*devanagari]rmLohit Devanagari [*arabic]rmNoto Sans Arabic 1 Introduction Large language models (LLMs) make it easy to generate large numbers of product ideas. The practical bottleneck is not generation but evaluation. Organizations must decide which ideas are sufficiently concrete, technically feasible, differentiated, and commercially meaningful to justify further investment. These decisions are often made by multiple experts with different backgrounds, such as technical reviewers and business strategists. Unlike tasks where evaluation targets factual correctness or task completion, business idea evaluation is multi-dimensional and judgment-driven. Even when reviewers follow the same rubric, they may apply different assumptions about feasibility, novelty, risk, or market opportunity. In practice, this leads to persistent disagreement across experts. Such disagreement is not necessarily annotation noise; it reflects heterogeneous standards about what constitutes a promising idea. Most LLM-as-a-Judge approaches assume that a single scoring standard exists. Under this assumption, labels from multiple reviewers are aggregated, and the judge is optimized to reproduce this pooled signal. However, when expert disagreement is systematic, aggregation may obscure meaningful differences between evaluators. This raises a methodological question for evaluation design: should an automatic judge approximate an aggregate consensus, or should it model evaluators individually? This paper studies this question in the context of business-oriented product idea evaluation. We introduce PBIG-DATA, a dataset of approximately 3,000 individual scores across 300 patent-grounded product ideas, provided by domain experts along six business-oriented dimensions: specificity, technical validity, innovativeness, competitive advantage, need validity, and market size. Our empirical findings reveal a substantial gap between common evaluation assumptions and actual expert behavior. First, inter-annotator agreement on fine-grained ordinal scores is often close to zero and occasionally negative, indicating that expert scoring does not converge to a single shared scale. At the same time, agreement increases when the task is framed as coarse selection, suggesting that disagreement reflects heterogeneous but structured standards rather than random noise. Second, this heterogeneity has direct implications for LLM-as-a-Judge design. We compare three judge configurations: a rubric-only zero-shot judge, an aggregate judge conditioned on mixed evaluator histories, and a personalized judge conditioned on the target evaluator’s own scoring history. Across dimensions, personalized judges align more closely with the corresponding evaluator than aggregate judges. This indicates that individual evaluators are internally consistent even when they disagree with each other. Taken together, these results suggest that business idea evaluation is inherently pluralistic. Treating pooled labels as a single ground truth may be an inadequate target for automatic judging. Modeling evaluator-specific standards provides a more faithful representation of expert judgment, but it also highlights how far current judge designs are from fully supporting heterogeneous decision processes in real-world settings. This paper makes the following contributions: 1. A dataset of expert-scored product ideas under business-oriented criteria. 2. A quantitative analysis of structured expert disagreement. 3. Evidence that aggregate and personalized judge designs behave differently under heterogeneous standards, with implications for evaluation methodology in practical ideation systems. 2 Related Work 2.1 Human Evaluation in Creative Generation Human evaluation plays a central role in creative natural language generation tasks. Prior surveys have shown that inter-annotator agreement is often low in creative settings, especially when tasks involve open-ended judgments such as novelty and emotional impact ( 5 ) . 1 argue that low agreement does not necessarily indicate unreliable data, but may reflect irreducible variability in human language interpretation. They recommend complementing agreement metrics with correlation-based analyses to better understand evaluation reliability. Recent large-scale studies further confirm that even domain experts disagree substantially when evaluating generative outputs. For example, 13 report significant variation among NLP researchers assessing the novelty of LLM-generated research ideas. These findings suggest that disagreement is a structural property of creative evaluation rather than annotation noise. Our work extends this line of investigation to business-oriented product idea evaluation, focusing specifically on how disagreement affects the design of automatic judges. 2.2 LLMs for Ideation LLMs have increasingly been used to support ideation tasks across domains. Interactive systems such as Wordcraft demonstrate how language models can assist human writers in creative writing workflows ( 21 ) . In scientific and research ideation, multi-agent LLM frameworks have been explored to improve diversity and feasibility of generated proposals ( 17 ) . Large-scale evaluations of idea generation benchmarks also examine LLM performance on novelty and feasibility metrics ( 13 ) . While these studies focus primarily on improving generation quality, evaluation methodology remains comparatively underexplored. Most ideation benchmarks rely either on automatic scoring or on pooled human labels treated as ground truth. In contrast, our work centers on the evaluation stage itself and investigates how heterogeneous expert standards influence judge modeling. 2.3 LLM-as-a-Judge and Evaluation Modeling LLMs are increasingly used as automatic judges for diverse NLP tasks. Early evidence shows that large models can approximate human preferences under certain prompting strategies ( 10 ) . Subsequent studies systematically examine judge robustness and vulnerabilities, highlighting issues such as prompt sensitivity and alignment bias across domains ( 16 ; 14 ) . Self-preference bias further complicates judge reliability when models evaluate outputs similar to their own generations ( 18 ) . Recent work explores whether LLMs can model individual evaluators rather than aggregate labels. 4 investigate personalized judging and show that conditioning on evaluator-specific history can improve alignment. However, these studies primarily focus on general NLP evaluation tasks. The implications of personalization under systematically low inter-annotator agreement remain insufficiently studied. Our work contributes to this discussion by examining aggregate versus personalized judge configurations in a business ideation setting. By explicitly quantifying expert disagreement and comparing judge alignment under heterogeneous standards, we provide evidence that evaluation modeling choices must account for pluralistic expert judgments rather than assuming a single unified ground truth. 3 Data and Evaluation Setup This section describes the data, scoring dimensions, and annotation protocol used in our study. The objective is to define an evaluation setting where multiple experts score the