LLM-as-a-Judge for Human-AI Co-Creation: A Reliability-Aware Evaluation Framework for Coding
Md Faizul Ibne Amin, Yutaka Watanobe, Daniel M. Muepu, Haruto Suzuki, Kenta Nanaumi, Md Mostafizer Rahman · 2026
Read LLM-as-a-Judge for Human-AI Co-Creation: A Reliability-Aware Evaluation Framework for Coding by Md Faizul Ibne Amin, Yutaka Watanobe, Daniel M. Muepu, Haruto Suzuki, Kenta Nanaumi, Md Mostafizer Rahman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27727.txt.
Introduction
Ibne Amin and five colleagues present a rubric-driven judge framework for contest-style human-AI teamwork. Four judges face 517 attempts across 15 participants, 13 problems, and 184 trajectories, with grouped validation and defect-injection probes. Giving the problem statement for an identifier rebuilds rankings and raises ROC-AUC markedly. Schema compliance does not imply scale validity: judges emit as few as 8 values, leaving 6 of 99 thresholds effective. Accuracy and defect sensitivity prove separable, and success rises from about 60 percent at first try to about 84 percent by the tenth.
Catalog record
| CREATOR | Md Faizul Ibne Amin, Yutaka Watanobe, Daniel M. Muepu, Haruto Suzuki, Kenta Nanaumi, Md Mostafizer Rahman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77979 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27727
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-as-a-Judge for Human-AI Co-Creation: A Reliability-Aware Evaluation Framework for Coding Md Faizul Ibne Amin, Yutaka Watanobe, Daniel M. Muepu, Haruto Suzuki, Kenta Nanaumi, Md Mostafizer Rahman Abstract Large language models (LLMs) are increasingly employed both as judges for evaluating open-ended outputs and as co-creation partners in AI-assisted programming; yet rigorous evaluation in human-AI co-creation settings remains underdeveloped as judgments must be reliable, comparable across models, and interpretable over multi-turn interaction. To address this gap, a rubric-driven LLM-as-a-Judge framework is presented for contest-style human-AI co-creation in coding and software engineering (SE). The framework is built around schema-constrained judge outputs, validation and repair mechanisms, grouped splitting by ( user , problem ) (\text{user},\text{problem}) to prevent trajectory leakage, and participant-level nonblind context for attempt-level evaluation. Multiple LLM judges are assessed through a multi-metric protocol covering discrimination (ROC-AUC, PR-AUC), thresholded decision quality (MCC with validation-selected thresholds applied to held-out test data), probabilistic reliability (LogLoss, Brier score, ECE), and inter-judge agreement (Cohen’s and Fleiss’ κ \kappa ). Human-AI co-creation is further examined through trajectory-level signals, including turn-wise confidence, Success@Turn, time-to-success via Kaplan-Meier survival analysis, revision churn, and code-aware CodeBLEU similarity. Co-creation success is found to concentrate early, with Success@Turn rising to 0.8533 at the first observed turn and stabilizing at 0.8641 by turn 6. Revision behavior, however, remains heterogeneous, suggesting that productive progress can emerge through either incremental refinement or broader restructuring. On the judging side, the best held-out scores reach 0.5937 for ROC-AUC, 0.6904 for PR-AUC, and 0.5000 for MCC test \mathrm{MCC}_{\mathrm{test}} , while inter-judge consistency remains modest overall (mean pairwise Cohen’s κ \kappa = 0.1592, Fleiss’ κ \kappa = 0.0696). Taken together, this work offers an auditable and reproducible evaluation methodology that links reliability-aware LLM judging with trajectory-based analysis of human-AI co-creation , providing a practical evaluation template for future AI-assisted coding and SE studies. I Introduction Recent advances in AI and natural language processing (NLP) have been driven by Transformer-based LLM foundation models [ 54 , 45 ] and large-scale pretraining with instruction tuning, enabling LLMs to follow natural-language (NL) specifications and produce controllable outputs [ 43 , 6 ] . In SE and programming-centric settings, code-capable LLMs are employed to generate, summarize, explain, and refine solutions, as well as to support iterative debugging from NL prompts [ 13 , 9 ] . This evolution aligns with a broader shift toward human-AI co-creation , wherein artifacts are jointly constructed through multi-turn interaction, explicit constraints, and mixed initiative [ 48 , 42 , 23 , 29 ] . More broadly, AI adoption in creative processes is increasingly framed not as automation, but as collaboration, with AI serving as a generative partner rather than a substitute for human judgment. In such settings, evaluation becomes considerably more challenging: co-creative traces contain partial attempts, revisions, and explanations, while the evaluation target often extends beyond final correctness to encompass constraint satisfaction, clarity, and usefulness [ 24 , 37 , 7 ] . These challenges motivate the development of scalable and reliable evaluation protocols, which in turn give rise to the emerging paradigm of LLM-as-a-Judge , wherein an LLM is prompted to assess open-ended artifacts under explicit criteria, reducing reliance on slow and inconsistent human grading [ 21 , 34 ] . This is especially relevant for coding and SE, where evaluation has traditionally emphasized outcome-only signals such as compile success and pass/fail functional testing; in co-creative workflows, these legacy metrics often miss the practical value of intermediate artifacts such as partially correct patches, debugging hypotheses, and explanations [ 36 , 35 ] . Consequently, coding and SE-oriented co-creation settings require more nuanced, reasoning-aware evaluation frameworks that capture both product quality and process quality in a consistent and auditable manner [ 56 , 26 , 32 ] . In the present study, this problem is examined in the AI-permitted co-creation track conducted in parallel with the official PC Koshien (PCK) Finals, where participants attempt the same released contest problems under the same time constraints while being allowed to use AI tools [ 52 ] . In response to these challenges, co-creative traces are operationalized as attempt-level instances with grouped (\text{participant},\text{problem}) splits to avoid trajectory leakage, and judges are provided with participant-level nonblind context and required to return schema-constrained rubric outputs. In practice, LLM judging protocols are commonly implemented either as pairwise preference judgments for ranking and benchmarking systems or as direct assessment under task-specific rubrics [ 64 , 37 , 44 , 24 ] . Recent work also explores specialized and open judge models to improve evaluation reproducibility [ 59 , 66 ] . At the same time, prior studies show that judge outputs can vary with prompt wording, formatting, and presentation order and may exhibit systematic biases such as favoring longer responses, specific candidate positions, or outputs that better match the judge model’s own distribution [ 10 , 15 , 60 , 50 , 55 , 61 , 63 ] . LLM-based judging can also be manipulated, and bias profiles may differ between human and LLM evaluators [ 8 ] . These concerns are particularly consequential in coding and SE, where tasks may involve long-context reasoning, complex code changes, and realistic engineering benchmarks such as SWE-bench and related judge-oriented evaluation efforts [ 28 , 27 , 51 ] . Therefore, applying LLM judges to co-creative programming artifacts requires protocols that are both auditable and reliability-aware [ 7 , 35 ] . Although LLM-as-a-Judge has become a practical solution for scalable evaluation, existing research has largely emphasized judge bias, benchmark alignment, or general judge design rather than standardizing the end-to-end measurement pipeline needed for human-AI co-creation artifacts in coding and SE [ 50 , 61 , 8 , 55 , 51 , 27 ] . In co-creative programming settings, artifacts are heterogeneous and iterative, with NL interleaved with code, partial revisions, intermediate reasoning, and evolving constraints, which amplifies protocol fragility and makes conclusions sensitive not only to judge reliability but also to execution-level failures such as malformed or underspecified outputs [ 10 , 15 ] . A practical gap therefore remains for a framework that yields auditable decisions instead of free-form critiques, is robust to invalid outputs, and supports reliability-aware reporting for trajectory-level co-creation analysis. To fill this gap, we propose a unified rubric-driven LLM-as-a-Judge framework for coding and SE co-creation in which multiple judges (OpenAI, DeepSeek, Gemini, and Claude) return schema-constrained criterion-level outputs that are validated and repaired when necessary, and are analyzed through complementary metrics for discrimination, thresholded decision quality, probabilistic reliability, and inter-judge agreement. In parallel, the same framework supports trajectory-based analysis of human-AI co-creation through turn-wise confidence, Success@Turn, time-to-success, revision churn, and code-aware similarity. This design enables a single empirical setting in which both co-creation behavior and judge behavior can be evaluated in a transparent and reproducible manner. Beyond co-creation analysis, the framework also has the potential to support more transparent and reliability-aware evaluation of programmers through their iterative coding attempts under AI-assisted conditions. Key points of this research. • Framework: We introduce a rubric-driven LLM-as-a-Judge measurement pipeline for human-AI co-creation in coding and SE, emphasizing auditable outputs, schema validation and repair, and reliability-aware evaluation across multiple judge models. • Co-creation Dynamics: We model human-AI interaction through trajectory-level signals including turn-wise progress, Success@Turn, and