Comparing Developer and LLM Biases in Code Evaluation
Aditya Mittal ∗ Ryan Shar ∗ Zichu Wu Shyam Agarwal Tongshuang Wu Chris Donahue Ameet Talwalkar Wayne Chi † Valerie Chen † Carnegie Mellon University {adityamittal307, ryan.shar01}@gmail.com · 2026
Read Comparing Developer and LLM Biases in Code Evaluation by Aditya Mittal ∗ Ryan Shar ∗ Zichu Wu Shyam Agarwal Tongshuang Wu Chris Donahue Ameet Talwalkar Wayne Chi † Valerie Chen † Carnegie Mellon University {adityamittal307, ryan.shar01}@gmail.com (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24586.txt.
Description
Mittal and co-authors present TRACE, a framework for comparing developer and language model biases in code evaluation. As models increasingly judge code, they should be tested in realistic settings with partial context and ambiguous intent. TRACE measures how well model judges predict human preferences and extracts rubric items that reveal systematic bias. Across chat coding, IDE autocompletion, and instructed editing, the best of 13 models trail human annotators by 12 to 23 percent. TRACE finds 35 significant sources of misalignment, most tied to known code quality criteria.
Catalog record
| CREATOR | Aditya Mittal ∗ Ryan Shar ∗ Zichu Wu Shyam Agarwal Tongshuang Wu Chris Donahue Ameet Talwalkar Wayne Chi † Valerie Chen † Carnegie Mellon University {adityamittal307, ryan.shar01}@gmail.com |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28708 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24586