Comparing Developer and LLM Biases in Code Evaluation

Aditya Mittal ∗ Ryan Shar ∗ Zichu Wu Shyam Agarwal Tongshuang Wu Chris Donahue Ameet Talwalkar Wayne Chi † Valerie Chen † Carnegie Mellon University {adityamittal307, ryan.shar01}@gmail.com · 2026

Read Comparing Developer and LLM Biases in Code Evaluation by Aditya Mittal ∗ Ryan Shar ∗ Zichu Wu Shyam Agarwal Tongshuang Wu Chris Donahue Ameet Talwalkar Wayne Chi † Valerie Chen † Carnegie Mellon University {adityamittal307, ryan.shar01}@gmail.com (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24586.txt.

Description

Mittal and co-authors present TRACE, a framework for comparing developer and language model biases in code evaluation. As models increasingly judge code, they should be tested in realistic settings with partial context and ambiguous intent. TRACE measures how well model judges predict human preferences and extracts rubric items that reveal systematic bias. Across chat coding, IDE autocompletion, and instructed editing, the best of 13 models trail human annotators by 12 to 23 percent. TRACE finds 35 significant sources of misalignment, most tied to known code quality criteria.

Catalog record

CREATORAditya Mittal ∗ Ryan Shar ∗ Zichu Wu Shyam Agarwal Tongshuang Wu Chris Donahue Ameet Talwalkar Wayne Chi † Valerie Chen † Carnegie Mellon University {adityamittal307, ryan.shar01}@gmail.com
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28708
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.24586

Browse the full library · About the Tower of Babel