Tricky²: Towards a Benchmark for Evaluating Human and LLM Error Interactions Conference: 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering; April 12–13, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM Third International Conference on AI Foundation Models and S

Cole Granger, Dipin Khati, Daniel Rodriguez-Cardenas, Denys Poshyvanyk · 2026

Read Tricky²: Towards a Benchmark for Evaluating Human and LLM Error Interactions Conference: 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering; April 12–13, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM Third International Conference on AI Foundation Models and S by Cole Granger, Dipin Khati, Daniel Rodriguez-Cardenas, Denys Poshyvanyk (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18949.txt.

Description

Cole Granger and three co-authors build a test set where human bugs and model bugs mix. Large language models now help write code, but they add sly logic slips that differ from the slips people make. To study how the two kinds of error play off each other, the team made a hybrid set.

The new set, Tricky squared, grows an older corpus of human written defects. Two large models then inject fresh errors into code in three languages. The mix lets researchers see when model bugs hide near human ones, and when each kind is caught or missed by checks.

Catalog record

CREATORCole Granger, Dipin Khati, Daniel Rodriguez-Cardenas, Denys Poshyvanyk
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-18949

Browse the full library · About the Tower of Babel