Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks

Amir Molzam Sharifloo 1 , Maedeh Heydari 1 , Parsa Kazerooni 1 , Daniel Maninger 1 2 , Mira Mezini 1 2 3 · 2025

Read Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks by Amir Molzam Sharifloo 1 , Maedeh Heydari 1 , Parsa Kazerooni 1 , Daniel Maninger 1 2 , Mira Mezini 1 2 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04355.txt.

Description

Amir Molzam Sharifloo, Maedeh Heydari, Parsa Kazerooni, Daniel Maninger, and Mira Mezini ask where language models still fail at code generation, beyond what leaderboards show. Rankings say little about which tasks models consistently miss, yet those failures are exactly what developers need to see to improve the next generation.

Tasks from four popular benchmarks were screened for ones major models most likely fail. Static complexity of solution code was checked as an explanation, then one hundred fourteen stubborn tasks were inspected systematically. Four recurring weakness patterns emerged, alongside common complications inside benchmark tasks that trigger failure. The analysis separates model shortcomings from benchmark artefacts, a distinction raw scores blur.

Catalog record

CREATORAmir Molzam Sharifloo 1 , Maedeh Heydari 1 , Parsa Kazerooni 1 , Daniel Maninger 1 2 , Mira Mezini 1 2 3
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-04355

Browse the full library · About the Tower of Babel