Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks
Amir Molzam Sharifloo 1 , Maedeh Heydari 1 , Parsa Kazerooni 1 , Daniel Maninger 1 2 , Mira Mezini 1 2 3 · 2025
Read Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks by Amir Molzam Sharifloo 1 , Maedeh Heydari 1 , Parsa Kazerooni 1 , Daniel Maninger 1 2 , Mira Mezini 1 2 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04355.txt.
Description
Amir Molzam Sharifloo, Maedeh Heydari, Parsa Kazerooni, Daniel Maninger, and Mira Mezini ask where language models still fail at code generation, beyond what leaderboards show. Rankings say little about which tasks models consistently miss, yet those failures are exactly what developers need to see to improve the next generation.
Tasks from four popular benchmarks were screened for ones major models most likely fail. Static complexity of solution code was checked as an explanation, then one hundred fourteen stubborn tasks were inspected systematically. Four recurring weakness patterns emerged, alongside common complications inside benchmark tasks that trigger failure. The analysis separates model shortcomings from benchmark artefacts, a distinction raw scores blur.
Catalog record
| CREATOR | Amir Molzam Sharifloo 1 , Maedeh Heydari 1 , Parsa Kazerooni 1 , Daniel Maninger 1 2 , Mira Mezini 1 2 3 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04355