Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming

Sama Hadhoud Alaa Elsetohy Frederikus Hudi, [0.2em] Jan Christian Blaise Cruz Steven Halim Alham Fikri Aji, [0.6em] MBZUAI · 2026

Read Idea First, Code Later: Disentangling Problem Solving from Code Generation in Evaluating LLMs for Competitive Programming by Sama Hadhoud Alaa Elsetohy Frederikus Hudi, [0.2em] Jan Christian Blaise Cruz Steven Halim Alham Fikri Aji, [0.6em] MBZUAI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11332.txt.

Introduction

Sama Hadhoud and six coauthors split problem solving from code writing. Their 2026 preprint tests language models on competitive programming. Models now solve many such problems end to end. But a single score hides two unlike skills in the score.

One skill is the plan: the algorithm that cracks the task. The other is the code: turning the plan into a program. The team centers plain-language editorials in the test. Editorials state the plan in words, with no code at all. Solving and grading both pass through that plan layer. That split shows if a model fails on thought or on syntax. Contest-style tasks are at heart problem-solving tasks, they argue.

Catalog record

CREATORSama Hadhoud Alaa Elsetohy Frederikus Hudi, [0.2em] Jan Christian Blaise Cruz Steven Halim Alham Fikri Aji, [0.6em] MBZUAI
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE122 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11332

Browse the full library · About the Tower of Babel