Coding Triangle: How Does Large Language Model Understand Code?

Taolin Zhang, Zihan Ma, Maosong Cao, Junnan Liu, Songyang Zhang, Kai Chen · 2025

Read Coding Triangle: How Does Large Language Model Understand Code? by Taolin Zhang, Zihan Ma, Maosong Cao, Junnan Liu, Songyang Zhang, Kai Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06138.txt.

Introduction

Zhang and five co-authors probed how language models understand code in this paper from twenty twenty-five. Their frame, the Code Triangle, tests three skills at once. A model must analyze a problem, write a solution, and create test cases. Contests in programming supplied the test material.

Across the three skills, models formed a self-consistent picture, yet their answers lacked human variety and toughness. Errors clustered in set patterns, which the authors blame on training data bias. Feeding in human editorials, solutions, and varied tests improved results a lot. Mixing several models also helped. The authors point to self-reflection as a route to stronger coding models.

Catalog record

CREATORTaolin Zhang, Zihan Ma, Maosong Cao, Junnan Liu, Songyang Zhang, Kai Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06138

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Coding Triangle: How Does Large Language Model Understand Code? Taolin Zhang, Zihan Ma, Maosong Cao, Junnan Liu, Songyang Zhang, Kai Chen Abstract Large language models (LLMs) have achieved remarkable progress in code generation, yet their true programming competence remains underexplored. We introduce the Code Triangle framework, which systematically evaluates LLMs across three fundamental dimensions: editorial analysis, code implementation, and test case generation. Through extensive experiments on competitive programming benchmarks, we reveal that while LLMs can form a self-consistent system across these dimensions, their solutions often lack the diversity and robustness of human programmers. We identify a significant distribution shift between model cognition and human expertise, with model errors tending to cluster due to training data biases and limited reasoning transfer. Our study demonstrates that incorporating human-generated editorials, solutions, and diverse test cases, as well as leveraging model mixtures, can substantially enhance both the performance and robustness of LLMs. Furthermore, we reveal both the consistency and inconsistency in the cognition of LLMs that may facilitate self-reflection and self-improvement, providing a potential direction for developing more powerful coding models. 1 1 1 This work is done when Taolin Zhang and Zihan Ma are on internship at Shanghai AI Laboratory, * means equal contribution, † means corresponding author, ‡ means project lead. 1 Introduction Recent advances in large language models (LLMs) with rapid development in model design and data scaling [ 6 , 1 , 25 , 29 , 10 , 16 , 5 , 31 , 27 , 2 , 23 ] have driven remarkable progress on code generation benchmarks [ 9 , 4 , 7 , 15 , 20 , 18 , 13 , 30 ] . For example, DeepSeek-V3 [ 19 ] achieves a score of 65.2 on HumanEval [ 12 , 9 ] , Qwen3 attains 65.7 on LiveCodeBench [ 27 , 15 ] , and o3 reaches a Codeforces Elo rating of 2724 [ 25 , 11 ] , demonstrating impressive coding abilities of modern foundation models. With rapid advancements in coding capabilities of large language models, a growing concern is that current coding benchmarks fail to accurately and comprehensively evaluate the coding ability of LLMs. In this work, we address a foundational question: How should the coding capability of LLM be defined? When human developers solve coding problems, they typically follow a structured pipeline involving problem analysis, designing a preliminary solution strategy, code implementation, manual testing, and iterative refinement until the solution passes all test cases. Motivated by this, we investigate the coding capabilities of LLMs through three interconnected dimensions: Editorial , Code , and Cases . We develop such a three-dimensional analysis framework called Coding Triangle to study the coding behaviors of LLMs. Our goal is to understand how LLMs fundamentally interpret coding problems within each dimension and to explore the interactions across them. For example: (a) Does performance across dimensions exhibit consistency?(b) Does a model’s code generation benefit from its natural language problem breakdowns?(c) Does the generated code reliably pass self-generated test cases?(d) To what extent do these test cases adequately reflect the reasoning outlined in the editorial? Based on our Coding Triangle framework, we conduct extensive experiments on 200 problems collected from AtCoder and evaluate various LLMs including general models, coding model and reasoning model. By analyzing their capabilities and interactions among different dimensions, we gain deeper insights into how LLMs truly comprehend and tackle coding tasks with several surprising and interesting findings. Our results demonstrate prevalent self-consistency across three dimensions in various LLMs. This self-consistency often confines LLM reasoning, leading it to converge on narrow patterns and repeat similar errors, particularly with corner cases or implementation details. Consequently, a significant distribution shift emerges between LLM predictions and human submissions. Notably, ensembling multiple models can effectively mitigate these cognitive biases, enhancing performance diversity and robustness. Conversely, we observe self-inconsistency across these dimensions, indicating that their respective capabilities are not always fully aligned. An LLM might, for example, accurately analyze its own failed solutions to pinpoint root causes and effectively use self-generated test cases to differentiate correct from incorrect solutions; yet, these individual strengths may not be cohesively integrated, exemplifying the aforementioned misalignment. These findings suggest potential for self-reflection and self-improvement by iteratively aligning these dimensions. To summarize, this paper makes the following contributions. Figure 1 : The framework of Coding Triangle. Editorial, code, and cases form the three fundamental vertices of the triangle, with each vertex can be sampled from either human solutions or model predictions. These vertices are interconnected, influencing one another, and their relationships form the six directed edges of the triangle, representing the mutual interactions between Editorial, Code, and Cases. \bullet We propose a framework called Coding Triangle to systematically examine the internal knowledge of LLMs in programming, enabling a comprehensive evaluation of their coding abilities. \bullet We investigate the distribution shift between the LLM predictions and actual solutions from human, and find that incorporating human information can substantially improve performance. \bullet By analyzing the self-consistency and self-inconsistency inside LLMs, we observe that their strengths and weaknesses vary across the three dimensions in Coding Triangle, demonstrating the advantages of model mixtures and the potential for self-reflection and self-improvement. 2 Overview In this section, we first introduce the three fundamental vertices of Coding Triangle and define how to evaluate the ability of LLM across these three dimensions. We will proceed with the analysis and explore the distribution shift compared to the distribution of human coding in Section 3 , and discuss the interaction among these three dimensions in Section 4 . Our experiments cover general models, coder models and reasoning models 2 2 2 We denote the DeepSeek-V3 [ 19 ] , Qwen2.5-72B-Instruct [ 32 ] , Qwen2.5-Coder-32B-Instruct [ 14 ] and QwQ-32B [ 28 ] , as DS-V3, 72B, Coder and QWQ for short, respectively. All the results are obtained with Nvidia-A800. and we utilize AtCoder 3 3 3 Problems A - F from AtCoder Beginner Contest (ABC) 175 - 374. as the evaluation problem sets. 2.1 Coding Triangle In this study, we systematically examine the comprehensive understanding of LLMs when addressing competitive programming challenges. Motivated by how human solve the problem steps by steps through analyzing, coding and manually testing, we formally decompose coding ability into three interconnected perspectives, and establish the overall framework of Coding Triangle : \bullet Editorial captures how LLM interprets and analyzes the the problem in natural language, providing the most accessible explanation for human readers. \bullet Code reflects the programming logic and ability of algorithm implementation, serving as the machine-executable counterpart to the human-readable editorial. \bullet Cases indicate the depth of understanding in terms of validation criteria, including edge scenarios and boundary conditions. Intuitively, these three dimensions create a comprehensive system that captures all aspects of a coding problem, from interpretation to execution and validation. We then introduce evaluation metrics for each dimension, allowing us to quantify its strengths and weaknesses in a structured way. 2.2 Evaluation Metric Editorial. We adopt an LLM-as-a-judge approach to evaluate the quality of model-generated editorials. With the model-generated editorial E_{\mathrm{model}} and the ground-truth editorial E_{\mathrm{gt}} , we require o3-mini [ 26 ] to judge them and predict a correctness score. Let N be the total number of problems, the

Read the complete text