Deep Graph-Language Fusion for Structure-Aware Code Generation Conference: 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering; April 12–13, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering (FORGE ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793655.3793724 ISBN: 979-8-4007-2477-0/2026/04 CCS: Software and its engineering Software notations and tools CCS: Computing methodologies Neural networks

Mert Tiftikci, Amir Molzam Sharifloo, Mira Mezini · 2026

Read Deep Graph-Language Fusion for Structure-Aware Code Generation Conference: 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering; April 12–13, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering (FORGE ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793655.3793724 ISBN: 979-8-4007-2477-0/2026/04 CCS: Software and its engineering Software notations and tools CCS: Computing methodologies Neural networks by Mert Tiftikci, Amir Molzam Sharifloo, Mira Mezini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03689.txt.

Introduction

Mert Tiftikci, Amir Molzam Sharifloo, and Mira Mezini introduce CGFuse, a framework that gives pre-trained language models direct access to the structure of program code. Language models read code as token sequences, so they miss control flow and data dependencies that a graph represents naturally. Earlier fixes extracted graph features or added them to prompts, losing information along the way.

CGFuse instead fuses graph-derived representations at token level inside the model's intermediate layers. A graph neural network learns the structural features, and the language model keeps its fine-grained detail during generation. The authors present the work as the first systematic exploration of deep token-level graph-language fusion for structure-aware code generation.

Catalog record

CREATORMert Tiftikci, Amir Molzam Sharifloo, Mira Mezini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03689

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deep Graph-Language Fusion for Structure-Aware Code Generation Conference: 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering; April 12–13, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering (FORGE ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793655.3793724 ISBN: 979-8-4007-2477-0/2026/04 CCS: Software and its engineering Software notations and tools CCS: Computing methodologies Neural networks Mert Tiftikci, Amir Molzam Sharifloo, Mira Mezini Abstract . Pre-trained Language Models (PLMs) have the potential to transform software development tasks. However, despite significant advances, current PLMs struggle to capture the structured and relational attributes of code, such as control flow and data dependencies. This limitation is rooted in an architectural mismatch: whereas code structure is best represented by graphs, transformer-based LLMs process input as sequential token patterns and therefore lack explicit structural awareness. While recent research has explored integrating graph-based code representations using techniques like graph feature extraction, retrieval-augmented generation, and prompt engineering, existing approaches suffer from information loss during dense feature extraction or prompt encoding; notably, the potential of deep, token-level fusion of graph features within model internals has not been systematically explored. In this paper, we initiate such an exploration by introducing CGFuse 1 1 1 The implementation is available at https://github.com/stg-tud/cgfuse . , a novel framework that enables token-level integration of graph-derived representations by infusing learned graph features directly into the intermediate layers of pre-trained language models. CGFuse combines a graph neural network (GNN) with a language model to explicitly preserve and exploit fine-grained structural information from code graphs, including abstract syntax trees and data-flow graphs. We systematically evaluate CGFuse across multiple LLMs, demonstrating up to 10–16% BLEU and 6–11% CodeBLEU improvements in code generation performance. These results highlight the potential of deep graph-PLM integration to advance the field toward more robust, capable AI-driven software development. † † cc-license: by ⬇ 1 /** 2 * Actually walks the bag to make sure the 3 * count is correct and resets the running total 4 * @return the current total size 5 */ (a) Example documentation. Example documentation. ⬇ 1 int calcTotalSize () { 2 _total = extractList (). size (); 3 return _total ; 4 } (b) Example Java code. Example Java code. (c) Code graph with AST (solid lines) and DFG edges (dashed/dotted). Syntax nodes s are orange, terminal nodes t are green. Code graph with AST (solid lines) and DFG edges (dashed/dotted). Syntax nodes $s$ are orange, terminal nodes $t$ are green. Figure 1 . A sample Java snippet with documentation (a), code (b), and its augmented code graph (c). A sample Java snippet with documentation (a), code (b), and its augmented code graph (c). 1. Introduction Large Language Models (LLMs) are rapidly transforming software engineering practice. State-of-the-art systems such as ChatGPT, Gemini, and CodeLLaMA now deliver significant capabilities in code generation, bug detection, and automated testing. Their integration is advancing the field toward fully automated software development ( 17 ; 24 ) . However, recent studies reveal that, despite their strong performance on many tasks, these models struggle to explicitly capture the structured and relational nature of code, such as control flow and data dependencies ( 2 ; 16 ) . We posit that this limitation stems from a fundamental architectural mismatch: while graph-based representations naturally encode structural relationships, transformer architectures primarily operate on sequential token patterns, providing only implicit structural awareness. To bridge this gap, several approaches have been proposed that integrate graph-based code representations through techniques such as graph-based feature extraction, retrieval-augmented generation (RAG), and prompt engineering. Some approaches operate at the repository level, creating complex hybrid code graphs that incorporate project file structures and sophisticated RAG systems ( 14 ; 22 ) . At the token-and-model level, earlier work such as GraphCodeBERT ( 10 ) and UniXcoder ( 9 ) incorporate Abstract Syntax Trees (ASTs) ( 12 ) and data-flow graphs by flattening them as prompts and adding graph-based auxiliary training objectives for full pertaining. More recent approaches ( 26 ; 7 ) employ soft prompting, wherein GNN-generated features are appended to the input prompt and the model is fine-tuned accordingly. While this approach enables direct learning from graphs, it incurs information loss during dense feature extraction and prompt encoding, limiting the depth of graph-LLM fusion. Pass-Tuning ( 5 ) advances this direction by using GNN-based experts within a prefix-tuning setup to achieve deeper fusion at the model level. However, it still relies on dense graph representations, which constrains the model’s capacity to capture rich and fine-grained relational code patterns. Notably, existing approaches have not systematically explored token-level fusion of extracted graph features. This gap raises an important research question: can deeper integration achieved by infusing graph knowledge directly into the internal layers of pre-trained language models (PLMs) better preserve and exploit fine-grained structural information to enhance model capabilities on coding tasks? Towards addressing this question, we developed CGFuse , a novel framework that seamlessly integrates graph-based representations of code into PLMs to enhance their understanding and generation capabilities. Our approach trains a graph neural network to process code graphs and infuses its learned representations directly into the intermediate layers of PLMs, enabling token-level fusion. Specifically, our main contributions are as follows: Novel Framework for Deep Graph-PLM Integration: We propose CGFuse , the first framework that systematically fuses graph neural networks (GNNs) with pre-trained language models (PLMs) at intermediate layers, enabling PLMs to directly exploit fine-grained structural and relational information from code graphs such as ASTs and control/data-flow graphs at the token level. Systematic Evaluation Across Models and Architectures: We conduct a systematic evaluation across nine selected PLMs – including Encoder and Encoder-Decoder architectures – and three GNN architectures, demonstrating that CGFuse improves code generation performance and effectively leverages structural information. 2. Our Approach We aim to enhance code generation quality by combining GNNs’ structural understanding with transformer-based PLMs’ contextual knowledge. Our approach consists of three stages: (1) constructing code graphs from source code, (2) pre-training GNNs as graph experts, and (3) fusing GNN representations into a PLM for downstream tasks. 2.1. Constructing Code Graphs To capture both syntactic and semantic dependencies in code, we represent each snippet as a code graph \mathcal{G}_{c} . This graph is built by augmenting an Abstract Syntax Tree (AST) with data flow edges ( 10 ) . The AST encodes the syntactic structure of the program, while data flow edges model how values propagate through variables. Figure 1 illustrates a sample Java snippet and its corresponding graph. Terminal nodes ( \mathcal{T} , green) represent code tokens, while syntax nodes ( \mathcal{S} , orange) capture higher-level constructs such as expressions and statements. Edges include AST hierarchy edges (solid lines) and data flow edges (dashed/dotted lines) labeled comingFrom and calculatedBy , representing the origin and computation of variables, respectively. Formally, the graph is defined as: \mathcal{G}_{c}=(\mathcal{V},\mathcal{E},\mathcal{R}),\quad\mathcal{V}=\mathcal{S}\cup\mathcal{T},\quad\mathcal{R}=\{p,co,ca\}, where p denotes AST edges, co denotes comingFrom , and ca denotes calculatedBy edges. 2.2. Graph Expert Pre-training We pre-train a GNN to serve as a graph expert, learning node-level representations for both syntax and terminal nodes. The GNN aggregates features from each node’s neighborhood \mathcal{N}(v) : (1) \begin{split}a_{v}^{l}&=\texttt{aggregate}^{l}(\{h_{u}^{l-1}:u\in\mathcal{N}(v)\}),\\ h_{v}^{l}&=\texttt{combine}^{l}(h_{v}^{l-1},a_{v}^{l}),\end{split} where h_{v}^{0} are initial node embeddings, which include PLM token embeddings for terminal nodes

Read the complete text