Deep Graph-Language Fusion for Structure-Aware Code Generation Conference: 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering; April 12–13, 2026; Rio de Janeiro, Brazil 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering (FORGE ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3793655.3793724 ISBN: 979-8-4007-2477-0/2026/04 CCS: Software and its engineering Software notations and tools CCS: Computing methodologies Neural networks Mert Tiftikci, Amir Molzam Sharifloo, Mira Mezini Abstract . Pre-trained Language Models (PLMs) have the potential to transform software development tasks. However, despite significant advances, current PLMs struggle to capture the structured and relational attributes of code, such as control flow and data dependencies. This limitation is rooted in an architectural mismatch: whereas code structure is best represented by graphs, transformer-based LLMs process input as sequential token patterns and therefore lack explicit structural awareness. While recent research has explored integrating graph-based code representations using techniques like graph feature extraction, retrieval-augmented generation, and prompt engineering, existing approaches suffer from information loss during dense feature extraction or prompt encoding; notably, the potential of deep, token-level fusion of graph features within model internals has not been systematically explored. In this paper, we initiate such an exploration by introducing CGFuse 1 1 1 The implementation is available at https://github.com/stg-tud/cgfuse . , a novel framework that enables token-level integration of graph-derived representations by infusing learned graph features directly into the intermediate layers of pre-trained language models. CGFuse combines a graph neural network (GNN) with a language model to explicitly preserve and exploit fine-grained structural information from code graphs, including abstract syntax trees and data-flow graphs. We systematically evaluate CGFuse across multiple LLMs, demonstrating up to 10–16% BLEU and 6–11% CodeBLEU improvements in code generation performance. These results highlight the potential of deep graph-PLM integration to advance the field toward more robust, capable AI-driven software development. † † cc-license: by ⬇ 1 /** 2 * Actually walks the bag to make sure the 3 * count is correct and resets the running total 4 * @return the current total size 5 */ (a) Example documentation. Example documentation. ⬇ 1 int calcTotalSize () { 2 _total = extractList (). size (); 3 return _total ; 4 } (b) Example Java code. Example Java code. (c) Code graph with AST (solid lines) and DFG edges (dashed/dotted). Syntax nodes s are orange, terminal nodes t are green. Code graph with AST (solid lines) and DFG edges (dashed/dotted). Syntax nodes $s$ are orange, terminal nodes $t$ are green. Figure 1 . A sample Java snippet with documentation (a), code (b), and its augmented code graph (c). A sample Java snippet with documentation (a), code (b), and its augmented code graph (c). 1. Introduction Large Language Models (LLMs) are rapidly transforming software engineering practice. State-of-the-art systems such as ChatGPT, Gemini, and CodeLLaMA now deliver significant capabilities in code generation, bug detection, and automated testing. Their integration is advancing the field toward fully automated software development ( 17 ; 24 ) . However, recent studies reveal that, despite their strong performance on many tasks, these models struggle to explicitly capture the structured and relational nature of code, such as control flow and data dependencies ( 2 ; 16 ) . We posit that this limitation stems from a fundamental architectural mismatch: while graph-based representations naturally encode structural relationships, transformer architectures primarily operate on sequential token patterns, providing only implicit structural awareness. To bridge this gap, several approaches have been proposed that integrate graph-based code representations through techniques such as graph-based feature extraction, retrieval-augmented generation (RAG), and prompt engineering. Some approaches operate at the repository level, creating complex hybrid code graphs that incorporate project file structures and sophisticated RAG systems ( 14 ; 22 ) . At the token-and-model level, earlier work such as GraphCodeBERT ( 10 ) and UniXcoder ( 9 ) incorporate Abstract Syntax Trees (ASTs) ( 12 ) and data-flow graphs by flattening them as prompts and adding graph-based auxiliary training objectives for full pertaining. More recent approaches ( 26 ; 7 ) employ soft prompting, wherein GNN-generated features are appended to the input prompt and the model is fine-tuned accordingly. While this approach enables direct learning from graphs, it incurs information loss during dense feature extraction and prompt encoding, limiting the depth of graph-LLM fusion. Pass-Tuning ( 5 ) advances this direction by using GNN-based experts within a prefix-tuning setup to achieve deeper fusion at the model level. However, it still relies on dense graph representations, which constrains the model’s capacity to capture rich and fine-grained relational code patterns. Notably, existing approaches have not systematically explored token-level fusion of extracted graph features. This gap raises an important research question: can deeper integration achieved by infusing graph knowledge directly into the internal layers of pre-trained language models (PLMs) better preserve and exploit fine-grained structural information to enhance model capabilities on coding tasks? Towards addressing this question, we developed CGFuse , a novel framework that seamlessly integrates graph-based representations of code into PLMs to enhance their understanding and generation capabilities. Our approach trains a graph neural network to process code graphs and infuses its learned representations directly into the intermediate layers of PLMs, enabling token-level fusion. Specifically, our main contributions are as follows: Novel Framework for Deep Graph-PLM Integration: We propose CGFuse , the first framework that systematically fuses graph neural networks (GNNs) with pre-trained language models (PLMs) at intermediate layers, enabling PLMs to directly exploit fine-grained structural and relational information from code graphs such as ASTs and control/data-flow graphs at the token level. Systematic Evaluation Across Models and Architectures: We conduct a systematic evaluation across nine selected PLMs – including Encoder and Encoder-Decoder architectures – and three GNN architectures, demonstrating that CGFuse improves code generation performance and effectively leverages structural information. 2. Our Approach We aim to enhance code generation quality by combining GNNs’ structural understanding with transformer-based PLMs’ contextual knowledge. Our approach consists of three stages: (1) constructing code graphs from source code, (2) pre-training GNNs as graph experts, and (3) fusing GNN representations into a PLM for downstream tasks. 2.1. Constructing Code Graphs To capture both syntactic and semantic dependencies in code, we represent each snippet as a code graph \mathcal{G}_{c} . This graph is built by augmenting an Abstract Syntax Tree (AST) with data flow edges ( 10 ) . The AST encodes the syntactic structure of the program, while data flow edges model how values propagate through variables. Figure 1 illustrates a sample Java snippet and its corresponding graph. Terminal nodes ( \mathcal{T} , green) represent code tokens, while syntax nodes ( \mathcal{S} , orange) capture higher-level constructs such as expressions and statements. Edges include AST hierarchy edges (solid lines) and data flow edges (dashed/dotted lines) labeled comingFrom and calculatedBy , representing the origin and computation of variables, respectively. Formally, the graph is defined as: \mathcal{G}_{c}=(\mathcal{V},\mathcal{E},\mathcal{R}),\quad\mathcal{V}=\mathcal{S}\cup\mathcal{T},\quad\mathcal{R}=\{p,co,ca\}, where p denotes AST edges, co denotes comingFrom , and ca denotes calculatedBy edges. 2.2. Graph Expert Pre-training We pre-train a GNN to serve as a graph expert, learning node-level representations for both syntax and terminal nodes. The GNN aggregates features from each node’s neighborhood \mathcal{N}(v) : (1) \begin{split}a_{v}^{l}&=\texttt{aggregate}^{l}(\{h_{u}^{l-1}:u\in\mathcal{N}(v)\}),\\ h_{v}^{l}&=\texttt{combine}^{l}(h_{v}^{l-1},a_{v}^{l}),\end{split} where h_{v}^{0} are initial node embeddings, which include PLM token embeddings for terminal nodes and learned embeddings for syntax nodes. For graphs with multiple relation types (AST, comingFrom , calculatedBy ), we optionally use relational aggregation ( 11 ) . The GNN is trained on a node classification task, learning to predict properties of each node from its local context, capturing structural and semantic code patterns. We initialize the GNN with PLM embeddings for terminal nodes and random embeddings for syntax nodes, and pre-train on node classification, masking a portion of nodes for validation/testing. 2.3. Fusing GNN and PLM Features After pre-training, we integrate GNN embeddings into a transformer-based PLM. Figure 2 illustrates this fusion mechanism. At a target layer l , the transformer produces token representations h_{\theta}^{l} , while the GNN outputs node representations h_{g}^{l} . We fuse them as: (2) \displaystyle h_{g}^{l}=\texttt{GNN}^{l}(\mathcal{G}_{c}^{l}),\quad h_{\theta}^{l}=\theta^{l}(h_{f}^{l-1}),\quad h_{f}^{l}=h_{\theta}^{l}+\lambda h_{g}^{l}, where \lambda controls fusion strength. Only nodes in the current context are fed into the GNN; out-of-context edges are masked. As for the implementation, we first attach the pre-trained GNN to target PLM layers and train it alone for several warmup epochs. Then, we jointly fine-tune the fused model on the downstream task, optimizing both PLM and GNN parameters. Fusion of GNN features into a PLM layer. Only context nodes are included and edges outside the context are masked. Figure 2 . Fusion of GNN features into a PLM layer. Only context nodes are included and edges outside the context are masked. Fusion of GNN features into a PLM layer. Only context nodes are included and edges outside the context are masked. 3. Experiments In this section, we first present the experimental setup, including the datasets, model configurations, and evaluation metrics used to assess the effectiveness of our approach. We then describe the experiments and report the results to demonstrate the impact of incorporating graph-based representations on model performance. 3.1. Experimental Setup Baseline Models: We apply our approach to various architectures, trained with natural language, code, and flattened AST inputs. Table 1 provides a comprehensive list of the models used in our experiments. We construct encoder-decoder models for our experiments by adapting encoder-only models. As a result, the parameter counts reported for these models correspond to their encoder-decoder versions, which approximately double the parameters of the original encoder models. GNN experts: We investigated the impact of different GNN architectures and the number of intermediate layers. Specifically, we experimented with the Relational Graph Convolutional Network (R-GCN) ( 11 ) , the GraphSAGE model (GS) ( 11 ) , and the Graph Isomorphism Network (GIN) ( 25 ) . Dataset: For our experiments, we use the CONCODE dataset 2 2 2 Dataset available at https://huggingface.co/datasets/AhmedSSoliman/CodeXGLUE-CONCODE . – an established benchmark for evaluating code generation models. CONCODE comprises 104k Java classes (100k for training and 2k each for validation and testing) collected from GitHub. The code generation task takes a natural-language intent (see Fig. 1 ) and generates code to fulfill it. Metrics: For evaluation, we use three string-matching-based metrics: Exact Match (EM) : Measures the percentage of predictions that exactly match the ground truth. BLEU (B) : A widely used metric for evaluating text generation tasks, introduced by 18 . CodeBLEU (CB) : An extension of BLEU tailored for code, which incorporates syntactic and semantic features ( 21 ) . It aggregates four complementary signals of n - gram match BLEU, weighted n - gram match that up-weights language keywords, syntax match computed over ASTs, and semantic match computed over DFGs, all contribute equally. Implementation details: All experiments were conducted with a single run and the same seed value, using a single-layer GNN expert attached to the decoder’s last layer. All code graphs, except for sub-tokens, are generated before the training. Sub-tokens are generated during model training and testing using the corresponding tokenizer. The GNNs are pretrained for three epochs on these code graphs, which is sufficient to achieve approximately 99% accuracy. Table 1 . Code Models evaluated in this work. We evaluate two types of architectures (Arch): encoder ( E ) and encoder-decoder ( ED ) models. In addition to natural language inputs ( NL ), models can be trained with code ( C ) or flattened AST inputs ( AST f ), where the subscript indicates flattening. For all encoder models converted to encoder-decoder models, the number of parameters is specified for the encoder-decoder versions. Models Param Arch Data BERT ( 6 ) 255M E NL RoBERTa ( 15 ) 285M E NL BART ( 13 ) 143M ED NL T5 ( 19 ) 226M ED NL CodeBERT ( 8 ) 277M E C GraphCodeBERT ( 10 ) 277M E C + DFG f UniXCoder ( 9 ) 280M E C + AST f PLBART ( 1 ) 139M ED C CodeT5 ( 23 ) 222M ED C 3.2. Results Our results in Table 2 reveal several key observations: Strong gains from GNN augmentation for natural language models: Models pre-trained solely on natural language benefit substantially from GNN-based code graph infusion. For example, BERT improves +15.1 BLEU (33.4 \rightarrow 48.5) and +11.9 CodeBLEU (31.2 \rightarrow 43.1) when combined with a 1-layer R-GCN. This demonstrates that structural embeddings effectively infuse code knowledge into models that were not exposed to code during pre-training. Selected NL models served as the base checkpoints from which the corresponding code-pretrained transformer models are initialized. To assess sample efficiency, we apply CGFuse to RoBERTa and BART. The initialization chains are: (3) \begin{split}&\text{RoBERTa}\rightarrow\text{CodeBERT}\rightarrow\text{GraphCodeBERT},\\ &\text{RoBERTa}\rightarrow\text{UniXcoder},\quad\text{BART}\rightarrow\text{PLBART}\end{split} Comparing the underlined entries in Table 2 shows that CGFuse -enhanced NL models outperform their code-pretrained counterparts, despite using substantially fewer training samples and iterations (see Table 3 ) 3 3 3 A notable limitation of T5 ( 20 ) is its limited vocabulary. This makes CodeBLEU evaluation significantly unreliable; therefore, the T5 family is omitted from this comparison. . Table 2 . Code generation results for baseline and fused models. For each fused model, the GNN architecture is specified with a subscript. \lambda represents the fusing strength; see Eq. 2 . Models \lambda BLEU CodeBLEU EM Natural Language BERT 33.4 31.2 8.0 BERT R-GCN 1 48.5 43.1 10.1 RoBERTa 36.8 33.1 11.6 RoBERTa GIN 0.5 52.6 45.0 15.6 BART 36.9 31.3 15.4 BART GIN 0.5 52.1 42.7 20.4 T5 33.5 29.9 15.2 T5 GIN 0.5 40.0 31.7 19.5 Programming Language CodeBERT 36.9 34.4 11.1 CodeBERT GIN 1 54.1 46.2 19.0 GraphCodeBERT 49.3 35.4 13.7 GraphCodeBERT GS 1 56.4 47.3 20.0 UniXCoder 40.5 35.9 14.0 UniXCoder GIN 1 57.5 49.5 21.2 PLBART 42.1 36.6 18.2 PLBART R-GCN 1 59.3 51.5 27.6 CodeT5 42.7 36.8 18.2 CodeT5 GIN 1 60.0 51.5 30.3 Significant improvements for code-pretrained models: PLBART, CodeBERT, GraphCodeBERT, and UniXCoder consistently improve across all metrics when augmented with a GNN. In particular, PLBART shows greater improvement than the other models, with differences of +17.2, +14.9, and +9.4 on BLEU, CodeBLEU, and EM scores, respectively. Overall, these results demonstrate that GNN-based code graph embeddings can improve code generation accuracy. We further investigated how the number of layers in a GNN affects performance on the code-generation task. We experimented with 1, 2, and 3 intermediate GNN layers to assess the effect of depth. Each additional layer increases the node’s receptive field, allowing aggregation from more distant neighbors, but it also adds parameters and increases training cost. Results ( Table 4 ) show that single-layer GNNs consistently outperform deeper variants across all tested architectures. Adding a second or third layer leads to a substantial drop in scores, suggesting that deeper GNNs may introduce noise or over-smooth node representations, thereby reducing their usefulness for code generation. Among the models, R-GCN achieves the highest scores in BLEU and CodeBLEU, whereas GIN achieves the highest score in EM, both with a single layer. Overall, these results indicate that shallow GNN embeddings provide the most effective structural information for code generation. Table 3 . The table shows the difference in training time (normalized #Step), the amount of pretraining code data that models are exposed to (#Sample), and the performance gains of NL-based models when CGFuse is applied, compared to their corresponding code model counterparts (e.g., GraphCodeBERT vs. RoBERTa GIN ; see Eq. 3 ). Models \times #Step +#Sample - \Delta B - \Delta CB - \Delta EM CodeBERT \times 137 \sim 8M 15.7 10.6 4.5 GraphCodeBERT \times 274 \sim 12M 13.3 9.6 1.9 UniXcoder \times 547 \sim 36M 12.1 9.1 1.6 PLBART \times 293 \sim 575M 10.0 6.1 2.2 Table 4 . The average code generation results of CGFuse using different GNN architectures and numbers of layers on 9 PLMs at the last decoder layer ( \lambda=1 ). The numbers in the model names indicate the number of GNN layers (L). Models BLEU CodeBLEU EM 1-L 2-L 3-L 1-L 2-L 3-L 1-L 2-L 3-L R-GCN 53.3 27.9 24.3 46.0 28.4 27.7 17.5 1.2 1.3 GS 52.4 29.4 23.9 44.6 30.5 28.0 18.0 1.1 1.4 GIN 52.0 14.1 12.1 44.3 20.4 20.0 20.0 1.7 2.0 4. Conclusion and Future Directions In this work, we investigated how code PLMs can be enriched with representations generated by GNN experts at the token and layer levels. We evaluated our approach on various code and natural language PLMs in the code generation task, experimenting with nine different models pre-trained on both natural and programming languages, covering different architectures and training strategies. In general, we achieved better performance across all models. Moreover, the comparison between the NL models and their code counterparts shows significant sample efficiency in learning. The encouraging results presented in the paper motivate further pursuing the research in several directions. First, developing strategies for efficient GNN training, handling incomplete code during decoding, and mitigating ambiguity in AST generation are promising avenues for further improvement. Training a new GNN for each architectural or vocabulary change is computationally costly and integrating GNN experts into decoders for code generation tasks is challenging, as code graphs must often be constructed from incomplete or ambiguous code, complicating the generation process. Second, applying the presented approach to larger PLMs could test its scalability and benefits for more complex generation tasks. Finally, evaluating performance on benchmarks that measure functional correctness such as HumanEval ( 4 ) and MBPP ( 3 ) would provide a more rigorous assessment of practical utility. Acknowledgements. We gratefully acknowledge support from the hessian.AI Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant no. 16IS22091) and the hessian.AI Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/003) and also by the National Research Center for Applied Cybersecurity ATHENE within the project Athene SecureCoder, and by the LOEWE initiative (Hesse, Germany) [LOEWE/4a//519/05/00.002(0013)/95]. The work has benefited from the early stages of the funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy— EXC-3057.