SAFT: Structure-Aware Fine-Tuning of LLMs for AMR-to-Text Generation

Rafiq Kamel, Filippo Guerranti, Simon Geisler, Stephan Günnemann · 2025

Read SAFT: Structure-Aware Fine-Tuning of LLMs for AMR-to-Text Generation by Rafiq Kamel, Filippo Guerranti, Simon Geisler, Stephan Günnemann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13381.txt.

Introduction

Four researchers teach language models to turn meaning graphs into text in a paper from 2025. A meaning graph, called an AMR, encodes a sentence's content as a web of concepts and links. Turning it back into fluent text tests whether a model respects structure. Common methods flatten the graph into a line and lose key cues.

Their method, SAFT, feeds the graph's shape into the model without changing its architecture. Position codes are computed from the graph so the model senses direction and distance. A projection step places those codes where the model can use them. On a standard test set the method set a new best score. Gains grew as graphs got more complex.

Catalog record

CREATORRafiq Kamel, Filippo Guerranti, Simon Geisler, Stephan Günnemann
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13381

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAFT: Structure-Aware Fine-Tuning of LLMs for AMR-to-Text Generation Rafiq Kamel, Filippo Guerranti, Simon Geisler, Stephan Günnemann Abstract Large Language Models (LLMs) are increasingly applied to tasks involving structured inputs such as graphs. Abstract Meaning Representations (AMRs), which encode rich semantics as directed graphs, offer a rigorous testbed for evaluating LLMs on text generation from such structures. Yet, current methods often arbitrarily linearize AMRs, discarding key structural cues, or rely on architectures incompatible with standard LLMs. We introduce SAFT, a structure-aware fine-tuning approach that injects graph topology into pretrained LLMs without architectural changes. We compute direction-sensitive positional encodings from the magnetic Laplacian of transformed AMRs and project them into the embedding space of the LLM. While possibly applicable to any graph-structured inputs, we focus on AMR-to-text generation as a representative and challenging benchmark. SAFT sets a new state-of-the-art on AMR 3.0 with a 3.5 BLEU improvement over baselines. Gains scale with graph complexity, highlighting the value of structure-aware representations in enhancing LLM performance. SAFT offers a general and effective pathway for bridging structured data and language models. 1 Introduction Figure 1 : SAFT : structure-aware fine-tuning of LLMs for AMR-to-text ( {\mathcal{A}}\rightarrow{\mathcal{S}} ), using linearized AMRs ( {\mathcal{L}}_{\mathcal{A}} ) augmented with AMR-specific PEs ( \textsc{AmrPE}_{\mathcal{A}} ). Large Language Models (LLMs) have become the dominant paradigm for natural language processing (NLP), demonstrating strong generalization across a wide range of sequential tasks. Increasingly, researchers are exploring how to extend the capabilities of LLMs to structured data domains such as graphs ( Jin et al. 2024a ; Jiang et al. 2023 ; Fatemi et al. 2024 ; Zhang et al. 2022 ; Tang et al. 2024 ) , driven by growing interest in extending the reasoning and representation capabilities of LLMs beyond sequential data to more expressive, structured modalities. However, existing approaches that adapt LLMs to graphs often require architectural modifications, or auxiliary components. These strategies compromise a core advantage of LLMs: their scalability and flexibility as pretrained, general-purpose sequence models. A particularly well-defined and linguistically grounded graph representation is the Abstract Meaning Representation (AMR) ( Banarescu et al. 2013 ) , a rooted, directed acyclic graph that encodes predicate-argument structure and core semantic relations. We focus on the AMR-to-text generation task: producing a natural language sentence that accurately expresses the meaning of an AMR graph. This task presents a strong benchmark for evaluating the ability of LLMs to interface with structured semantic representations, as it demands sensitivity to graph topology and semantic content while preserving fluency and coherence in the generated output. Despite its importance, AMR-to-text generation remains challenging due to the inherent relational and semantical structure of AMRs. Sequence-to-sequence models ( Bevilacqua et al. 2021 ; Cheng et al. 2022 ) linearize AMRs, discarding structural information crucial for semantic fidelity. Graph-to-sequence methods ( Song et al. 2018 ; Zhu et al. 2019 ; Ribeiro et al. 2021 ) preserve structure through Graph Neural Networks (GNNs), but their reliance on specialized encoders breaks compatibility with pretrained LLMs. More recent work attempts to repurpose LLMs for this task via prompting or fine-tuning on linearized AMRs ( Mager et al. 2020 ; Yao et al. 2024a ; Raut et al. 2025 ) , but these methods still overlook the underlying graph structure critical to meaning preservation. This fragmentation reveals a critical gap: How can graph-structured information be integrated into LLMs in a lightweight, architecture-agnostic way to enable structure-aware generation? We address this question with SAFT , a structure-aware fine-tuning method that augments LLM inputs with positional encodings derived from graph topology. Specifically, we compute direction-sensitive graph positional encodings from the magnetic Laplacian ( Furutani et al. 2020 ; Geisler et al. 2023 ) of an AMR-derived graph and inject them into the embeddings of the graph linearization tokens via a lightweight projection network. This design ensures compatibility with any decoder-only LLM and avoids architectural changes to the model, as illustrated in Fig. 1 . While our approach is grounded in AMR-to-text generation, its design is conceptually applicable to other tasks involving graph-structured inputs, such as drug design ( Zheng et al. 2024 ) , code representation learning ( Allamanis et al. 2017 ) , and scene graph-to-text generation ( Yang et al. 2019 ) . We focus on AMRs as they provide a linguistically motivated, semantically rich benchmark that allows for precise evaluation of structural understanding in language generation. Their formalism enables controlled experimentation with topology and meaning, making them an ideal foundation for this line of work. SAFT provides a concrete step toward aligning structured graph representations with pretrained LLMs, focusing on AMRs as a high-value benchmark for studying structure-aware generation. Our contributions include: • A structure-aware fine-tuning framework for LLMs that incorporates graph positional encodings into token embeddings, enabling relational inductive bias without modifying the model architecture. • A novel formulation of AMR-specific positional encodings derived from the eigenvectors of the magnetic Laplacian, effectively capturing directionality and global semantic structure in AMRs. • Comprehensive experiments showing that SAFT achieves state-of-the-art performance on AMR 3.0, with a +3.5 BLEU improvement over baselines, and increased gains on graphs with higher structural complexity, such as document-level AMRs. 2 Background We introduce the foundational concepts necessary for our method, focusing on graph representations and graph positional encodings, Abstract Meaning Representations (AMRs), and the application of Large Language Models (LLMs) to structured input. 2.1 Graph Representations and Graph Positional Encodings Edge-labeled directed graphs. AMRs are a prime example of edge-labeled directed graphs. We formally define these as tuples {\mathcal{A}}=({\mathcal{V}}_{\mathcal{A}},{\mathcal{E}}_{\mathcal{A}},{\mathcal{R}}_{\mathcal{A}})\in\mathbb{A} , where {\mathcal{V}}_{\mathcal{A}} is the set of n_{\mathcal{A}}=|{\mathcal{V}}_{\mathcal{A}}| nodes, {\mathcal{E}}_{\mathcal{A}}\subseteq{\mathcal{V}}_{\mathcal{A}}\times{\mathcal{V}}_{\mathcal{A}} is the set of m_{\mathcal{A}}=|{\mathcal{E}}_{\mathcal{A}}| directed edges, and {\mathcal{R}}_{\mathcal{A}} is a finite set of relation types (edge labels), such that each edge (u,v)\in{\mathcal{E}}_{\mathcal{A}} is associated with a label r_{u,v}\in{\mathcal{R}}_{\mathcal{A}} . The space of these graphs is denoted by \mathbb{A} . Edge-unlabeled directed graphs. To understand positional encodings on graphs, it is useful to first consider edge-unlabeled directed graphs, defined as tuples {\mathcal{G}}=({\mathcal{V}}_{\mathcal{G}},{\mathcal{E}}_{\mathcal{G}})\in\mathbb{G} , where {\mathcal{V}}_{\mathcal{G}} is the set of n_{\mathcal{G}}=|{\mathcal{V}}_{\mathcal{G}}| nodes, and {\mathcal{E}}_{\mathcal{G}}\subseteq{\mathcal{V}}_{\mathcal{G}}\times{\mathcal{V}}_{\mathcal{G}} is the set of m_{\mathcal{G}}=|{\mathcal{E}}_{\mathcal{G}}| directed, binary, and unlabeled edges, with e_{u,v}=1 if there is a directed edge from node u to node v , and 0 otherwise. The space of such graphs is denoted by \mathbb{G} . The relational structure is encoded in the adjacency matrix {\bm{A}}\in\{0,1\}^{n_{\mathcal{G}}\times n_{\mathcal{G}}} , where {\bm{A}}_{u,v}=e_{u,v} . We define the out-degree matrix {\bm{D}} as a diagonal matrix with \smash{{\bm{D}}_{u,u}=\sum_{v}{\bm{A}}_{u,v}} . Symmetrizing the adjacency matrix as \smash{{\bm{A}}_{S}={\bm{A}}\lor{\bm{A}}^{\top}} (element-wise logical OR) yields an undirected representation of the graph, with a corresponding symmetrized degree matrix {\bm{D}}_{S} . This allows for the computation of the symmetric normalized Laplacian \smash{{\bm{L}}_{S}={\bm{I}}-{\bm{D}}_{S}^{-1/2}{\bm{A}}_{S}{\bm{D}}_{S}^{-1/2}} , which has a real eigendecomposition \smash{{\bm{L}}_{S}={\bm{U}}{\bm{\Lambda}}{\bm{U}}^{\top}} , where {\bm{U}}\in\mathbb{R}^{n_{\mathcal{G}}\times n_{\mathcal{G}}} contains orthonormal eigenvectors and {\bm{\Lambda}}=\text{diag}(\lambda_{1},\dots,\lambda_{n_{\mathcal{G}}})\in\mathbb{R}^{n_{\mathcal{G}}\times n_{\mathcal{G}}} is a diagonal matrix of real eigenvalues. However, this symmetrization inherently loses the directional information present in the original directed graph. Magnetic Laplacian. To address the loss of directionality, we can employ the magnetic Laplacian ( Furutani et al. 2020 ) , which introduces directional information via complex-valued phase shifts. For q\in\mathbb{R}_{\geq 0} , the magnetic Laplacian \smash{{\bm{L}}^{(q)}\in\mathbb{C}^{n_{\mathcal{G}}\times n_{\mathcal{G}}}} is defined as: {\bm{L}}^{(q)}:={\bm{D}}_{S}-{\bm{A}}_{S}\odot\exp\left(i\bm{\Theta}^{(q)}\right), (1) where i=\sqrt{-1} , \odot denotes the Hadamard product (element-wise multiplication), and the phase matrix \bm{\Theta}^{(q)}\in\mathbb{R}^{n_{\mathcal{G}}\times n_{\mathcal{G}}} is given by (\bm{\Theta}^{(q)})_{u,v}=2\pi q(({\bm{A}})_{u,v}-({\bm{A}})_{v,u}) . The magnetic

Read the complete text