Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics.

Alain Vázquez, Maria Inés Torres · 2026

Read Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics. by Alain Vázquez, Maria Inés Torres (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29518.txt.

Introduction

Vázquez and Torres study whether richer inputs improve language generation for dialogue systems. Generation engines turn meaning representations into sentences. These inputs encode the dialogue act and slot-value content. Their twist adds a task demonstrator: one example pair from the dataset. The enriched input is used in training and at run time. They test five metrics across four datasets with varied domains. Enriched inputs helped most on complex tasks and small, varied datasets. Semantic metrics judged quality better than lexical ones.

Catalog record

CREATORAlain Vázquez, Maria Inés Torres
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76265
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29518

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Impact of enriched meaning representations for language generation in dialogue tasks: A comprehensive exploration of the relevance of tasks, corpora and metrics. Alain Vázquez, Maria Inés Torres Abstract Conversational systems should generate diverse language forms to interact fluently and accurately with a variety of users. In this context, Natural Language Generation (NLG) engines convert Meaning Representations (MRs) into sentences, directly influencing user perception. These MRs usually encode the communicative function (e.g., inform, request, confirm) via DAs and enumerate the semantic content with slot–value pairs. In this work, our objective is to analyse whether providing a task demonstrator to the generator enhances the generations of a fine-tuned model. This demonstrator is an MR–sentence pair extracted from the original dataset that enriches the input at training and inference time. The analysis involves five metrics (BLEU, BLEURT, LaBSE, Slot Accuracy and Dialogue Act Accuracy) that focus on different linguistic aspects, and four datasets (E2E, ViGGO, MultiWOZ, and EMPATHIC) that differ in multiple features, such as domain, size, lexicon, MR variability, and acquisition process. To the best of our knowledge, this is the first study on dialogue NLG, implementing a comparative analysis of the impact of MRs on the generation quality across domains, corpora characteristics and the metrics used to evaluate these generations. Our key insight is that the proposed enriched inputs are effective for complex tasks and small datasets with high variability in MRs and sentences. They are also beneficial in zero-shot settings for any domain. Moreover, the analysis of the metrics shows that the semantic metrics capture the generation quality more accurately than lexical metrics. In addition, among these semantic metrics, those trained with human ratings can detect omissions and other subtle semantic issues that embedding-based metrics often miss. Finally, the evolution of the metric scores and the excellent results for Slot Accuracy and Dialogue Act Accuracy demonstrate that the generative models present fast adaptability to different tasks and robustness at semantic and communicative intention levels. Keywords Natural Language Generation \cdot Dialogue Systems \cdot Dialogue Acts \cdot Prompt-based Learning \cdot Evaluation of NLG 1 Introduction The impressive development of Transformers, along with the rapid increase in computing capabilities and the accessibility of vast amounts of data, has revolutionised natural language processing (NLP) in recent years. Large language models (LLMs) have excelled in tasks such as transcription, translation or summarisation, among others. However, LLMs present well-known challenges, including the opacity of training data and the lack of clarity in their behaviour. In conversational systems, a major concern is their tendency to hallucinate, generate misinformation and lack reasoning. Furthermore, while LLMs could recognise lexical and structural patterns, they struggle with semantic inferences and extralinguistic reasoning essential for real-world knowledge tasks ( 30 ) . Conversational systems should be able to generate diverse language forms to interact fluently and accurately with a variety of users. In this context, natural language generation (NLG) plays a crucial role in dialogue, directly influencing user perception. In open-domain dialogues, the NLG must be able to express opinions and emotions, convey facts, provide information, ask contextual questions ( who, what, where, when, why ), confirm user inputs or give instructions. Conversely, closed-domain dialogues must align with the specific task, objectives, and conversational context. In both cases, generated sentences should be semantically controlled, accurate and free of hallucinations. Although end-to-end architectures have advanced in the design of dialogue systems ( 26 ; 35 ) , they still have difficulty achieving dialogue coherence and interpretability. In this context, the concepts of dialogue acts (DA), which represent a sentence’s intentionality or purpose, and meaning representation (MR), which adds specific attributes and entities in the shape of slot-value pairs, remain essential for managing conversational flow. Specifically, NLG generates MR-based sentences, i.e., it takes the meaning representation as input and produces sentences as output. When combined with LLMs, both closed- and open-domain dialogue systems ( 69 ) benefit from improved adaptability to complex tasks ( 19 ) , as well as enhanced coherence and response accuracy ( 22 ; 63 ; 25 ) . In this work, we aim to explore different ways of representing meaning for sentence generation. Our objective is to analyse whether providing the generator model with additional information, in the form of a task demonstrator, enhances generation quality. This task demonstrator is a sample from the original dataset consisting of an MR–sentence pair and enriches the input both at training and inference time. Specifically, we investigate whether these one-shot representations improve generation compared to inputs containing only the MR. The comparison requires evaluation metrics. In NLG, classic metrics, some originally from translation or language model evaluation, do not always capture relevant differences in generated sentences. Nevertheless, each metric assesses a different generation aspect, making their comparison essential for an in-depth study like ours. In this work, we use reference-based metrics such as BLEU ( 44 ) , BLEURT ( 56 ) , and LABSE ( 18 ) , as well as reference-free metrics like Slot Accuracy ( 32 ) , complemented by the more recent Dialogue Act Accuracy (DAC) ( 49 ) . Our goal is to analyse each metric’s ability to identify differences in sentences generated from the proposed MR formats. Four corpora have been selected for this study: E2E ( 42 ) , VIGGO ( 28 ) , MULTIWOZ ( 10 ) , and EMPATHIC ( 62 ) . All four are dialogue-oriented NLG corpora in which each sample consists of a DA-based MR and its corresponding example sentence. These datasets differ significantly in domain, size, lexicon, input variability, and acquisition process, among other features. We will analyse the suitability of each MR and assess the discriminative power of each metric based on the characteristics of each corpus. In summary, this work aims to study the impact of meaning representations on language generation in dialogue tasks. To this end, we thoroughly analyse the relevance of the tasks, the characteristics of the corpora, and the aspects each metric evaluates in the experimental results. For this comparative study, we selected GPT-2 ( 47 ) , which is computationally more accessible and manageable than recent state-of-the-art LLMs such as GPT-4 ( 2 ) , Gemini ( 58 ) , and LLaMA 3 ( 17 ) . In addition, recent work has shown that enriched inputs with task demonstrators can substantially enhance the generation capabilities of medium-sized models ( 34 ; 62 ) . The main contributions can be outlined as follows: • Analysis of the impact of prompt forms on the quality of generated sentences, as well as their zero-shot capabilities. • Comparative evaluation of each metric’s ability to assess different quality aspects of generated sentences. • Assessment of the suitability of classifying generated sentences into dialogue acts as an evaluation metric for generation quality. • Analysis of the influence of corpus design and characteristics, such as the number of references and attributes, on the quality of sentence generation. • Cross-analysis of the distribution of the resulting quality scores in terms of representations, corpora and metrics. • Human qualitative analysis of the generated sentences. To the best of our knowledge, this is the first in-depth study on NLG in the context of dialogue, implementing a comparative analysis of the impact of meaning

Read the complete text