Using LLMs and Essence to Support Software Practice Adoption

Sonia Nicoletti, Paolo Ciancarini · 2025

Read Using LLMs and Essence to Support Software Practice Adoption by Sonia Nicoletti, Paolo Ciancarini (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16445.txt.

Introduction

Sonia Nicoletti and Paolo Ciancarini build a chat coach for a shared tongue of software craft, in a study from 2025. The tongue, Essence, frames all craft ways in one set of parts. Teams oft fail to take it up, since the frame reads as pure school lore.

Their bot, Essence Coach, weds a chat model to a fetch bank of Essence lore. Tests with sundry models show fetch lifts the grade of Essence replies each time. First trials hint both tech and non-tech learners grasp and use the frame more with the bot at hand. Wide user tests are owed, the pair grant; the seed is that a coach bot can land a dry frame in live hands.

Catalog record

CREATORSonia Nicoletti, Paolo Ciancarini
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16445

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Using LLMs and Essence to Support Software Practice Adoption Sonia Nicoletti, Paolo Ciancarini Abstract Recent advancements in natural language processing (NLP) have enabled the development of automated tools that support various domains, including software engineering. However, while NLP and artificial intelligence (AI) research has extensively focused on tasks such as code generation, less attention has been given to automating support for the adoption of best practices, the evolution of ways of working, and the monitoring of process health. This study addresses this gap by exploring the integration of Essence, a standard and thinking framework for managing software engineering practices, with large language models (LLMs). To this end, a specialised chatbot was developed to assist students and professionals in understanding and applying Essence. The chatbot employs a retrieval-augmented generation (RAG) system to retrieve relevant contextual information from a curated knowledge base. Four different LLMs were used to create multiple chatbot configurations, each evaluated both as a base model and augmented with the RAG system. The system performance was evaluated through both the relevance of retrieved context and the quality of generated responses. Comparative analysis against the general-purpose LLMs demonstrated that the proposed system consistently outperforms its baseline counterpart in domain-specific tasks. By facilitating access to structured software engineering knowledge, this work contributes to bridging the gap between theoretical frameworks and practical application, potentially improving process management and the adoption of software development practices. While further validation through user studies is required, these findings highlight the potential of LLM-based automation to enhance learning and decision-making in software engineering. I Introduction Large language models (LLMs) have rapidly advanced in recent years, demonstrating significant potential in various domains, including software engineering. These models, such as GPT-4o and Llama 3, have been extensively used for tasks ranging from code generation and bug detection to documentation support. The integration of artificial intelligence (AI) into software engineering has primarily focused on coding-related activities, leveraging LLMs to assist developers in writing, reviewing, and optimising code. However, software engineering is a broader discipline that encompasses not only coding but also other critical aspects such as methodology adoption, team collaboration, and the application of development practices. Despite the growing application of AI in software engineering, process-oriented tasks remain underexplored in terms of LLM integration. While research has investigated AI-driven tools for issue tracking, scheduling, and process automation [ 11 , 25 , 7 , 6 , 4 , 2 ] , the potential of LLMs to assist with software engineering practices, particularly through the Essence standard, remains largely unexamined. Essence provides a structured way to describe software engineering methods, offering a framework for teams to assess and adapt their processes. However, there is a lack of research on how LLMs can effectively retrieve and present Essence-related knowledge to support software development tasks. This paper addresses this gap by developing and evaluating a chatbot designed to assist software practitioners in managing and improving their processes using Essence. The chatbot leverages a RAG system to enhance its responses by retrieving relevant context from a curated knowledge base. By combining the generative capabilities of LLMs with targeted information retrieval, the system aims to provide more accurate and contextually relevant answers than general-purpose models [ 9 ] . In this study, we investigate three key research questions: RQ1: How effective is the chatbot’s information retrieval system in providing relevant information on Essence and software engineering practices? RQ2: To what extent does retrieval augmentation improve the quality of LLM responses for Essence-related queries compared to general-purpose LLMs? RQ3: Which types of Essence-related questions benefit most from retrieval augmentation across different model configurations? To answer these questions, we conducted a series of experiments evaluating the chatbot’s ability to retrieve relevant information. We measured its performance using both automated metrics, such as BERTScore, and human evaluation criteria, including relevance, accuracy, and completeness. Our findings indicate that the RAG-based chatbot consistently outperforms general-purpose LLMs in domain-specific queries, highlighting the benefits of integrating structured retrieval methods with LLMs. This paper makes the following contributions: • We developed a specialised chatbot to support software development activities, paired with a curated dataset of Essence-related documents. This represents a novel application of large language models in software engineering, aimed at assisting with the selection, application, and adaptation of software engineering practices, as well as facilitating reflection on team workflows and process health. • We conducted a comparative analysis of four state-of-the-art LLMs across three question types and two configurations (with and without RAG). This analysis provides empirical evidence for the effectiveness of a domain-specific retrieval-augmented approach, based on both automated semantic evaluation and human-assessed quality (relevance, accuracy, completeness). The rest of the paper is structured as follows. Section 2 discusses related work on LLMs in software engineering and retrieval-augmented systems. Section 3 describes the methodology used to develop and evaluate the chatbot. Section 4 presents experimental results and analysis. Section 5 outlines implications and threats to validity, and Section 6 concludes with future research directions. II Background and Related Works II-A The Essence Standard Essence is a software engineering standard developed by the SEMAT initiative and maintained by the Object Management Group (OMG) [ 12 ] . It provides a common framework for defining, adapting, and combining software engineering practices. Its goal is to improve communication across teams, encourage flexibility in methods, and support continuous improvement. At the core of Essence are two components: the Essence Kernel and the Essence Language. The language allows practitioners to describe practices in a modular, visual, and reusable way. It introduces structured elements such as Alphas (key progress areas in a software endeavour), Work Products, Competencies, Activities, and Patterns. The Essence Kernel contains the universal elements relevant to any software project. It is structured into three Areas of Concern: Customer, Solution, and Endeavour. Within these areas lie seven Kernel Alphas (Opportunity, Stakeholders, Requirements, Software System, Work, Team, and Way of Working), each tracking critical dimensions of project health and progress. The process of “essentialising” a practice means expressing it using the Essence Language to distil its essential components, such as activities and artefacts. This allows for better adoption, adaptation, and integration of practices like Scrum or Kanban within diverse team contexts. Essence games use physical or digital cards to represent Essence elements and improve collaboration. Examples include Progress Poker (team alignment on Alpha states), Chase the State (retrospective analysis across Alphas), and Objective Go (goal setting). These games help teams visualise progress and facilitate structured discussions during meetings. Essence has seen practical application in both industry and academia. In industry, it supports the integration of various methodologies (e.g., Agile, DevOps), offering flexibility and enhancing process health monitoring. In academia, Essence is used as a pedagogical tool to teach core software engineering principles. Its modular design allows students to engage with practices critically and adapt them to specific problems. Essence cards help make abstract concepts more tangible. II-B LLMs for Software Development LLMs have rapidly become a transformative technology in software engineering. While their most widely known application is code generation, their usage has begun to expand significantly across various stages of the software development lifecycle. A comprehensive systematic literature review by Hou et al. [ 11 ] highlights how LLMs are being integrated into tasks such

Read the complete text