Context Matters: Evaluating Context Strategies for Automated ADR Generation Using LLMs Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom

Aviral Gupta, Rudra Dhar, Daniel Feitosa, Karthik Vaidhyanathan · 2026

Read Context Matters: Evaluating Context Strategies for Automated ADR Generation Using LLMs Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom by Aviral Gupta, Rudra Dhar, Daniel Feitosa, Karthik Vaidhyanathan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03826.txt.

Introduction

Aviral Gupta and colleagues asked how much past design history a language model needs to draft Architecture Decision Records well. They gathered validated sequences from 750 open-source repositories and tested five context strategies across model families. A short recency window of 3 to 5 prior records gave the best balance of quality and efficiency. Retrieval helped mainly for tangled, cross-cutting choices, not for ordinary linear workflows.

Catalog record

CREATORAviral Gupta, Rudra Dhar, Daniel Feitosa, Karthik Vaidhyanathan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53975
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03826

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context Matters: Evaluating Context Strategies for Automated ADR Generation Using LLMs Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom Aviral Gupta, Rudra Dhar, Daniel Feitosa, Karthik Vaidhyanathan Abstract . Architecture Decision Records (ADRs) play a critical role in preserving the rationale behind system design, yet their creation and maintenance are often neglected due to the associated authoring overhead. This paper investigates whether Large Language Models (LLMs) can mitigate this burden and, more importantly, how different strategies for presenting historical ADRs as context influence generation quality. We curate and validate a large corpus of sequential ADRs drawn from 750 open-source repositories and systematically evaluate five context selection strategies (no context, All-history, First-K, Last-K, and RAFG) across multiple model families. Our results show that context-aware prompting substantially improves ADR generation fidelity, with a small recency window (typically 3–5 prior records) providing the best balance between quality and efficiency. Retrieval-based context selection yields marginal gains primarily in non-sequential or cross-cutting decision scenarios, while offering no statistically significant advantage in typical linear ADR workflows. Overall, our findings demonstrate that context engineering, rather than model scale alone, is the dominant factor in effective ADR automation, and we outline practical defaults for tool builders along with targeted retrieval fallbacks for complex architectural settings. 1. Introduction Software architecture is not merely a static structure but a continuous stream of design decisions made to satisfy evolving requirements and constraints ( 20 ) . To manage this complexity, Architecture Decision Records (ADRs) have emerged as a de facto standard for lightweight Architectural Knowledge Management (AKM) ( 24 ; 31 ; 4 ) . By capturing the context, decision, and consequences of architectural changes, ADRs facilitate team onboarding, prevent knowledge vaporization, and maintain traceability in long-lived systems. Despite their recognized value, the practice of writing ADRs often suffers from a misalignment of incentives: the cost of documentation is paid immediately by the architect, while the benefits are reaped primarily by future maintainers. Consequently, ADR adoption in practice is inconsistent. A recent large-scale study of open-source repositories revealed that while many projects initialize ADR collections, a significant portion abandons the practice shortly thereafter, leaving decision logs incomplete or outdated ( 3 ) . This “documentation debt” hinders architectural understanding and increases the risk of architectural drift. The rapid ascendancy of Large Language Models (LLMs) presents a compelling opportunity to mitigate this effort. LLMs have demonstrated significant impact across numerous Software Engineering (SE) tasks, including code summarization, requirements engineering, and text generation ( 14 ; 15 ) , making them theoretically well-suited for automating the drafting of ADRs. Preliminary studies have explored this potential, showing that models like GPT-4 can generate plausible decision summaries when provided with specific prompts ( 6 ) . However, existing research has largely treated ADR generation as an isolated task—generating a single decision based on immediate code changes or a narrow context window ( 7 ) . In the landscape of modern LLM applications, Retrieval-Augmented Generation (RAG) has emerged as a critical technique to enhance model capabilities by providing external knowledge. However, practitioners face significant challenges regarding fundamental engineering trade-offs: deciding how much information to retrieve and how to integrate that knowledge effectively without introducing noise ( 29 ) . These trade-offs are particularly acute in software architecture, where providing too little context leads to generic decisions, while too much context may exceed token limits or distract the model with irrelevant historical data. This isolated approach overlooks a fundamental property of software architecture: path dependency . An architectural decision is rarely made in a vacuum; it is constrained by the history of prior decisions, the established technology stack, and previously accepted trade-offs. We posit that the quality of an ADR generation depends critically not just on the capability of the underlying model, but on the context engineering strategy used to retrieve this historical narrative ( 23 ) . In this paper, we present a comprehensive empirical study evaluating how different context-provision strategies influence the quality of LLM-generated ADRs. Unlike previous works that focus primarily on comparing models, we rigorously control for the architectural context provided to those models. To support this investigation, we utilized a large-scale corpus of ADRs originally identified in an MSR study by Buchgeher et al. ( 3 ) . Building on this foundation, we performed extensive manual filtering and sequence verification across 750 repositories to ensure chronological integrity, thereby transforming the raw data into a validated sequential dataset that simulates real-world architectural evolution. Specific contributions of this paper are: • to provide a curated corpus of ADRs that preserves chronological ordering, enabling valid “next-step” generation experiments. • to study the influence of decision history in prompt for generating an ADR through LLM • to quantify the performance impact of four distinct context strategies: All-History , First-K (foundational context), Last-K (recent context), and RAFG (semantic relevance). The remainder of this paper is organized as follows. Section 2 reviews prior work and situates our study in the relevant literature. Next, Section 3 describes our experimental methodology, followed by Section 4 presenting our empirical findings. Section 5 interprets these results, outlines implications, and addresses validity threats. We then review additional related work in Section 6 , and conclude with a summary of contributions and directions for future research in Section 7 . Finally, we provide information on data availability in Section 8 . 2. Background This work sits at the intersection of Architectural Knowledge Management, Generative AI, and Empirical Software Engineering. In this section, we outline the foundation of ADR in practice, and recent studies that also touch upon the mentioned intersection, motivating our investigation into context-aware ADR generation. 2.1. Architecture Decision Records in Practice Software architecture can be fundamentally defined as a set of architectural design decisions ( 18 ) . To capture and manage these decisions, ADRs were popularized by Nygard as a lightweight, version-controlled method to capture the “why” behind software architecture. Despite their theoretical benefits for knowledge management, widespread industrial adoption remains challenging. Buchgeher et al. ( 3 ) conducted a large-scale Mining Software Repositories (MSR) study analyzing over 900 repositories on GitHub. Their findings reveal a critical “adoption gap”: while the number of projects initializing ADRs is increasing, approximately 50% of these repositories contain fewer than five records, suggesting a high rate of abandonment after an initial pilot phase. Furthermore, they observed that sustained ADR maintenance is typically a collaborative effort rather than the responsibility of a single architect. This empirical evidence demonstrates the need for automated tooling to reduce the friction of creating and maintaining documentation, which motivates our investigation into LLM-driven generation. ADR 01: Use Jest as the Testing Framework Status: Accepted Context: We want a testing framework with strong support for TypeScript and Node.js. Jest provides fast execution, rich mocking capabilities, and a mature ecosystem. Decision: We will use Jest as the primary testing framework. Consequences: • All tests will be written using Jest. • Jest’s built-in mocking and TypeScript support can be leveraged. Figure 1. Example of an Architecture Decision Record (ADR) 2.2. Context Engineering and Retrieval-Augmented Generation A recurring finding in the literature is that generative quality is bounded by the relevance of the provided

Read the complete text