Agile Story-Point Estimation: Is RAG a Better Way to Go? Conference: 34th IEEE/ACM International Conference on Program Comprehension; April 12–13, 2026; Rio de Janeiro, Brazil 34th IEEE/ACM International Conference on Program Comprehension (ICPC ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3794763.3798175 ISBN: 979-8-4007-2482-4/2026/04 CCS: Software and its engineering Requirements analysis
Lamyea Maha, Tajmilur Rahman, Chanchal Roy · 2026
Read Agile Story-Point Estimation: Is RAG a Better Way to Go? Conference: 34th IEEE/ACM International Conference on Program Comprehension; April 12–13, 2026; Rio de Janeiro, Brazil 34th IEEE/ACM International Conference on Program Comprehension (ICPC ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3794763.3798175 ISBN: 979-8-4007-2482-4/2026/04 CCS: Software and its engineering Requirements analysis by Lamyea Maha, Tajmilur Rahman, Chanchal Roy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03443.txt.
Introduction
Maha, Rahman, and Roy ask whether RAG tools can take over Agile story-point estimation, replacing manual Planning Poker. They test two embedding models on 23 open-source projects, varying retrieval settings, project sizes, and baselines. The RAG approach sometimes beats baselines, but gaps across projects and models are not statistically significant, so more tuning is needed.
Catalog record
| CREATOR | Lamyea Maha, Tajmilur Rahman, Chanchal Roy |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52154 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03443
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agile Story-Point Estimation: Is RAG a Better Way to Go? Conference: 34th IEEE/ACM International Conference on Program Comprehension; April 12–13, 2026; Rio de Janeiro, Brazil 34th IEEE/ACM International Conference on Program Comprehension (ICPC ’26), April 12–13, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3794763.3798175 ISBN: 979-8-4007-2482-4/2026/04 CCS: Software and its engineering Requirements analysis Lamyea Maha, Tajmilur Rahman, Chanchal Roy Abstract . The sprint-based iterative approach in the Agile software development method allows continuous feedback and adaptation. One of the crucial Agile software development activities is the sprint-planning session where developers estimate the effort required to complete tasks through a consensus-based estimation technique such as Planning Poker. In the Agile software development method, a common unit of measuring development effort is Story Point (SP) which is assigned to tasks to understand the complexity and development time needed to complete them. Despite the benefits of this process, it is an extremely time-consuming manual process. To mitigate this issue, in this study, we investigated if this manual process can be automated using Retrieval Augmented Generation (RAG) which comprises a “Retriever” and a “Generator”. We applied two embedding models - bge-large-en-v1.5 , and Sentence-Transformers’ all-mpnet-base-v2 on 23 open-source software projects of varying sizes and examined four key aspects: 1) how retrieval hyper-parameters influence the performance, 2) whether estimation accuracy differs across different sizes of the projects, 3) whether embedding model choice affects accuracy, and 4) how the RAG-based approach compares to the existing baselines. Although the RAG-based approach outperformed the baseline models in several occasions, our results did not exhibit statistically significant differences in performance across the projects or across the embedding models. This highlights the need for further studies and refinement of the RAG, and model adaptation strategies for better accuracy in automatically estimating user stories. † † cc-license: by 1. Introduction Many organizations have been using agile methodology for their software development processes for a long time ( 6 ) . The agile development setting allows developers to work by sprints (iterations of two-weeks cycle) during which team members complete a specific set of tasks ( 10 ) from product backlog items (tasks). This iterative approach allows developers to receive customer feedback frequently and test features early ( 3 ) . Based on the feedback from the customer, they create new tasks for improvements that go to the backlog and then plan for the next sprint. This discussion session is typically known as the sprint planning session that takes place right before each sprint begins. Due to the progressive nature of agile development processes, it is important that the development team knows how much effort is required to complete the tasks in the upcoming sprints which will help to utilize available resources accurately ( 23 ) . Planning and estimation in agile processes is different compared to traditional (non-agile) development processes ( 5 ) . One of the most studied and widely used effort estimation techniques during the sprint planning session in the agile development process is “Planning Poker” ( 23 ) which allows developers to assess product backlogs and agree on the required effort to complete each task ( 11 ) . The metric used for effort estimation is called Story Point (SP) . In this study, our focus is to automatically estimate SPs to reduce the time and effort developers need during the sprint planning session. Story points are measured in numbers that indicate the overall effort in terms of hours and the complexity of the task ( 22 ) . This helps the team plan for the next sprint ( 18 ) . Planning Poker session involves people from different levels of experience, such as junior developers, senior developers, team leaders, software testers, scrum masters, and project managers. This diversity gives the team better estimations of story points as it reduces over-optimism that might be exerted by the more experienced developers. ( 12 ) However, this process may introduce biases during effort estimation ( 9 ) . For example, factors such as the influence of senior team members, or the presence of dominant personalities during the meeting can affect the story point estimations by the junior members which can lead to issues like cost overruns, idle developer time, inefficient use of resources, or project failure ( 14 ) . Moreover, Planning Poker can be time-consuming too. According to Cohn ( 5 ) , generally, it takes between 5 and 10 minutes to estimate each task in the product backlog. This means, for a project like Linux Kernel, Planning Poker session would take about 6 months ( 2 ) . To address the challenges of the manual sprint planning approach, there is a growing interest in using machine learning for estimating story points ( 16 ) . Automated systems can make use of historical data to offer consistent, unbiased, and efficient estimates of tasks more efficiently ( 14 ) . Previous studies have primarily focused on either using different features of a task to estimate the story points or learning from the dataset of pre-estimated tasks s to produce an estimation ( 14 ; 4 ) . In another study ( 8 ) although the authors have tried to make the story point estimation process interpretable, they have only worked on the titles of the tasks. Moreover, a replication study ( 21 ) has shown that there is a bug in that study which affected the reported results. In this study, we explored how well a Retrieval-Augmented Generation (RAG) based story point estimation approach would work. To the best of our knowledge, there are no previous studies which explored RAG based story point estimation. In this approach, there are two components, the retriever and the generator. First, given a new task, the retriever uses embedding models to generate embeddings and hence retrieves similar tasks from the embeddings of the training dataset of a particular project. We then used Llama-3.2-3B-Instruct as the generator component to estimate story points based on the retrieved tasks. So, in our study, we have tried to explicitly retrieve and represent concrete examples that include both titles and descriptions of the tasks—just like developers reviewing similar tasks that they have already estimated before voting during a planning poker session. Our approach would also ensure interpretability and transparency in effort estimation using story points since the developers can see the retrieved similar tasks based on which the generator has based its answers on mimicking the widely used planning poker session. However, before such an automated approach can be considered a reliable alternative to established human-based estimation practices such as Planning Poker, it is necessary to rigorously evaluate its effectiveness. Planning Poker incorporates the collective knowledge, negotiation, and experience of developers, whereas a RAG-based method removes the human element entirely and relies solely on retrieved context and model inference. This raises the question of whether such a system can produce estimates of comparable quality in real-world settings. To investigate this, we formulated the following research questions and a set of hypotheses to conduct a systematic empirical study across multiple open-source projects and project sizes. Our research questions are as follows: RQ1 - How do the retrieval parameters (top_k and temperature) influence performance, and how do the optimal parameter settings differ