Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback 2026 Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback – References
Sandra Loop, Erik Bertram, Sebastian Juhl, Martin Schrepp · 2026
Read Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback 2026 Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback – References by Sandra Loop, Erik Bertram, Sebastian Juhl, Martin Schrepp (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-23018.txt.
Introduction
Sandra Loop, Erik Bertram, Sebastian Juhl, and Martin Schrepp speed up the study of user feedback for software. Product teams pair survey scores with open-ended comments, which explain the scores. Reading thousands of comments by hand is slow, so many insights stay buried.
Their approach joins multi-label classification with generative AI to sort comments at scale. Each comment can carry several labels at once, matching how users mix praise, bugs, and wishes. Faster, faithful sorting helps teams fix what users actually complain about. Open-ended comments are the hard part at scale, because they mix topics freely and resist simple counting methods.
Catalog record
| CREATOR | Sandra Loop, Erik Bertram, Sebastian Juhl, Martin Schrepp |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-23018
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback 2026 Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback – References Sandra Loop, Erik Bertram, Sebastian Juhl, Martin Schrepp Abstract In highly competitive software markets, user experience (UX) evaluation is crucial for ensuring software quality and fostering long-term product success. Such UX evaluations typically combine quantitative metrics from standardized questionnaires with qualitative feedback collected through open-ended questions. While open-ended feedback offers valuable insights for improvement and helps explain quantitative results, analyzing large volumes of user comments is challenging and time-consuming. In this paper, we present techniques developed during a long-term UX measurement project at a major software company to efficiently process and interpret extensive volumes of user comments. To provide a high-level overview of the collected comments, we employ a supervised machine learning approach that assigns meaningful, pre-defined topic labels to each comment. Additionally, we demonstrate how generative AI (GenAI) can be leveraged to create concise and informative summaries of user feedback, facilitating effective communication of findings to the organization and especially upper management. Finally, we investigate whether the sentiment expressed in user comments can serve as an indicator for overall product satisfaction. Our results show that sentiment analysis alone does not reliably reflect user satisfaction. Instead, product satisfaction needs to be assessed explicitly in surveys to measure the user’s perception of the product. 1 Introduction User experience (UX) evaluation plays a central role in ensuring software quality, supporting informed design decisions, and fostering long-term product success. In highly competitive software markets, products that fail to meet users’ expectations in terms of usability, usefulness, and overall experience are unlikely to achieve sustained adoption. Consequently, continuous UX measurement has become a core element of modern quality assurance and product management processes, particularly in large-scale and long-lived software systems ( 3 ; 18 ; 10 ; 9 ) . In practice, UX evaluation typically combines quantitative metrics derived from standardized questionnaires with qualitative feedback collected through open-ended questions. Well-established instruments such as the System Usability Score (SUS) ( 3 ) , UMUX-Lite and UX-Lite ( 12 ; 10 ) , UEQ-S ( 18 ) , or the Net Promoter Score (NPS) ( 17 ) provide compact, reliable, and interpretable numerical indicators of overall usability, user satisfaction, and perceived quality. These metrics are especially valuable for benchmarking, trend monitoring, and cross-product comparisons. However, quantitative UX metrics alone are often insufficient to fully explain why users perceive a product in a certain way. For this reason, open-text user comments are frequently included in UX surveys. Such comments can provide concrete hints about specific strengths and weaknesses of a system, describe contextual factors that influence user perceptions, and highlight issues that are difficult to capture with predefined questionnaire items. In some cases, comments even point directly to actionable problems that can be transferred into concrete development tasks, such as bug fixes or usability improvements. As a result, the combination of quantitative metrics and qualitative comments is widely regarded as a best practice in UX evaluation. Despite their potential value, user comments are notoriously difficult to analyze at scale. In continuous UX measurement programs, organizations may collect thousands of comments over time, making manual analysis costly and inefficient. Moreover, comments are often vaguely formulated, focus disproportionately on negative aspects, and vary greatly in length and specificity. A further challenge is the frequently observed mismatch between quantitative metrics and comment sentiment collected from the same respondent. For example, users may report high satisfaction or usability scores while simultaneously describing isolated negative issues in their textual feedback, resulting in an overall negative sentiment classification of the comment. In our previous work, we addressed these challenges in the context of a long-term, large-scale UX measurement initiative and proposed methods to systematically transfer user feedback into product improvements ( 1 ) . Building on this foundation, the present paper focuses specifically on the analysis and interpretation of user comments and their relationship to quantitative UX metrics. Compared to quantitative measures obtained from standard questionnaires described above, the analysis of comments requires substantially more sophisticated methodological support. In this paper, we describe techniques developed and refined within our long-term UX measurement project ( 1 ) to handle large volumes of user comments. These techniques include automated multi-label classification of comments into meaningful topic categories, the generation of structured summaries using generative AI (GenAI), and sentiment analysis. Furthermore, we present empirical analyses that compare the sentiment of user comments with responses to quantitative UX metrics derived from questionnaires. The goal of these analyses is to support UX professionals in interpreting comment sentiments appropriately and to avoid common pitfalls, such as equating negative comment sentiment with overall user dissatisfaction. Overall, this work aims to demonstrate how qualitative and quantitative UX data can be combined in a complementary way: quantitative metrics provide robust indicators of overall experience quality, while user comments enrich these metrics with contextualized explanations, concrete examples, and actionable insights, provided that suitable analytical methods are applied. 2 Developing a Multi-Label Comment Classification Model Open-text survey comments span from broad product impressions to fine-grained issue reports, making manual synthesis slow and inconsistent. Automated multi-label classification organizes this diversity into topic categories. Classifying comments based on their content also allows product teams to monitor changes in the share of comments related to a specific topic over time. For example, they can quickly verify if investments into performance improvements of a system lead, as expected, to a decrease in the share of comments that complain about slow response times. Additionally, assigning each comment to topic categories also helps to search for user feedback related to a certain topic of interest. A product manager, for example, may be interested in identifying technical issues and errors users are facing in order to identify and address the underlying problems. Being able to filter out comments where users report system errors or bugs substantially reduces the time to identify and eventually resolve these issues. In order to support product teams to efficiently use open-ended feedback from users, we develop and train a multi-label comment classification model using supervised machine learning and a human-in-the-loop approach that automatically assigns one or multiple topic labels to each comment. Our approach consists of multiple steps which we outline here. 2.1 Deriving Topic Labels Since our supervised learning approach requires a set of pre-defined labels to train the model, we start by defining a topic taxonomy. The labels need to meet certain criteria for our use case. First, they need to be collectively exhaustive, implying that the labels are general enough to be broadly applicable across a diverse product portfolio and to be able to apply to user comments in the past as well as to those in the future. Second, they need to be consistently defined to ensure a high inter- and also intracoder reliability which is important for the quality of the manually labeled training dataset and, eventually, the performance of the classification model ( 15 ) . Third, they need to represent topics relevant to the users, meaning that each label should not occur too infrequently (say in >1\% of comments). Finally, the number of unique labels should be manageable such that they provide