CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews
Aakash Sorathiya, Gouri Ginde · 2025
Read CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews by Aakash Sorathiya, Gouri Ginde (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-09049.txt.
Introduction
App reviews hide useful complaints about privacy and security inside a flood of everyday feedback. Aakash Sorathiya and Gouri Ginde present CMER, a method that mines those ethics-related reviews without hand-labeled training data. It pairs a language inference model, fed with domain-tuned test statements, with a large language model that does the final sorting.
The team tests the pipeline on more than 382,000 reviews of mobile investment apps. They compare several inference models and three language models, then combine the best of each. The combined system found 2,178 privacy and security reviews that a keyword search had missed. Those reviews can be turned into concrete requirements for app teams. The authors show the approach scales to a problem where ethical language is rare and specialized.
Catalog record
| CREATOR | Aakash Sorathiya, Gouri Ginde |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-09049
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews Aakash Sorathiya, Gouri Ginde Abstract With the increasing proliferation of mobile applications in our daily lives, the concerns surrounding ethics have surged significantly. Users communicate their feedback in app reviews, frequently emphasizing ethical concerns, such as privacy and security. Incorporating these reviews has proved to be useful for many areas of software engineering (e.g., requirement engineering, testing, etc.). However, app reviews related to ethical concerns generally use domain-specific language and are typically overshadowed by more generic categories of user feedback, such as app reliability and usability. Thus, making automated extraction a challenging and time-consuming effort. This study proposes CMER (A C ontext-Aware Approach for M ining E thical Concern-related App R eviews), a novel approach that combines Natural Language Inference (NLI) and a decoder-only (LLaMA-like) Large Language Model (LLM) to extract ethical concern-related app reviews at scale. In CMER, NLI provides domain-specific context awareness by using domain-specific hypotheses, and the Llama-like LLM eliminates the need for labeled data in the classification task. We evaluated the validity of CMER by mining privacy and security-related reviews (PSRs) from the dataset of more than 382K app reviews of mobile investment apps. First, we evaluated four NLI models and compared the results of domain-specific hypotheses with generic hypotheses. Next, we evaluated three LLMs for the classification task. Finally, we combined the best NLI and LLM models (CMER) and extracted 2,178 additional PSRs overlooked by the previous study using a keyword-based approach, thus demonstrating the effectiveness of CMER. These reviews can be further refined into actionable requirement artifacts. I Introduction Mobile applications are designed with particular user goals in mind [ 1 ] . For instance, the goal of investing apps is to enhance the financial capability of users [ 2 ] . However, the competitive landscape of app development often leads to rapid production cycles that compromise the original user-focused goals [ 1 ] , raising serious ethical issues such as privacy and security breaches [ 3 ] . App marketplaces like the Apple App Store and Google Play allow users to rate and review apps, sharing their experiences regarding the app’s quality, performance, and concerns. Developers rely on these reviews to gather feedback and accordingly improve their apps [ 4 ] . Numerous studies have explored user perspectives on ethical concerns in mobile apps. Research by Besmer et al. [ 5 ] and Nema et al. [ 6 ] highlights user concerns about privacy and data security in mobile apps. The issues related to fairness and accessibility in mobile apps are further emphasized by the findings of Rezaei Nasab et al. [ 7 ] and Reyes Arias et al. [ 8 ] . However, these studies primarily rely on keyword- or regular expression-based sampling techniques, which may lack contextual information and restrict ethical concerns to specific keywords. For instance, consider the following review: “Listen!! ok.. so why is there no phone number for customer service? why? if you are a great company and trustworthy why can’t you get a phone line. only email?? c’mon sounds like you can take advantage if need be since nobody cant get threw only on email. if someone is missing money and you don’t follow threw that can be scary esp if people invested $1,000 in this. also, the glitch with updating id and social security stating it’s a photocopy when it is a real deal. no good.” Here, the “social security” phrase is mentioned in regards to a glitch in document uploads functionality; however, it was considered as a privacy-indicative keyword in the previous study [ 1 ] . Consequently, the identification of ethical concerns-related reviews is significantly dependent on contextual interpretation; thus, merely conducting searches for related keywords within the review might not be an effective approach. Harkous et al. [ 9 ] proposed the Natural Language Inference (NLI) method to address this. However, their approach relies on generic privacy hypotheses, overlooking the fact that users’ ethical concerns are domain-dependent [ 1 ] . For example, users of ridesharing apps may worry about constant location tracking, while those on financial apps may be concerned about sharing sensitive financial information. Moreover, all of these studies are based on the supervised classification of app reviews, which requires manual efforts and can be time-consuming. To address these challenges, in this paper, we propose a novel hybrid approach: CMER (A C ontext-Aware Approach for M ining E thical Concern-related App R eviews), that combines NLI and a decoder-only Large Language Model (Llama-like LLM 1 1 1 In this paper, we refer to decoder-only LLMs, such as Llama, ChatGPT, and Claude, as Llama-like LLMs ) to mine ethical concern-related app reviews at scale. CMER utilizes NLI with domain-specific hypotheses (providing domain-specific context awareness) to extract potential concern-related app reviews. It then employs a Llama-like LLM with zero-shot learning (without any fine-tuning) to ultimately extract the actual concern-related app reviews from a set of potentially related reviews. We evaluate the effectiveness of CMER by mining privacy- and security-related reviews (PSRs) from mobile investment apps, guiding our research through three research questions (RQs). RQ1. To what extent can domain-specific NLI accurately identify potential PSRs in investment apps? NLI assesses the relationship between a premise and a hypothesis to identify whether the premise entails, contradicts, or is neutral to the hypothesis [ 10 ] . NLI offers a more refined and linguistically consistent evaluation of text relationships, surpassing keyword-based methods by delivering context-aware results [ 10 ] . Therefore, we investigate to what extent we can use NLI with domain-specific hypotheses to identify the potential PSRs. This helps to filter out the large set of unrelated app reviews and focus on only potentially relevant app reviews. Additionally, we compare our domain-specific hypotheses with generic hypotheses provided by the previous study [ 9 ] and show that domain-specific hypotheses yield better results due to the domain dependence of users’ ethical concerns. RQ2. To what extent can we leverage LLaMA-like LLMs to classify PSRs? NLI helps to extract potential PSRs from a large corpus of irrelevant app reviews however, further steps are required to extract actual PSRs from the set of potential reviews. Previous studies have employed supervised classification techniques, which required fine-tuning classification models on the labeled dataset. However, this technique requires manual efforts to create the labeled data, which can be labor-intensive and time-consuming. Therefore, we investigate to what extent we can leverage Llama-like LLMs with zero-shot learning, which do not require any fine-tuning on the labeled data, to classify PSRs. RQ3. How effective is CMER compared to traditional keyword- and regex-based approaches in mining PSRs? After evaluating NLI and LLM individually, we select the best-performing models at both stages and compare our context-aware approach (CMER) with traditional keyword- and regex-based approaches. Our goal is to evaluate CMER for extracting PSRs that do not contain any predefined terms used in the keyword-based method by the previous study [ 1 ] . The main contributions of our study are: • To our knowledge, CMER is the first hybrid approach utilizing domain-specific NLI and a Llama-like LLM to extract ethical concern-related app reviews. CMER demonstrated better results compared to traditional keyword-based and generic privacy hypotheses methods.