Bridging Subjective and Objective QoE: Operator-Level Aggregation Using LLM-Based Comment Analysis and Network MOS Comparison

Parsa Hassani Shariat Panahi ID, Amir Hossein Jalilvand ID, M. Hassan Najafi ID · 2025

Read Bridging Subjective and Objective QoE: Operator-Level Aggregation Using LLM-Based Comment Analysis and Network MOS Comparison by Parsa Hassani Shariat Panahi ID, Amir Hossein Jalilvand ID, M. Hassan Najafi ID (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00924.txt.

Introduction

Three authors led by Parsa Hassani build a two-lens gauge for how good a video stream feels to its viewers. Network teams know line marks such as loss and lag, yet those only hint at felt quality. User comments on live streams hold the felt side, but they pour in as raw text at huge scale. Their frame reads both sides and rolls them up per operator for fair checks.

One lens trains a model on lab mean scores to call felt quality from line marks alone. The next lens sifts live comments with a language model and scores the apt ones on the same scale. A set of near 48,000 comments, with fake operator tags and time stamps, backs the build. A gap score then sets felt and line reads side by side.

Catalog record

CREATORParsa Hassani Shariat Panahi ID, Amir Hossein Jalilvand ID, M. Hassan Najafi ID
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00924

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging Subjective and Objective QoE: Operator-Level Aggregation Using LLM-Based Comment Analysis and Network MOS Comparison Parsa Hassani Shariat Panahi ID, Amir Hossein Jalilvand ID, M. Hassan Najafi ID Abstract This paper introduces a dual-layer framework for network operator-side quality of experience (QoE) assessment that integrates both objective network modeling and subjective user perception extracted from live-streaming platforms. On the objective side, we develop a machine learning model trained on mean opinion scores (MOS) computed via the ITU-T P.1203 reference implementation, allowing accurate prediction of user-perceived video quality using only network parameters such as packet loss, delay, jitter, and throughput–without reliance on video content or client-side instrumentation. On the subjective side, we present a semantic filtering and scoring pipeline that processes user comments from live streams to extract performance-related feedback. A large language model is used to assign scalar MOS scores to filtered comments in a deterministic and reproducible manner. To support scalable and interpretable analysis, we construct a labeled dataset of 47,894 live-stream comments, of which approximately 34,000 are identified as QoE-relevant through multi-layer semantic filtering. Each comment is enriched with simulated Internet Service Provider attribution and temporally aligned using synthetic timestamps in 5-minute intervals. The resulting dataset enables operator-level aggregation and time-series analysis of user-perceived quality. A delta MOS ( Δ ​ MOS \Delta\text{MOS} ) metric is proposed to measure each Internet service provider’s deviation from platform-wide sentiment, allowing detection of localized degradations even in the absence of direct network telemetry. A controlled outage simulation confirms the framework’s effectiveness in identifying service disruptions through comment-based trends alone. The system provides each operator with its own subjective MOS and the global platform average per interval, enabling real-time interpretation of performance deviations and comparison with objective network-based QoE estimates. The proposed architecture is scalable, privacy-preserving, and infrastructure-agnostic, offering a practical solution for enhancing operator awareness through platform-derived user feedback. 1 Introduction The exponential growth of internet-based services and multimedia applications has dramatically increased the demand for reliable, user-oriented quality assessments. As network environments become more dynamic and heterogeneous, ensuring optimal user satisfaction requires going beyond traditional quality of service metrics. This has led to the emergence of quality of experience (QoE) as a more holistic and user-centric indicator that reflects how users perceive service quality under diverse and often unpredictable conditions [ 1 , 2 ] . Traditionally, QoE assessment has relied on either intrusive subjective testing, which is resource-intensive and difficult to scale, or objective estimation methods based solely on static network-level parameters. These approaches often fail to account for the dynamic and context-sensitive nature of human perception, resulting in limited accuracy and generalizability–especially in real-world deployment scenarios. To overcome these challenges, researchers have increasingly turned to hybrid QoE frameworks that fuse objective network indicators with user-centric feedback and machine learning techniques. For example, [ 3 ] proposed a model that integrates perceptual factors such as buffering duration alongside physical-layer metrics like RSRP and SINR, using an artificial neural network to improve prediction accuracy in mobile environments. Expanding on this paradigm, Gao et al. [ 4 ] emphasized the integration of contextual and psychological dimensions, including device type, emotional state, and usage environment, into deep learning and Bayesian models–enabling richer and more individualized QoE inference. Lightweight and scalable implementations of such frameworks have also gained traction. Ref. [ 5 ] , for instance, provides cross-application QoE estimation using decision tree models trained on passively collected network and device-level metrics. By crowdsourcing end-user feedback and minimizing overhead via UDP-based probing, this framework offers a real-time, practical solution for mobile network monitoring and experience forecasting. In a similar spirit, [ 6 ] proposed a refined approach for fixed wireless access networks, where a calibrated customer experience index aligns conventional service KPIs with subjective user feedback–yielding a low-complexity but effective model for adaptive QoE evaluation. Crowdsourcing has also proven critical for bridging the gap between subjective perception and objective signals. The user-centric web QoE evaluating framework [ 7 ] exemplifies this by combining live subjective scores with automated logging of network metrics during real web usage sessions. The resulting data-driven models demonstrated superior accuracy–about 5% higher than strong machine learning baselines–highlighting the value of integrating user subjectivity into predictive algorithms. From a methodological standpoint, Charonyktakis et al. [ 8 ] introduced a modular framework designed to dynamically select the most suitable machine learning algorithm ( e.g., support vector regression, artificial neural networks, gaussian naive bayes, decision trees) for a given dataset or user profile. Leveraging nested cross-validation and Bayesian feature selection, this framework adapts to both data characteristics and user heterogeneity. The framework consistently outperforms conventional methods such as PESQ and the E-model across VoIP and video streaming datasets, emphasizing the importance of adaptive, per-user modeling for robust QoE prediction. To this end, we propose a dual-layer QoE assessment framework that bridges standardized network-side metrics with user-perceived quality extracted from live content platforms. On the network operator side, objective MOS is estimated using an enhanced ITU P.1203-compliant model [ 9 ] applied to key performance indicators. In parallel, subjective MOS is inferred from user-generated feedback through large language model (LLM) analysis. This integrated approach enables comparative QoE evaluation across operators and platforms. 2 Contributions This research’s primary contribution lies in developing a dual-layer QoE assessment framework that facilitates operator-level comparison between subjective and objective quality metrics across mobile and multimedia networks. The framework synthesizes LLM-processed mean opinion scores (MOS) from user-generated content and contrasts them with network parameter-driven MOS predictions generated by our enhanced ITU P.1203-compliant machine learning model. The principal innovations of this work include: • Aggregated Subjective QoE Estimation: We introduce a processing pipeline that systematically filters and evaluates real-world user feedback from live streaming platforms through semantic analysis and LLM integration. The derived quality scores are aggregated according to network operator identification (via IP ranges), establishing a scalable framework for passive subjective QoE assessment. • Objective QoE Estimation Through ITU-Compliant Modeling: Building on our prior research, we implement an optimized Random Forest model inspired by ITU P.1203 standards to calculate objective MOS from network key performance indicators (KPIs) including latency, jitter, packet loss, and bitrate. These metrics are aggregated per operator for comparative analysis at the group level. • Subjective–Objective MOS Discrepancy Analysis: Through systematic alignment of comment-derived subjective MOS with network-based objective MOS at the operator level, our framework identifies critical QoE mismatches – particularly instances where technical measurements overestimate actual user satisfaction or fail to detect service quality issues. • Three-Tier QoE Architecture Implementation: Our solution organizes quality metrics into an Access-Service-Experience hierarchy, building upon established QoE modeling literature [ 10 , 11 , 12 ] while maintaining alignment with industry-standard monitoring architectures like ETSI TS 103 294. This stratified approach enables coherent interpretation of both network performance metrics and user quality perceptions through operator-centric aggregation. • Open Architecture with Operational Scalability: All core components – including content filtration algorithms, MOS calculation modules, and data aggregation systems – are developed with reproducibility and open-source principles. The privacy-conscious design supports real-time deployment in operational QoE monitoring dashboards while ensuring scalability across network infrastructures. 2.1 Paper Structure In § 3 , we review related work on QoE estimation and the use of LLMs in subjective quality assessment. § 4 introduces

Read the complete text