Visualizing Public Opinion on X: A Real-Time Sentiment Dashboard Using VADER and DistilBERT

Y Abhiram Reddy, Siddhi Agarwal, Vikram Parashar, Arshiya Arora · 2025

Read Visualizing Public Opinion on X: A Real-Time Sentiment Dashboard Using VADER and DistilBERT by Y Abhiram Reddy, Siddhi Agarwal, Vikram Parashar, Arshiya Arora (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15448.txt.

Introduction

Yanampally Abhiram Reddy and co-authors build a live dashboard in a 2025 paper that tracks public opinion about companies on social media. Investors and policymakers watch that mood closely, but raw posts are noisy and fast-moving. Their system cleans the text first, then scores its sentiment two ways.

A rule-based scorer that handles slang and emphasis pairs with a transformer model that reads context. Merging the two aims for both clarity and accuracy. Charts then show the mood's spread over time and across firms. Real-time mood tracking could flag reputation shifts early, while the firms can still respond.

Catalog record

CREATORY Abhiram Reddy, Siddhi Agarwal, Vikram Parashar, Arshiya Arora
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15448

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visualizing Public Opinion on X: A Real-Time Sentiment Dashboard Using VADER and DistilBERT Y Abhiram Reddy, Siddhi Agarwal, Vikram Parashar, Arshiya Arora Abstract In the era of social media dominance, understanding public sentiment towards entities in real-time offers crucial insights for businesses, researchers, and policymakers [ 25 , 27 ] . This paper presents a comprehensive sentiment analysis framework leveraging both rule-based and transformer-based Natural Language Processing (NLP) techniques to analyze public opinion expressed on X (formerly Twitter) [ 42 , 28 ] . We introduce a hybrid approach combining VADER (Valence Aware Dictionary and sEntiment Reasoner) [ 46 ] and DistilBERT [ 37 ] to balance computational efficiency with contextual understanding. Our methodology encompasses real-time data collection via web scraping [ 16 ] , robust preprocessing [ 31 , 48 ] , and a dual-model sentiment classification pipeline. Experimental results demonstrate that our hybrid system achieves 87.6% accuracy and an F1-score of 0.84, outperforming single-model approaches [ 4 , 24 ] . When applied to corporate reputation monitoring, our system reveals significant sentiment disparities across major corporations, with companies like Amazon and Samsung receiving notably positive sentiment scores, while others like Microsoft and Walmart exhibit predominantly negative public perception [ 15 , 20 ] . These findings validate the effectiveness of our multi-model approach in providing actionable insights for strategic decision-making based on public sentiment analysis. I Introduction Social media platforms have evolved into powerful channels of public expression, offering unprecedented volumes of real-time data reflecting opinions, reactions, and sentiments [ 30 , 27 , 29 ] . Among these platforms, X (formerly Twitter) stands out for its concise format and real-time nature, making it an invaluable source for sentiment analysis [ 39 , 26 ] . The ability to accurately interpret sentiment from these streams enables organizations to gauge public perception, react to emerging trends, and make data-driven decisions with minimal latency [ 49 ] . Traditional sentiment analysis approaches often struggled with the unique challenges presented by social media content-abbreviations, emojis, hashtags, and rapidly evolving slang [ 4 ] . However, recent advancements in Natural Language Processing (NLP) have yielded more sophisticated models capable of addressing these complexities. Rule-based systems like VADER (Valence Aware Dictionary and sEntiment Reasoner) [ 46 ] offer computational efficiency and interpretability, while transformer-based models like DistilBERT [ 37 ] provide deeper contextual understanding at the cost of increased computational demands [ 4 ] . The practical applications of sentiment analysis span numerous domains: • Corporate reputation management – Monitoring public perception of brands and products [ 26 ] • Financial markets – Gauging investor sentiment as a leading indicator for market movements [ 26 ] • Political campaigns – Assessing public reactions to policies and statements [ 39 ] • Crisis management – Tracking sentiment during emergencies or controversial events [ 39 ] Despite these potential applications, existing systems often fail to balance real-time performance with contextual accuracy, typically favoring one attribute at the expense of the other [ 4 ] . Furthermore, many research efforts focus on either rule-based or transformer-based approaches in isolation, without exploring hybrid architectures that could leverage the strengths of both [ 4 ] . This paper presents a novel framework for real-time sentiment analysis that integrates web scraping capabilities [ 16 ] with a dual-model classification approach. By combining VADER’s efficiency with DistilBERT’s contextual understanding, our system achieves both speed and accuracy in sentiment classification. We demonstrate the practical utility of this approach through a case study on corporate reputation monitoring, analyzing public sentiment toward major global corporations. The main contributions of our work include: 1. A real-time web scraping module specifically designed for X, capable of collecting targeted data streams based on keywords, hashtags, or user accounts [ 16 ] 2. A hybrid sentiment classification system combining rule-based and transformer-based approaches to balance speed and contextual accuracy [ 4 ] 3. A comprehensive evaluation framework comparing our hybrid approach against single-model baselines 4. A practical application of sentiment analysis for corporate reputation monitoring with actionable insights [ 26 ] The remainder of this paper is organized as follows: Section II reviews relevant literature in sentiment analysis, focusing on social media applications and model architectures. Section III details our methodology, including data collection, preprocessing, and the hybrid sentiment classification approach. Section IV presents experimental results and performance metrics. Section V discusses the implications of our findings and potential applications, while Section VI concludes the paper and suggests directions for future research. II Literature Review II-A Sentiment Analysis on Social Media Sentiment analysis on social media platforms has evolved significantly in recent years, driven by both methodological advancements and increasing recognition of its value. Early work by Go et al. [ 42 ] demonstrated the feasibility of sentiment classification on Twitter using distant supervision, achieving reasonable accuracy despite the platform’s character limitations and informal language. More recently, researchers have focused on addressing the unique challenges of social media text. Zimbra et al. [ 40 ] provided a comprehensive survey of sentiment analysis techniques specifically tailored for social media, highlighting the importance of handling platform-specific features like hashtags, mentions, and URLs. Their work emphasized that standard NLP pipelines often perform poorly without specialized preprocessing for social media content. The temporal dimension of social media sentiment has been explored by Wang et al. [ 33 ] , who developed a real-time sentiment tracking system for emergency response during crisis events. Their findings suggest that sentiment shifts can serve as early indicators of emerging situations, underscoring the value of minimizing latency in sentiment analysis systems. II-B Rule-Based vs. Transformer-Based Models The landscape of sentiment analysis models has traditionally been divided between lexicon-based approaches and machine learning methods, with recent advances introducing transformer architectures as a powerful new paradigm. Hutto and Gilbert [ 46 ] introduced VADER as a lexicon and rule-based sentiment analysis tool specifically attuned to social media. Their evaluations demonstrated VADER’s competitive performance against machine learning approaches while maintaining computational efficiency and interpretability. The model’s ability to handle social media-specific elements like emojis and slang made it particularly suitable for platforms like X. The emergence of transformer architectures like BERT [ 36 ] represented a paradigm shift in NLP, enabling deeper contextual understanding through bidirectional training. Sun et al. [ 34 ] explored how to effectively fine-tune BERT for sentiment analysis tasks, demonstrating substantial improvements over previous state-of-the-art methods. However, the computational intensity of these models presented challenges for real-time applications. As a more efficient alternative, Sanh et al. [ 37 ] introduced DistilBERT, which retained 97% of BERT’s performance while reducing parameters by 40% through knowledge distillation. This made transformer-based sentiment analysis more feasible for time-sensitive applications, though still more computationally demanding than lexicon-based approaches. Comparative studies by Barnes et al. [ 35 ] evaluated different sentiment models across domains, finding that transformer-based models excel at cross-domain generalization but struggle with computational efficiency, while lexicon-based models like VADER offer faster processing but less contextual understanding. This performance trade-off motivates our hybrid approach, which aims to capture the benefits of both paradigms. II-C Real-Time NLP Applications The development of real-time NLP applications presents unique challenges that extend beyond model accuracy to include system architecture, data processing efficiency, and

Read the complete text