Optimizing Big Active Data Management Systems
Shahrzad Haji Amin Shirazi, Xikui Wang, Michael J. Carey, Vassilis J. Tsotras · 2024
Read Optimizing Big Active Data Management Systems by Shahrzad Haji Amin Shirazi, Xikui Wang, Michael J. Carey, Vassilis J. Tsotras (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14519.txt.
Introduction
Shahrzad Haji Amin Shirazi and co-authors speed up active data systems in this 2024 paper. Most data systems sit passive, answering queries when asked. Users now want proactive updates on topics they follow, pushed as data change.
Big Active Data frameworks serve such subscriptions to millions of users. As data and subscriber counts swell, the systems strain. The paper adds three optimizations, starting with strategic aggregation of shared work. Smarter choices about when to compute and when to reuse follow. The aim is scale without waste. The work suits news, finance, and sensor feeds where standing interests number in the millions.
Catalog record
| CREATOR | Shahrzad Haji Amin Shirazi, Xikui Wang, Michael J. Carey, Vassilis J. Tsotras |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14519
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing Big Active Data Management Systems Shahrzad Haji Amin Shirazi, Xikui Wang, Michael J. Carey, Vassilis J. Tsotras Abstract Within the dynamic world of Big Data, traditional systems typically operate in a passive mode, processing and responding to user queries by returning the requested data. However, this methodology falls short of meeting the evolving demands of users who not only wish to analyze data but also to receive proactive updates on topics of interest. To bridge this gap, Big Active Data (BAD) frameworks have been proposed to support extensive data subscriptions and analytics for millions of subscribers. As data volumes and the number of interested users continue to increase, the imperative to optimize BAD systems for enhanced scalability, performance, and efficiency becomes paramount. To this end, this paper introduces three main optimizations, namely: strategic aggregation, intelligent modifications to the query plan, and early result filtering, all aimed at reinforcing a BAD platform’s capability to actively manage and efficiently process soaring rates of incoming data and distribute notifications to larger numbers of subscribers. 1 Introduction In today’s fast-paced digital world, we are inundated with a tremendous amount of data every second. Managing and analyzing this ocean of data, widely known as Big Data, presents formidable challenges and numerous systems have been developed for addressing them. However, the majority of these systems operate in a passive mode, merely processing and returning data in response to user queries. This passive approach often falls short for users who not only want to analyze data but also actively receive updates on new data items that interest them, explore their relationships with other data, and even enrich them with additional information existing in different datasets. These demands have led to the creation of Big Active Data (BAD) frameworks [ 13 , 23 , 33 ] that aim to support extensive data subscriptions and analytics for millions of subscribers. A BAD framework circumvents the inefficiencies of cobbling together multiple independent systems (each dealing with a part of the needed processing, i.e., accessing Big Data, managing incoming streaming data, matching subscribers to information etc). As we confront the relentless expansion of data volumes and the burgeoning number of users interested in this data, the challenge to manage ever-larger datasets becomes increasingly acute. This work focuses on optimizing a BAD platform to better handle soaring rates of incoming data and a growing roster of subscriptions. By deploying a suite of optimization techniques including strategic aggregation, intelligent modifications to the query plan, and early result filtering we have enhanced the BAD framework to its most optimized form yet. These improvements not only streamline data processing without adding more resources, but also significantly broaden the system’s ability to serve a larger and more diverse subscriber base, marking a significant stride forward in the ongoing quest to activate Big Data. 2 Related Work Tapestry [ 30 ] first introduced Continuous Queries as queries that are issued once and then return results continuously as they become available. Tapestry also defined continuous query semantics and created rewrite rules for transforming user-provided queries into incremental database queries. Subsequent research has primarily concentrated on queries involving streaming data. NiagaraCQ [ 16 ] enhanced the scalability and efficiency of continuous queries by breaking them down into smaller, manageable components and clustering similar queries based on their expression signatures. It organized signature constants in a specialized table and used joins to process similar queries as a group. Furthermore, to enhance computational efficiency, the system employed delta files that allow for incremental updates and evaluations of the data. STREAM represents a research prototype that was designed to handle continuous queries across both data streams and persistent storage [ 8 ] . It provided a Continuous Query Language (CQL) for constructing continuous queries against streams and updatable relations [ 9 ] . Most continuous query projects have struggled with scalability, making them less suitable for Big Data use cases, as they often fail to scale effectively in distributed environments. Streaming engines can be used for data processing and data customizing pipelines and to provide real-time analytics. Apache Kafka [ 25 , 5 ] , Apache NiFi [ 6 ] , Apache Flink [ 24 ] , and Amazon Kinesis [ 4 ] are prominent platforms designed to handle and process large-scale, real-time data streams. Similarly, Azure Stream Analytics [ 10 ] and Google Cloud Dataflow [ 19 ] are specialized for stream processing and real-time analytics. These systems are optimized for high-throughput and low-latency processing, enabling them to handle vast amounts of data generated in real-time. However, they are not inherently equipped to provide long-term data storage solutions. Instead, these systems typically require integration with external storage solutions to persist processed data for later use or analysis. This ”glued-systems” approach has been shown to have performance disadvantages in [ 33 ] . Traditional publication/subscription (pub/sub) systems [ 17 , 1 , 21 , 27 , 15 , 14 ] allow subscribers to register their interests in events and to be asynchronously notified about events from publishers. Although pub/sub services can handle a large number of subscribers, users often have to integrate such services with other systems for data processing, and complex computations across multiple data sources are not supported. The systems mentioned above generally face challenges in scaling, persisting data, or handling complex subscription queries, which then requires integrating them with additional systems for effective data processing. The BAD platform, summarized in the next section, was designed and built to address these constraints. In this current work, we focus on further optimizing BAD. One of the key steps involves creating the BAD index, an idea that is conceptually similar to partial indexing [ 29 ] ; it focuses on indexing only the most frequently accessed or queried segments of a dataset, yet is distinct from partial indexing in its implementation as explained in Section 4.3.1 . 3 BAD Preliminaries The BAD platform can enable millions of users to subscribe to data of interest and receive updates continuously. It is different than continuous queries, streaming engines and pub/sub systems as it also supports Big Data analytics with a declarative language, SQL++ (a SQL-inspired query language for semi-structured data [ 12 , 18 ] ). An overview of the BAD platform is shown in Figure 1 . BAD has five basic blocks [ 22 ] : (i) Data Feeds, which manage the ingestion of rapidly arriving data; (ii) Persistent Storage, responsible for storing the data; and an (iii) Analytical Engine, which enables analytic queries on the stored and incoming data to reveal useful information. Additionally, BAD includes (iv) Data Channels and (v) Brokers , which are important for this paper and are described in detail later in this section using an example application. Figure 1: An overview of the BAD platform. There are 3 different types of users for the BAD system; Subscribers who subscribe to channels so they can get the data of interest, Developers who create the BAD channels, and Analysts who run queries on the data stored in the system. For this work we obtained a copy of the BAD open source platform and have extended it further. The existing implementation of the BAD platform [ 23 , 28