Synergy: Quality of Service Support for Distributed Stream Processing Systems

Many emerging applications in domains such as network traffic management, financial trades surveillance, customized e-commerce applications, and analysis of sensor data, require the real-time processing of large amounts of data that are updated continuously. Distributed stream processing systems offer a scalable and efficient means of in-network processing of such data streams. We propose Synergy, a peer-to-peer middleware that provides Quality of Service support for distributed stream processing applications. Synergy provides sharing-aware component composition, for efficiently reusing data streams and processing components, when composing applications with QoS demands. Utilizing a set of fully distributed algorithms, Synergy discovers and evaluates the reusability of available data streams and processing components when instantiating new stream applications. For QoS provision, Synergy performs QoS impact projection to examine whether the shared processing can cause QoS violations on currently running applications. To proactively identify application hot-spots at run-time, Synergy employs a prediction framework that binds workload forecasting using linear regression with execution time forecasting using correlation. To react to predicted QoS violations and alleviate hot-spots, nodes autonomously migrate the execution of stream processing components using a non-disruptive migration protocol. When deploying new components, Synergy utilizes a decentralized replica placement protocol that aims to maximize availability, while respecting resource constraints, and making performance-aware placement decisions. We have implemented a prototype of the Synergy middleware and evaluated its performance using a real stream processing application operating on real streaming data on PlanetLab, as well as on a simulation testbed. The experimental results demonstrate substantial benefits in performance and QoS provision, while achieving good resource utilization. More information on the Synergy middleware can be found at http://synergy.cs.ucr.edu

cd /home