Estimating quantiles from the union of historical and streaming data

Date
2016-11-01
Authors
Singh, Sneha
Tirthapura, Srikanta
Tirthapura, Srikanta
Major Professor
Advisor
Committee Member
Journal Title
Journal ISSN
Volume Title
Publisher
Altmetrics
Authors
Research Projects
Organizational Units
Journal Issue
Series
Department
Electrical and Computer Engineering
Abstract

Modern enterprises generate huge amounts of streaming data, for example, micro-blog feeds, financial data, network monitoring and industrial application monitoring. While Data Stream Management Systems have proven successful in providing support for real-time alerting, many applications, such as network monitoring for intrusion detection and real-time bidding, require complex analytics over historical and real-time data over the data streams. We present a new method to process one of the most fundamental analytical primitives, quantile queries, on the union of historical and streaming data. Our method combines an index on historical data with a memory-efficient sketch on streaming data to answer quantile queries with accuracy-resource tradeoffs that are significantly better than current solutions that are based solely on disk-resident indexes or solely on streaming algorithms.

Comments

This is a manuscript of a proceeding published as Singh, Sneha Aman, Divesh Srivastava, and Srikanta Tirthapura. "Estimating quantiles from the union of historical and streaming data." Proceedings of the VLDB Endowment 10, no. 4 (2016): 433-444. 10.14778/3025111.3025124. Posted with permission.

Description
Keywords
Citation
DOI