What I Do At Work

part of a series. A note sent out today to the worldwide financial services crew on how I spent my weekend:

FYI: Continuous Query, the Next Big Thing in Streaming Data

In doing some research for solutions around real-time streaming data engines (e.g. Thompson, Reuters), it’s becoming clear that the next generation of quote engines is going to look quite different from the current one. Today a lot of streaming data technology is based on proprietary versions of the client/server model that we are all familiar with. Often the client side is a smart client, the server side is a highly tuned database and the transiting protocol is a web service (or it’s nearest moral variant). However, a lot of research is currently being done on Continuous Query Engines, which are vastly more efficient for processing multiple end nodes than current designs.

The basic idea behind a CQE is to identify which nodes are asking the same types of query , then group them for more efficient service, sending only deltas of information to the waiting nodes rather than long bursts of data. Clients are able to keep track of the deltas and cut down on complex query processing server side.

Below are a couple of papers I found in my research that I thought were interesting and did a good job of getting the concept across, along with at least one implementation.

I pass this along for informational purposes although, for the capital markets folks, this is something we need to really start thinking about. Many thanks to Ed Muth for suggesting this line of inquiry.


http://db.cs.berkeley.edu/papers/sigmod02-cacq.pdf

and
http://delivery.acm.org/10.1145/1170000/1164132/p31-agarwal.pdf?key1=1164132&key2=9067059611&coll=&dl=ACM&CFID=15151515&CFTOKEN=6184618


an interesting implementation

http://java.sys-con.com/read/260054.htm