Brenda Michelson over on ebizQ’s Business-Driven-Architect blog had an interesting post on an article published in Communications of the ACM titled “Data in Flight”. The article tries to explain the idea of “data in motion” and the benefits of event stream processing, presumably assuming an audience of database folks.

Brenda’s quotes from the article are probably a good place for a bit of commentary:

“The streaming query engine is a new technology … “
… if you mean “new” is 10 years old or more. But maybe the author means “new” to the presumed reader? Although I would have thought that readers of this particular journal would be well advised on such technology trends…

“CEP has been used within the industry as a blanket term to describe the entire field of streaming query systems.”
Actually CEP covers multiple types of event processing, including continuous queries on event streams. Is there something fundamentally different between an event stream (from some external source) and a “data stream” (for some internal source)? No, I don’t think so.

“This is regrettable because it has resulted in a religious war between SQL-based and non-SQL-based vendors…”
Religious war? Most people accept that there are many types of event processing languages, and these are best suited to different types of event operations - and we see this in a variety of leading CEP vendors. Indeed, 2 out of the top 3 market leading CEP vendors (including TIBCO as the market leader) do not rely on SQL-based continuous queries (although this is indeed available, for tasks like stream processing, in TIBCO BusinessEvents).

“… in overly focusing on financial services applications, has caused other application areas to be neglected.”
Overly focusing on financial services applications? Well, it is certainly true that many CEP vendors focus on algorithmic trading, probably contributing to some vendors’ eventual demise during the recent financial downturn. But TIBCO’s main CEP customer base is in transport, logistics, and telecom, closely followed by government and then financial services.

“Because of their shared SQL language, streaming query engines and relational databases can collaborate to solve problems in monitoring and realtime business intelligence. SQL makes them accessible to a large pool of people with SQL expertise.”
Well, continuous query languages tend to be loosely based on SQL, but have different semantics. For example, in TIBCO BusinessEvents Query Language (BQL), there is a policy statement that defines things like time window, resultset size, etc. Such a continuous query, executing asynchronously, is probably not something a synchronous SQL / stored procedure developer will be automatically comfortable with.

“…streaming query systems can support patterns such as enterprise messaging, complex event processing, continuous data integration, and new application areas that are still being discovered.”

Well, this is truly where the ACM article reviewers missed some hype. As a subset of CEP, for sure, continuous queries “support” operations such as routing events, identifying event patterns, merging event data, and so forth. But so do other event processing technologies (and TIBCO customers do these with both production rules and queries…).

So one might conclude that this article is some sort of clever marketing from a stream-processing start-up. Which, indeed, it probably was - and all kudos to them for their success in getting this published in ACM. I believe it was aimed at encouraging database-users out of their “transactional shells” into the “real-time world of events” - which is possibly a frightful proposition to the CIOs and DBAs used to the staid and stable world of “data at rest” (and presumably on a database somewhere).

And hopefully someone (maybe from EPTS?) will counter with a more balanced article on event processing applications and use cases in the near future.

Interesting to see the rumors were true and Sybase has acquired the “assets of” specialist vendor Aleri. This was a bit bizarre, given that not a year ago Sybase announced they had acquired a source code license for Coral8, shortly before Coral8 merged into Aleri. Now the Sybase folks appear to have paid twice for the same privilege, having no doubt found that “acquiring a technology” does not gain you the brains needed to sell, deploy and develop it.  This acquisition has 2 minor consequences:

• the CEP Vendor family tree for 2010 is out of date already!
• the remaining CEP vendors focusing mainly on financial markets and algo-trading will no doubt have to deal with a competitor willing to give query-based stream-processing software away as a part of  database deals; they will need to be redoubling their efforts to try and break into other markets.

Meanwhile, good luck to the Aleri and Coral8 folks joining Sybase! (Or moving elsewhere, of course…)

A colleague asked for some comments on the terms “CEP” and “ESP” - and in particular why, if a customer already had a stream processing product, they might still need some alternative or non-stream technology for certain other event processing duties…

1. There is growing industry consensus on the “CEP” and “ESP” terminology. “Event stream processing” is for things like market data streams (usually a high ratio of event throughput vs numbers of queries), whereas “complex event processing” is a superset term that also covers event-by-event processing and aggregation (e.g. on potentially out-of-order events from a variety of sources - often with large numbers of rules or business logic).

Sources:
a. The EPTS Glossary by most of the CEP vendor community covers these terms. In particular it defines Event Stream Processing as the processing of in-order events.
b. The “inventor” of the term CEP, David Luckham, provides an excellent differentiation on his web site.
c. Analyst comments on this differentiation by Bloor .
d. 3rd party blog comments by Opher Etzion , by Jack vanHoof , and by Tim Bass .

2. Generally “stream processing” processes event streams as “sets” and does “set type” operations, typically using “continuous queries” (SQL-type queries that operate over time and buffer windows). The wider “complex event processing” term additionally covers other mechanisms like ECA rules, production rules, and so forth. By definition, ESP products are therefore also CEP products in the same way that both Edsels and Bugattis are all cars.

While streaming query engines are very useful technologies they are not optimal for all CEP problems - in particular they have different characteristics regarding, for example, comparing out-of-order or out-of-stream events, applying decisions and reactions to event patterns, and so on. For this reason multiple CEP language classes have evolved, typically described as queries, rules and procedural approaches (to event pattern detection). This is also why TIBCO BusinessEvents explicitly supports state models and production rules as well as continuous queries, and is architected to allow additional Event Processing Languages to be added in future if desired.

Sources:
a. TIBCO Blog on Queries based on TIBCO’s coverage of ESP. Other ESP vendors are commented on in another post.
b. EPTS Language Working Group is defining some differentiations here but have not published their work yet: we blogged on a preview at a recent conference.

3. Because ESP is but a specialised subset of CEP, some people may have tried to make “ESP” synonymous with “CEP”. This remains a very grey area as while event stream processing using queries is a very useful capability (i.e. event set handling and detection), so is event-by-event handling and correlation using, say, inference rules.

So… in most enterprises there are usually multiple use cases for multiple types of CEP that are best handled by multiple paradigms (such as specialist ESP, event-driven business processes, rule-driven event processing, event-based business rules, event-driven analytics, etc). One should no more expect a large company to rely on a single CEP paradigm as it would on a single computer hardware technology.

… with the announcements and discussions of Microsoft’s SQLServer-or-.NET-library-no-one -seems-quite-sure-yet stream processing tooling, and then IBM’s you-can’t-have-too-many-CEP-tools System S announcement. Looks like IBM has donated this software to some good causes as part of its beta program - good on IBM - although presumably future users will be expected to pay for the technology.

These arrivals mean that, finally, all the Big 3 (Oracle, IBM, Microsoft) have now declared some interest in query-based stream processing. Possibly this will increase interest (from a somewhat low base) for standardising SQL extensions (syntax and semantics) for continuous queries (as also provided for in TIBCO BusinessEvents 3.0 and later). We’ll see…

In TIBCO BusinessEvents 3.0 TIBCO introduced the BusinessEvents Query Language (BQL) [*1] for continuous (and static) queries against cached events and concepts - thereby adding a “stream processing” component to the “rule processing” BusinessEvents event processing platform, like the bespoke event stream processing engines out there.

Interestingly, there seems to be a trend in real-time event store / database technologies adopting continuous query languages. For example, apart from the event stream processing engines, there are also continuous query languages for both Oracle’s and IBM’s cache technologies. Even Sybase has apparently licensed an ESP engine. One wonders when the the big DB tools and ANSI SQL guys will start supporting and promoting stream processing constructs?

Notes:

[1] BQL is based on the object query language OQL but with continuous query extensions.