Microsoft’s entry into complex event processing, their StreamInsight or MSSI product (a SDK for continuous queries), is likely to provide lots of opportunities for the .NET developer community to try out event stream processing. So far as one can tell, this has not had a huge effect on the existing CEP vendors focussing on query processing. This lack of effect was also true when Microsoft bundled a rule engine into .NET, which had little effect on the Business Rule Engine market. Probably MSSI will simply educate the .NET community, some of whom will use it, while others will move on to explore more mature or capable solutions.

One interesting follow-up question is: will the Java community, now under the direction of Oracle, follow suit and add some stream processing capability? Will some add-on for J2SE deal with some Java notion of events, EDA and CEP?

Probably not. Most of the business world has moved on from the notion of 3GL programming, and instead business analysts tend to specify model solutions using UML or BPMN etc. In other words, they have moved on to a model-driven view of the world. Of course, the Java and C# worlds continue on, with developers working hard on areas that BPM (or design-by-pictures) cannot handle (for various reasons). So I won’t rule out a JSR-933 Continuous Query API, but I suspect an ANSI committee would rather standardise some continuous query extension to SQL first…

Notes: possibly interesting exiting JSR s include:

• JSR-159 (withdrawn) - a Process Component (JPC) API for matching event to process, which probably died because it had to layer ontop of the J2EE stuff (hence making it an extension to non-event-centric heavyweight API).
• JSR-190 - an Event Tracking API for J2ME - potentially quite interesting for use in non-mobile environments too.
• JSR-158 - a Java Stream Assembly API for media stream multiplexing.

Brenda Michelson over on ebizQ’s Business-Driven-Architect blog had an interesting post on an article published in Communications of the ACM titled “Data in Flight”. The article tries to explain the idea of “data in motion” and the benefits of event stream processing, presumably assuming an audience of database folks.

Brenda’s quotes from the article are probably a good place for a bit of commentary:

“The streaming query engine is a new technology … “
… if you mean “new” is 10 years old or more. But maybe the author means “new” to the presumed reader? Although I would have thought that readers of this particular journal would be well advised on such technology trends…

“CEP has been used within the industry as a blanket term to describe the entire field of streaming query systems.”
Actually CEP covers multiple types of event processing, including continuous queries on event streams. Is there something fundamentally different between an event stream (from some external source) and a “data stream” (for some internal source)? No, I don’t think so.

“This is regrettable because it has resulted in a religious war between SQL-based and non-SQL-based vendors…”
Religious war? Most people accept that there are many types of event processing languages, and these are best suited to different types of event operations - and we see this in a variety of leading CEP vendors. Indeed, 2 out of the top 3 market leading CEP vendors (including TIBCO as the market leader) do not rely on SQL-based continuous queries (although this is indeed available, for tasks like stream processing, in TIBCO BusinessEvents).

“… in overly focusing on financial services applications, has caused other application areas to be neglected.”
Overly focusing on financial services applications? Well, it is certainly true that many CEP vendors focus on algorithmic trading, probably contributing to some vendors’ eventual demise during the recent financial downturn. But TIBCO’s main CEP customer base is in transport, logistics, and telecom, closely followed by government and then financial services.

“Because of their shared SQL language, streaming query engines and relational databases can collaborate to solve problems in monitoring and realtime business intelligence. SQL makes them accessible to a large pool of people with SQL expertise.”
Well, continuous query languages tend to be loosely based on SQL, but have different semantics. For example, in TIBCO BusinessEvents Query Language (BQL), there is a policy statement that defines things like time window, resultset size, etc. Such a continuous query, executing asynchronously, is probably not something a synchronous SQL / stored procedure developer will be automatically comfortable with.

“…streaming query systems can support patterns such as enterprise messaging, complex event processing, continuous data integration, and new application areas that are still being discovered.”

Well, this is truly where the ACM article reviewers missed some hype. As a subset of CEP, for sure, continuous queries “support” operations such as routing events, identifying event patterns, merging event data, and so forth. But so do other event processing technologies (and TIBCO customers do these with both production rules and queries…).

So one might conclude that this article is some sort of clever marketing from a stream-processing start-up. Which, indeed, it probably was - and all kudos to them for their success in getting this published in ACM. I believe it was aimed at encouraging database-users out of their “transactional shells” into the “real-time world of events” - which is possibly a frightful proposition to the CIOs and DBAs used to the staid and stable world of “data at rest” (and presumably on a database somewhere).

And hopefully someone (maybe from EPTS?) will counter with a more balanced article on event processing applications and use cases in the near future.

Interesting to see the rumors were true and Sybase has acquired the “assets of” specialist vendor Aleri. This was a bit bizarre, given that not a year ago Sybase announced they had acquired a source code license for Coral8, shortly before Coral8 merged into Aleri. Now the Sybase folks appear to have paid twice for the same privilege, having no doubt found that “acquiring a technology” does not gain you the brains needed to sell, deploy and develop it.  This acquisition has 2 minor consequences:

• the CEP Vendor family tree for 2010 is out of date already!
• the remaining CEP vendors focusing mainly on financial markets and algo-trading will no doubt have to deal with a competitor willing to give query-based stream-processing software away as a part of  database deals; they will need to be redoubling their efforts to try and break into other markets.

Meanwhile, good luck to the Aleri and Coral8 folks joining Sybase! (Or moving elsewhere, of course…)

A colleague asked for some comments on the terms “CEP” and “ESP” - and in particular why, if a customer already had a stream processing product, they might still need some alternative or non-stream technology for certain other event processing duties…

1. There is growing industry consensus on the “CEP” and “ESP” terminology. “Event stream processing” is for things like market data streams (usually a high ratio of event throughput vs numbers of queries), whereas “complex event processing” is a superset term that also covers event-by-event processing and aggregation (e.g. on potentially out-of-order events from a variety of sources - often with large numbers of rules or business logic).

Sources:
a. The EPTS Glossary by most of the CEP vendor community covers these terms. In particular it defines Event Stream Processing as the processing of in-order events.
b. The “inventor” of the term CEP, David Luckham, provides an excellent differentiation on his web site.
c. Analyst comments on this differentiation by Bloor .
d. 3rd party blog comments by Opher Etzion , by Jack vanHoof , and by Tim Bass .

2. Generally “stream processing” processes event streams as “sets” and does “set type” operations, typically using “continuous queries” (SQL-type queries that operate over time and buffer windows). The wider “complex event processing” term additionally covers other mechanisms like ECA rules, production rules, and so forth. By definition, ESP products are therefore also CEP products in the same way that both Edsels and Bugattis are all cars.

While streaming query engines are very useful technologies they are not optimal for all CEP problems - in particular they have different characteristics regarding, for example, comparing out-of-order or out-of-stream events, applying decisions and reactions to event patterns, and so on. For this reason multiple CEP language classes have evolved, typically described as queries, rules and procedural approaches (to event pattern detection). This is also why TIBCO BusinessEvents explicitly supports state models and production rules as well as continuous queries, and is architected to allow additional Event Processing Languages to be added in future if desired.

Sources:
a. TIBCO Blog on Queries based on TIBCO’s coverage of ESP. Other ESP vendors are commented on in another post.
b. EPTS Language Working Group is defining some differentiations here but have not published their work yet: we blogged on a preview at a recent conference.

3. Because ESP is but a specialised subset of CEP, some people may have tried to make “ESP” synonymous with “CEP”. This remains a very grey area as while event stream processing using queries is a very useful capability (i.e. event set handling and detection), so is event-by-event handling and correlation using, say, inference rules.

So… in most enterprises there are usually multiple use cases for multiple types of CEP that are best handled by multiple paradigms (such as specialist ESP, event-driven business processes, rule-driven event processing, event-based business rules, event-driven analytics, etc). One should no more expect a large company to rely on a single CEP paradigm as it would on a single computer hardware technology.

… with the announcements and discussions of Microsoft’s SQLServer-or-.NET-library-no-one -seems-quite-sure-yet stream processing tooling, and then IBM’s you-can’t-have-too-many-CEP-tools System S announcement. Looks like IBM has donated this software to some good causes as part of its beta program - good on IBM - although presumably future users will be expected to pay for the technology.

These arrivals mean that, finally, all the Big 3 (Oracle, IBM, Microsoft) have now declared some interest in query-based stream processing. Possibly this will increase interest (from a somewhat low base) for standardising SQL extensions (syntax and semantics) for continuous queries (as also provided for in TIBCO BusinessEvents 3.0 and later). We’ll see…