Two events last month showed indicators of a convergence between the analytics world and CEP world.

Firstly Louis Bajuk-Yorgan from TIBCO Spotfire attended the Predictive Analytics World conference in San Francisco. He reported that:

Three key themes showed up multiple times throughout the talks-the growing importance of text mining, the application of net lift modeling to determine the real results of a marketing campaign (ignoring those people who would have bought anyway), and (most interesting to me) the importance of operationalizing predictive analytics.

In his opening keynote speech, Eric Siegel (the conference chair) saw the most important innovation in the field of Predictive Analytics focused on applying predictive analytics to operational decisions (as opposed to more established application areas such as customer churn & product recommendations). In a later talk, James Taylor of Decision Management Solutions (and co-author of the great book “Smart (Enough) Systems”), echoed Eric’s emphasis on operational results, encapsulated in the phrase “Action support, not just decision support.” James advised building an analytic platform that focused on the end game: the need to operationalize analytic decisions.

This is great validation for us, since operationalizing analytics is at the heart of TIBCO’s vision for its combined platform with Spotfire and S+ (as shown in products like Operations Analytics).

Then a week later Andreas Gerst from the TIBCO BusinessEvents team presented at cepconf in Munich, Germany. Andreas presented on CEP and Data Mining, and in particular how both these complement each other for advanced operational intelligence around customer management. Andreas used TIBCO BusinessEvents and TIBCO Spotfire Miner as his example technologies, mentioning techniques like PMML for moving from analytics to real-time event processing technologies.

Intelligent Enterprise invites a few “talking heads” to give their “Intelligent Enterprise Editors’ Choice Awards” every year, so its the time now for the 2010 edition. They selected a “top 12″ and then 12 each in categories of BI, Enterprise Applications and Information Management. Complex Event Processing got a fair share of references:

• Top 12 companies: CEP mentions were for 2 for of the 12 (Informatica and Sybase) who both acquired CEP technologies in the last 12 months; in addition 3 other companies in this list have CEP technology (IBM, Oracle, Microsoft).
• BI companies: CEP gets one mention in this list, under the TIBCO banner:

TIBCO could just as easily be listed among our Information Management Ones to Watch. But this firm sets itself apart by helping companies interpret — as well as manage and integrate — data and events. TIBCO Spotfire provides trend-spotting analysis through easily understood data visualizations. The company also offers complex event processing (CEP) for the broad market, where insight into events will make real-time BI a reality.

• Enterprise Applications: one mention here too, under Progress’ Apama division.
• Information Management: CEP gets a mention here, albeit for Streambase as a merger (aka “closer ties”) candidate for Vertica (although this seems unlikely IMHO).

So how did CEP do on the hypemeter here?
CEP:  5 company summaries included “CEP” or “Complex Event Processing”
Cloud: 7 company summaries included “cloud”
BPM: 2 mentions!
SOA: 1 mention!!!

But the winner was:
Analytics: 9 mentions, and they didn’t even mention this under TIBCO (whose Spotfire S+ is a pretty good contender)!

The CEP-BI-analytics theme was also recognised by Forrester’s James Kobelius in a recent blog posting. James identifies CEP under “analytics”, and says:

In this race, the vendor that has integrated BI, CEP, and PA/DM most effectively for the mass market is TIBCO Spotfire.

He goes on: Just as important,TIBCO’s offerings enable pushdown of predictive models to CEP, data warehouses, business process platforms, and other application platforms.

So if CEP should be counted in the same boat as analytics, then indeed there is a new world order for software technology in 2010:

Cloud, SOA and BPM: 10 mentions
CEP and Analytics: 14 mentions

One of the probable future trends in event processing will be the merger of Predictive Analytics (per Wikipedia: “analyze current and historical facts to make predictions about future events“) and Complex Event Processing… or basically the increasing use of advanced statistical techniques in event processing. Of course, for “current facts” (in Wikipedia) read “events” in the Event Processing community…

As an example, consider the diagram on the right, which details some of the algorithms available in such a predictive analytics tool (TIBCO Spotfire Miner), and how they relate to event processing (as when embedded in a CEP tool like TIBCO BusinessEvents). One thing to consider is that these algorithms are not likely to be specified by Subject Matter Experts or Business Analysts - who might ordinarily be relatively happy with the logical constructs in rule-based event patterns or queries.

Although several folks in TIBCO had used the term “visual analytics” to try and describe what TIBCO Spotfire does, as opposed to merely reporting on views of data, the analysts have continued to position Spotfire as a “BI” tool. But now, the Spotfire marketing folk are starting to use the term “Spotfire Analytics” to better describe the positioning of the product, especially with the combination of S+ and Miner predictive analytics capabilities.

How does this affect TIBCO Complex Event Processing users? Not one jot, of course. Spotfire remains the preferred analytic tool for visualising the event data in a TIBCO BusinessEvents backing store (a.k.a. the “event warehouse”) and comparing that data with other sources. Using Spotfire you can identify new patterns that can then be used to update rules, queries and decisions in a TIBCO BusinessEvents application - or indeed, execute an S+ or Miner analytics model against the incoming events in either a regular, or special purpose, event processing agent.

I was listening to a BBC radio article last weekend about how some attempts to donate by debit or credit card to charity sites supporting the relief efforts for the humanitarian disaster unfolding in Haiti, have been thwarted by banks’ automated anti-fraud measures. Apparently charity donation websites happen to be a favorite for fraudsters testing card numbers to see if they have been blocked or not. And seemingly, some banks’ automated anti-fraud systems are deciding to block donations.

It would be easy to claim that this sort of problem was typical of organisations relying too heavily on machine learning and analytics rather than common sense (prescribed as business rules, perhaps). But this sort of “banking error - oops - oh no - now bad publicity” could just as easily be down to embedded rules in a process that could not be overwritten (or had no process for being overwritten…). Clearly the banks in question have some agility issues - either in management, business processes, business rules, software apps - or some combination thereof. The good news was that their “exception management” processes seemed to work - angry customers contacted their bank and were thence able to make their donation, it seems. A win for the call center, then…