As I write the last of my event summary reports and follow up action items, I am now officially putting a bow on the TIBCO user conference (TUCON 2008)

What a madhouse. I was accountable for the TIBCO Analyst Summit (TIBCO’s first ever), two days of the Business Optimization track of presentations (combining engineering, customer and analyst presentations) and ensuring that TIBCO CEP BusinessEventsTM booth was staffed and overflowing with interested folks. Add in various podcasts and spokesperson duties for press and customers and it all equaled a stressful, crazed and wholly satisfying event at the same time, if that’s possible.

I met some great folks from all over the world: partners, analysts, customers - in particular, Southwest Airlines and Allstate Insurance – knowledgeable and fine folks in all respects, really pushing the edge of what is possible in IT and event processing.

What I experienced at the conference relative to CEP, was this. It appears that the mainstream messaging, integration and SOA folks are finally getting it and see the value in event driven SOA and are asking for CEP and BusinessEventsTM by name. There is also a groundswell of interest in Business Optimization and some analysts like IDC and other sundry vendors are seeing Business Optimization as the next big thing.

And they’re right. Booyah!.

Quite a few nice posts ended up in the blogosphere that comment on TUCON, SOA, CEP et al – , Sandy Kemsley, Joe McKendrick, Dana Gardner, Tony Baer among others with interesting perspectives.

However, my biggest AHA moment was the realization in listening to our other customers, besides Southwest and Allstate, that some of the greatest applications of CEP are not the big and noticeable “complex” events that marketing folks like to talk about. They are instead using TIBCO BusinessEvents much more low key and pragmatically, like making sure their customers are less irritated or their business can recover quicker from unforeseen circumstances.

I’ll quote my colleague Chris M. when he says “it opens BusinessEvents up to solving common everyday problems which should be easy to do, but for some reason are hard (e.g. Baggage Handling).”

For many of our customers, it’s not overly visible, but it works for them.

And that’s what really matters.

TIBCO has a very successful BPM product, iProcess, that is compliant with standards such as OMG BPMN and WFMC XPDL, has a powerful new AJAX-driven Forms capability, and is generally considered to be one of the leaders of the pack when it comes to current BPM technology. And as a reminder, current BPM technology involves mapping business processes to BPMN flow diagrams, which represents a form of “simple” event processing [*1], to coordinate manual and automated workflows involving human operators, automated services, and so forth.

Interestingly, some TIBCO BPM customers are beginning to exploit CEP techniques alongside BPM to assist in the modeling of complex processes, such as dynamic manufacturing processes. TIBCO iProcess Conductor, for example, uses TIBCO CEP technology to provide event rules that help drive the dynamic specification of processes defined in iProcess. Another example is TIBCO partner Accenture, who use [*2] TIBCO BusinessEvents (BE) to model some of their customers’ dynamic business processes, decisions and exceptions. They utilize BE’s state models and rules to drive their iProcess (/or other BPM tool) workflows that are modeled in BPMN, with a subsequent considerable reduction in BPMN complexity and increase in maintainability. This approach is used in their solution for “Complex Change Request process management” and is described as having the following solution architecture principles [*3] :

• State models used for change-related concepts
• Continuously monitor all events related to the change process to infer
  • State transitions
  • Workflow events (start workflows, receive responses…)
  • Back-office events (new parts, new process owners…)
• Create new monitoring concepts to measure process KPIs continuously
• De-couple meta-model (state & entity models) with workflow models
  • Better support for dynamic changes to the change process

This approach, they find, provides the following benefits in change-request applications:

• Agile change process
  • Dynamic business rules to determine process behavior
  • Separation of concerns of the process meta-model and the approval workflows
• Brings visibility to the status of change processes
  • Identify processes that take too long or that are stuck
  • Reduce the overall implementation time (and cost).

Decoupling complex event and state processing from a BPM system is conceptually similar to decoupling business logic into a decision service (and indeed, the CEP tool in this case also fulfils the role of an event-driven decision engine). Clearly these are not requirements for all Business Processes / BPM applications, but might be of interest to those hitting complexity or sophistication problems with their BPMN models. In addition, BPM pundits who prefer an “everything in BPM(N)” approach rather than a component-based solution / best-of-breed architecture will obviously not like this [*4]. But then again, the customers using this approach seem to like it, and they are the ones that count.

Notes

[1] BPM deals with events that kick off processes, and BPM represents the processing of that event…

[2] Example customers are in electronics manufacturing and banking.

[3] Reproduced here by kind permission of the original author. The Accenture solution here also uses a unified user interface / dashboard.

[4] Bruce Silver comments on the “BPMS Suite” vs “BPM component” market dynamics in his blog mentioning iProcess Conductor from TUCON08. The contradiction here is that if you want to use a CEP / event-driven state and rule engine to drive your BPM, you won’t find an integrated solution from another major BPM vendor. But luckily you *can* (i.e. some customers do) use TIBCO components with non-TIBCO BPM vendors…

Next week sees the TIBCO User Group, TUCON, take place in San Francisco. The schedule includes a number of CEP-related topics; here is a handy cheat-sheet for some of them [*1]:

• Weds 30 Apr, 16.00-16.50: Open the Door to Event Processing
  Track: Introduction: Tibco Products & technology direction
  Can your business sense, analyze, interpret, and respond to event-driven business situations in real time? In this session we explore how to get the maximum value from your SOA by understanding event-based architectures, complex event processing models, and how TIBCO BusinessEvents™ software fits into your architectural strategies.

  .

• Weds 30 Apr, 16.00-16.50: In-Process Analytics: Integration of TIBCO BusinessEvents and TIBCO Spotfire Enterprise Analytics
  Track: Optimization: From Event Management and Monitoring to Analytics
  As companies see the need to have accurate, up-to-date business intelligence (BI) driving their business processes, they are demanding adaptable software solutions to help them do it. The combination of business process and event processing with BI technology makes relevant information available in the right context at the right time. Integrated with the business process, Spotfire can equip every business person to quickly discover new insights and make better decisions in real time. In this session, we’ll look at some initial ways that Spotfire can work in conjunction with TIBCO’s BPM and CEP technologies.

  .

• Weds 30 Apr, 16.00-16.50: Spice Up Your Ajax Applications with Real-Time Data
  Track: Development: Interfaces, Services, and Composite Applications
  Break through the web’s request/response barrier into real-time information for web applications. Learn how to provide highly scalable real-time data services to Ajax applications, HTML pages, and other web technologies using TIBCO Ajax Message Service™. Participants will step through the integration of TIBCO General Interface™ with TIBCO Enterprise Message Service™ using TIBCO Ajax Message Service™ and TIBCO PageBus™. Experience first-hand these powerful new capabilities that make applets, active-x, and desktop-installed clients obsolete.

  .

• Thurs May 1, 10.30-11.20: Southwest Airlines: Using BusinessEvents for Real-Time Problem Detection and Resolution
  Track: Optimization: From Event Management and Monitoring to Analytics
  Southwest Airlines (SWA), the world’s leading low-cost airline, is constantly looking for ways to improve efficiencies and keep costs down while remaining profitable and providing Positively Outrageous Customer Service. Learn how SWA is using TIBCO BusinessEvents™ to build an “Early Alerts System” to help operational groups identify and solve potential issues before they impact operations. Members of the SWA Technology Department will present an overview of the applications they’re building leveraging TIBCO products and how those applications fit into the enterprise SOA.

  .

• Thurs May 1, 14.00-14.50: Trade Lifecycle Management with TIBCO BusinessEvents and TIBCO iProcess
  Track: Architecture 2: From Principles to Best Practice
  Cognizant and TIBCO have come together to create a technology platform addressing the needs of the securities industry to optimize straight through processing and minimize trade breaks. Through knowledge-based analysis, this unique platform provides a rules-driven workflow management of trades and a real-time prediction of threats. Learn how it lowers transaction risk and results in potential large financial savings.

  .

• Thurs May 1, 15.00-15.50: Allstate: Increasing Agility and Customer Centricity
  Track: Optimization: From Event Management and Monitoring to Analytics
  Allstate took an existing TIBCO BusinessWorks™ implementation and added TIBCO BusinessEvents™ to achieve greater agility and customer centricity. Their experience and best practices are applicable to any company considering how to incorporate complex event processing (CEP) into an existing TIBCO enterprise integration environment. Get pragmatic advice and answers to your questions from the developers and implementers behind the project.

  .

• Thurs May 1, 16.00-16.50: Advanced BusinessEvents Implementation Strategies
  Track: Optimization: From Event Management and Monitoring to Analytics
  This session, presented by experts in TIBCO BusinessEvents™ consulting and engineering, will cover topics such as caching techniques, rule and state management strategies, and BusinessEvents tuning techniques through demonstrations and case studies. Come to this session if you want to really understand some of the advanced concepts and deployment features within the BusinessEvents product set from the experts who work with the product on a regular basis.

  .

Notes:

[1] Of course, there are plenty of other interesting SOA and BPM sessions too. Some will also refer to TIBCO BusinessEvents…

Paul Haley has posted a clear and easy-to-read look at the symbiotic nature of analytics and business rules in the field of “Decision Management”. Furthermore, he goes on to describe a run-time merger of these concepts under the term “Adaptive Decision Management” [*1] or ADM.

Some thoughts on ADM from a CEP perspective:

• Decision performance monitoring is like BAM (or maybe “metaBAM”). Rule firings are events. CEP might be useful to correlate rule use versus data being processed, just as CEP is useful in monitoring BPM, SOA and so forth.

  .

• Running continuous statistical monitoring of rules and rulesets requires some stateful handling of the real-time models [*2]. But a stateful rule engine (like that used in rule-driven CEP) should be able to easily handle that.

  .

• You probably want to cross-correlate “decision performance” statistics (and the real-time equivalents of certain analytic models) across different decision services and decision types. Meaning across different event types. Which means CEP again.

  .

• Proponents of conventional SOA decision services and analytics tools will probably point to (parts of) PMML as the bridge between analytics tools and rule engines. This is very true, but doesn’t avoid the fact that analytics on historical events is, well, historical. In other words it is already out of date, which might (or might not) matter [*3]. So at the very least, doing real-time/”operational” analytics can complement conventional analytics [*4].

  .

ADM indicates another step in the convergence of BI, BAM and decision management, and could well describe some CEP use cases today.

.

Notes

[1] A quick Google search shows Paul to have successfully found an almost unused Three Letter Acronym (although of course the abbreviation has prior art).

[2] Possibly you could treat analytical / statistical results like other data and store them in a data service (/database). But that would require administration, and might be more difficult to handle across distributed rule services. Of course these aren’t unsurmountable - for example you could use some flexible (i.e. schema-free) data repository like an RDF store…

[3] For example, in fraud, finding a new fraud pattern next week doesn’t help this week’s fraud victims.

[4] At least one conventional Data Warehouse company seems to agree with this!

The recent InfoQ conference in London had some interesting banking sessions on high volume / low latency (implying event processing). These included Merrill Lynch on monitoring messages, and Betfair on their online betting system. One can be pretty sure there is CEP in production in these organizations; Merrill’s talk was about monitoring messages or events, which seems to be a popular CEP application area (for TIBCO CEP in any case).

Scenarios for “message monitoring” include:

• I have some high value events occuring that represent control messages for my internal systems. The value of these messages, and the requirement to be aware of any potential problems immediately, means I must monitor them end-to-end to deduce possible failure (events) in a timely manner. This is a classic “sense and respond” application for CEP.
  .
• I have some high value events representing the arrival of important documents in my (managed) business process. The value of these documents, and the requirement to be aware of any potential problems immediately, means I must monitor them end-to-end in my workflow to ensure that the appropriate service levels are met (from the document managament, as opposed to process management, perspective). This is a classic “situation awareness” application for CEP.
  .

In both these cases, we need to monitor the events and use rules (and the state of the entities being monitored) to deduce if all is well or of some complex business event [*1] has occured that requires attention. The CEP engine manages the state and rules and event channels.

Notes:

[1] Talking of business events, there is an ongoing marketing onslaught from IBM promoting “Business Events” for their new CEP acquisition (versus TIBCO BusinessEvents, er, the original-named product). Somehow, this reminds one of the Lord Of The Rings: the searchlight from IBM’s Marketing Tower has swept across the IT plains and stopped at CEP: now the blue hordes are trying to rename CEP for their own (dark?) purposes… .