This week sees the BPM Think Tank move to Europe, and a chance to observe any differences in BPM expectations and takeup over the previous US event. TIBCO were here to host the round table on “Combining Event Processing with BPMS for Agility” - more on that later.

As it happened I missed the first few minutes of Jan Baan’s  (of ERP fame) keynote, but walked in just as Jan was presenting on the need for complex event processing, and the need for BPM to handle complex events. Of course, BPM systems can consume complex events just like any “normal” events - its just that detection of complex events requires a continuous event-pattern-recognition engine.

Some of Jan’s other noteworthy points were:

• “BPM 2.0″ is human AND integration centric, with the observation that only 8% of processes today are automated.
  • No disagreement from us there: TIBCO iProcess and BusinessWorks are a common combination in TIBCO’s “BPM+”.
• “BPM 2.0″ includes SOA, RIA, business rules, BAM, model-driven, app-server-free.
  • More “BPM+” endorsement.
• “BPMN has no room for intelligence”.
  • Although Jan was probably talking about human collaboration (which AFAIK BPMN does not prevent), this could be considered true if you consider “intelligent processes” to be where rules drive the selection of processes. Of course, a BPMN diagram is as “intelligent” as the process flow it represents - nothing more and nothing less. This is not a fault of BPMN, just of the widespread (and incorrect) view that “process orchestration is the only way to model processes”.
• “ERP is sh** everywhere, and cannot be replaced”.
  • 
•   “BPM is the replacement for ERP”
  • Presumably this was meant to be considered separately from the prior statement on ERP…

As usual, the 2 Event Processing round tables were interesting, with the first as a diverse audience discussion, and the 2nd discussing dynamic / multi-level BPM with an existing iProcess user. Points covered in the former included:

• The role of events to define interfaces between process parts to allow easier re-factoring.
  • This is an interesting idea, and may be covered by the “process meta metamodel” i.e. BPDM. But I’ll have to check …
• Events and complex events driving ad hoc or unordered processes or case management.
  • Yes.
• Events originating from BPM systems (and being collated with other events) to provide insight into processes or BAM.
  • Yes.
• Events and context driving process exceptions without having to define such exceptions in (i.e. avoid polluting the business view of) the main process flow diagram.
  • Makes sense as a BPMN 3.0 suggestion!
• Whether events can play a role in mapping SBVR rule statements to BPMN process diagrams.
  • Possibly, but note that SBVR is about policy-type business rules, and processes are about business operations, there is no direct mapping from SBVR to process anyway. And if you were to create a process from SBVR statements, it’s probably easier to map to a set of event-driven inference rules rather than an arbitrary process flow. Probably we need some PhDs to check this theory out through the TIBCO University Program!

Paul Haley has posted a clear and easy-to-read look at the symbiotic nature of analytics and business rules in the field of “Decision Management”. Furthermore, he goes on to describe a run-time merger of these concepts under the term “Adaptive Decision Management” [*1] or ADM.

Some thoughts on ADM from a CEP perspective:

• Decision performance monitoring is like BAM (or maybe “metaBAM”). Rule firings are events. CEP might be useful to correlate rule use versus data being processed, just as CEP is useful in monitoring BPM, SOA and so forth.

  .

• Running continuous statistical monitoring of rules and rulesets requires some stateful handling of the real-time models [*2]. But a stateful rule engine (like that used in rule-driven CEP) should be able to easily handle that.

  .

• You probably want to cross-correlate “decision performance” statistics (and the real-time equivalents of certain analytic models) across different decision services and decision types. Meaning across different event types. Which means CEP again.

  .

• Proponents of conventional SOA decision services and analytics tools will probably point to (parts of) PMML as the bridge between analytics tools and rule engines. This is very true, but doesn’t avoid the fact that analytics on historical events is, well, historical. In other words it is already out of date, which might (or might not) matter [*3]. So at the very least, doing real-time/”operational” analytics can complement conventional analytics [*4].

  .

ADM indicates another step in the convergence of BI, BAM and decision management, and could well describe some CEP use cases today.

.

Notes

[1] A quick Google search shows Paul to have successfully found an almost unused Three Letter Acronym (although of course the abbreviation has prior art).

[2] Possibly you could treat analytical / statistical results like other data and store them in a data service (/database). But that would require administration, and might be more difficult to handle across distributed rule services. Of course these aren’t unsurmountable - for example you could use some flexible (i.e. schema-free) data repository like an RDF store…

[3] For example, in fraud, finding a new fraud pattern next week doesn’t help this week’s fraud victims.

[4] At least one conventional Data Warehouse company seems to agree with this!

The recent InfoQ conference in London had some interesting banking sessions on high volume / low latency (implying event processing). These included Merrill Lynch on monitoring messages, and Betfair on their online betting system. One can be pretty sure there is CEP in production in these organizations; Merrill’s talk was about monitoring messages or events, which seems to be a popular CEP application area (for TIBCO CEP in any case).

Scenarios for “message monitoring” include:

• I have some high value events occuring that represent control messages for my internal systems. The value of these messages, and the requirement to be aware of any potential problems immediately, means I must monitor them end-to-end to deduce possible failure (events) in a timely manner. This is a classic “sense and respond” application for CEP.
  .
• I have some high value events representing the arrival of important documents in my (managed) business process. The value of these documents, and the requirement to be aware of any potential problems immediately, means I must monitor them end-to-end in my workflow to ensure that the appropriate service levels are met (from the document managament, as opposed to process management, perspective). This is a classic “situation awareness” application for CEP.
  .

In both these cases, we need to monitor the events and use rules (and the state of the entities being monitored) to deduce if all is well or of some complex business event [*1] has occured that requires attention. The CEP engine manages the state and rules and event channels.

Notes:

[1] Talking of business events, there is an ongoing marketing onslaught from IBM promoting “Business Events” for their new CEP acquisition (versus TIBCO BusinessEvents, er, the original-named product). Somehow, this reminds one of the Lord Of The Rings: the searchlight from IBM’s Marketing Tower has swept across the IT plains and stopped at CEP: now the blue hordes are trying to rename CEP for their own (dark?) purposes… .

Interesting to see a fascinating blog entry by Roeland Loggen on “Future Patterns of BPM Technology” which showed CEP as a solution to semi-structured processes, between Straight Through Processing for full structured processes and Agent technology and AI for unstructured processes. As it happens, I had just been discussing Agent Technology with agent-guru James Odell, who is leading the OMG standardization efforts around Agent technologies. Earlier this week, reading the description of Event Processing given by Jack Van Hoof in his SOA-EDA blog on “Event Processor Tracks State of Objects”, it had seemed like CEP could easily map to agent technologies . But would it be worthwhile?

• Agent technology handles extreme scaleability issues by going from “database” processing (single instruction, multiple data records) to “via message” processing (instruction decomposition via separate multiple agents and their intercommunication). Note that CEP usually uses a half-way approach: “in-memory and cache”.
  .
  
• Agents are characterised by being autonomous, having interactions, and being adaptive. CEP engines can be autonomous and interactive to the extent that they simply respond to multiple (complex and continuous) events; adaptiveness could be via machine-learning or more commonly via statistical functions. The main attribute of CEP engines is “memory” (of event histories) which presumably is an advantage in “intelligent” distributed agents too!
  .
  
• Agent Communication Languages often combine rules with agent control commands: indeed CEP engines could be considered (or used as) coarse-grained agents (although typically their control and behaviour are relatively fixed compared to some more dynamic agent-based approaches). One suspects that agent communication languages could easily embed the W3C Rule Interchange working group rule format.
  .
  
• One needs to be aware of the latency issues in communications between agents, especially for time-constrained problems. Agent techniques may be ideal for things like continuous scheduling where they might be controlled by a CEP engine that determines when to reschedule.
  

.
With CEP being associated with event-driven middleware, and such middleware being an integral part of agent communities, it seems there is likely to be some cross-fertilization in future between these technologies.

Further reading

Apart from Jim’s papers, it is interesting to see some real-time / event driven thoughts on agents here, and thoughts on Enterprise Agents here and here.

There is increasing hope for a conceptual “IT supercollision” around the corner. From Business Architecture and Business Modeling, via Model-Driven Engineering and its mappings to BPM (primarily for human-oriented workflow) and Business Rules (primarily for decision automation), with an underlying SOA infrastructure of flexible re-usable services, with EDA from various systems being processed through CEP (for straight-through processing and BAM as well as real-time BI and analytics), software technologies just “seem” to be coming together.

One example of this view is Michael Dortch from BPM In Action: blogging on what he calls the “big mash-up”. I liked Michael’s comment on business core needs as 2 points:

1: The ability to base every business action, decision, and process on the most accurate, consistent, secure, and timely information available, without fail.

2: The ability to answer the “Journalism 101” questions about that information – who’s using what, when, where, why, and how – accurately and completely, on demand at any time.

The first need points to BPM (for human-oriented or workflow processing) and CEP (for timely a.k.a. event-driven decisions and processes). The second points to BAM and “situation assessment” - these also probably utilizing CEP, as well as things like MDM and BI. And, of course, you drive the development of processes and rules in 1. from the results of 2.

Where we might disagree with Michael (and not suprisingly, as his view is primarily BPM-focussed), is when he says “even if you could build an EDA or an SOA without BPM, I’m not sure I’d want to see the results.” Now BPM is “event driven” (usually in terms of orchestrated workflows) and represents simple/singleton event processing, but EDAs are usually associated with straight-through-processing or monitoring of events in an automated fashion, for which CEP may be more appropriate. And BPM features like BPMN (with its current semantics) do not apply to event processing technologies like CEP. I could use BPM to determine which events get processed automatically in a CEP system, but in reality the real-time nature of most events means that I am far more likely to do things the other way round. In other words, use CEP to determine which events / identify the appropriate “complex events” that get processed manually by BPM or in particular processes. In other words, use CEP as a preprocessor to BPM.

So we’d probably reword Michael’s statement as: “even if you can build BPM without an EDA or an SOA, I’m not sure it would be complete.”