The “season to be jolly” is coming up fast, which given the world news might be more of a challenge this year. Therefore I quote the following,  found accidentally while researching for a blog post the other day, and attributed to the late Spike Milligan:

Two hunters are out in the woods when one of them collapses. He doesn’t seem to be breathing and his eyes are glazed. The other guy whips out his phone and calls the emergency services. He gasps, “My friend is dead! What can I do?” The operator says, “Calm down. I can help. First, let’s make sure he’s dead.” There is a silence, then a shot is heard. Back on the phone, the guy says, “OK, now what?”

There was an interesting article on predictive analytics and CEP in Insurance & Technology this month, as reported on David Luckham’s site. The article commented on day-by-day / real-time high-volume data analysis using predictive analytics and complex event processing. Though the article didn’t draw the logical conclusion (i.e. there are use cases for applying both forms of pattern analysis, i.e. deep-historical and real-time), nor did it mention TIBCO (!) [*1], it’s a good read.

Notes:

[1] TIBCO is one of the few vendors with strong offerings in both the analytics and CEP technology markets.

Some people seem to be quite surprised to find that TIBCO BusinessEvents, classed as a Complex Event Processing product, includes a “rule engine”. Especially those that look at the CEP tools market and just see the stream-based CEP products doing continuous queries. Nonethless, there are good reasons why a production rule engine is ideal not just for business decisions, but also event filtering and correlation.

First, lets classify “rules” and “production rules”. Production rules are a type of rule that represent an “IF THEN” statement, but typically applied not to single data variables, but to classes of data (and events). Many commercial rule engines, including BusinessEvents, use developments of the Rete algorithm devised in the 1970s as an efficient pattern matching algorithm for dealing with multiple patterns and multiple data elements. The rule engines that use this algorithm typically provide “forward chaining” in that any rule’s action can update data tested in some other rule, and the rule engine can automatically fire the new rule too. The “interesting” aspect of this is that the rule engine determines when rules fire, not the programmer ordering the rules in some way. For this reason, production rules in rule engines are considered “declarative” - their execution is independent of the order they are defined, which makes design-time scalability easier. Needless to say, the Rete algorithm means that the cost of adding rules to a system is not linear - twice the rules take far less than twice the space and far less than twice the execution time.

The fact that production rules are defined in terms of classes adds another dimension in flexibility. These “declarations” in BusinessEvents, or RuleVariables in the OMG PRR standard, add runtime scalability - the same rule can execute for 1 or 1000 instances of a class, without any additional configuration.

So there are 3 basic parts of a (production) rule (used in a Rete-type rule engine): declaration/RuleVariables, condition, and action:

• Declarations extend the “scope” of a rule.
  The declaration is a list of event types and classes to be used in the rule. At runtime each declaration matches against every instance (and the rule is therefore effectively compared to every combination of instances - or RuleVariable tuples).
  Example: a declaration of CustomerCallEvent and ActiveCustomer for 2 CustomerCallEvents and 50 Customers will create 100 possible combinations or tuples to be considered. Adding a declaration of CustomerOrder with 100 instances increases this to 10,000 combinations or tuples.
  This illustrates both the flexibility and potential cost of production rules - although of course the Rete algorithm can scale to very large tuple numbers.
• Conditions limit the “scope” of a rule.
  The declarations are filtered and joined in the conditions. At runtime each combination of declated event types and data classes is filtered and joined to see if the rule is valid (and what instances of declarations pass the condition list).
  Example: we may filter CustomerCallEvents to just those that are order-related, and associate them with the relevant ActiveCustomer and CustomerOrder. Only 1 CustomerCallEvent is order-related, matches to only 1 ActiveCustomer, and 2 CustomerOrders. Therefore we end up with 1 x 1 x 2 valid tuples or (potential) rule firings.
  This illustrates how we can combine events and data in multiple ways, irrespective of event arrival order. Indeed, if we want to test a particular ordering of events, we have to add an explicit condition to handle that requirement.
• Actions execute the resultant event processing action or decision for a rule.
  Typically, the actions will be executed for each set of declarations (tuple) that satisfy the condition part. Note that there is one additional runtime possibility here: the action for 1 instance can update the data used by another, making the condition fail.
  Example: the action for the CustomerCallEvent and associated ActiveCustomer for the first CustomerOrder handles the event satisfactorily, abstracting it to a new event, and the action “consumes” (removes) the CustomerCallEvent. This means that the 2nd rule firing (for the same CustomerCallEvent and ActiveCustomer, but the 2nd CustomerOrder) is no longer valid as the CustomerCallEvent no longer exists. So this rule does not fire a 2nd time, in this case.
  This illustrates the dynamic nature of rule processing. Where the rule also creates new events or updates existing data instances, then new rule firings may become possible (rule declarations and conditions become valid). This process, and the subsequent firing of other rules, is called rule inferencing.
• In addition, rules may have names, priority numbers and other metadata. Rule priorities are to aid in “conflict resolution”, a fancy term for what happens when 2 or more rules are valid to fire: which do we want to fire first?

For event processing purposes, the production rule format and inferencing rule engine has a number of advantages:

1. Declarative rule definitions: I do not need to worry about the order I define my rules as this has no effect on their execution (except possibly where 2 rules can fire at the same time and I have not specified any priority - which normally means I don’t care). Adding new rules simply means adding them to a list, rather than worrying about where they need to be placed to be executed.
2. Efficient declaration-based definitions: defining rules at the class level makes for generic rules that scale regardless of the number of instances I expect them to match against - indeed I may not know at design time the number of instances expected at runtime.
3. Efficient pattern matching for rule conditions: the Rete-algorithm ensures (mostly) that runtime performance for event patterns, data patterns, and mixed event-data patterns is very good. All vendors have optimized their pattern matching for maximum performance (TIBCO is no exception with a combined code generation and Rete approach).
4. Dynamic pattern behavior: condition filter statements can be defined as variables (or in BusinessEvents, scorecard entries) and adjusted on-the-fly by monitoring the results of other rules and processing.

So production rules enable the definition of arbitrarily complex and numerous event patterns. When combined with an event-aware rule engine, production rules provide an Event Condition Action (ECA) rule mechanism combining the features of business rule engines with event-driven behavior. The rules can represent basic raw event processing, complex (abstract) event processing, and event-based decision processing - using rulesets to differentiate these as appropriate! They can be combined with mechanisms like state models and query languages. They are a proven technology, and they work well.

Other References:

Earlier posts covered the value of state modeling to CEP, the differences between rules and decisions and decisions and CEP [*1], and CEP vs BREs and BRMSs.

Notes:

[1] The difference between event-processing rules, event-driven rules, and rules for business decisions has been much-blogged-about. See these blogs by JT (twice!), Opher and Marco - the latter 2 pretty much covering the other, albeit ECA-based, rule-based CEP systems. In summary, of course,  event-processing rules, event-driven rules, and rules for business decisions can all overlap, depending on the application.

An industry colleague mentioned to me over the Summer that they had attended a “Cloud Computing” talk where it was mentioned that cloud computing would overtake Complex Event Processing. I didn’t get a good response as to what “overtake” meant in this context - that all CEP will be done in cloud computing land? - that cloud computing will overtake CEP in the hype charts (assuming CEP is there anyway)? - that clouds of events will be processed near-source in-the-cloud is the inevitable future for event processing?

To some cynics, cloud computing will be seen purely as a means of delivering virtual server farms, primarily for SaaS (Software As A Service) vendors and their customers, who in turn for the most part are simply part of a renamed ASP (Application Service Provider) market. Amazon’s EC2 seems to be a pretty good example of cloud computing today, and it’s name (”Elastic Compute Cloud”) pretty much describes its function - the virtual mainframe resource for everyone else.

“Computing resources for data / event processing” are a somewhat orthogonal problem to the actual software algorithms used in CEP and event-based decisioning. There is surely no theoretical reason why you could not deploy a CEP application onto EC2 / into a cloud, although latency from event source to the cloud and back again could be an issue, as well as the need for reliable high performance shared data stores. On the latter point, I’m not sure how easy or performant TIBCO BusinessEvents‘ data grid would be on EC2, whose current uses seem almost entirely to be web-service and SOA-based. On the other hand, “the cloud” is possibly a good solution for global-scale event sources, for example as would be found in a large-scale distributed social networking CEP application.

So is there any connection between the terms “event-cloud processing”, as done by CEP, and “cloud computing”? Not particularly, as today the sources of events are not the “same cloud” as the cloud computing centres. On the other hand, I would not be at all surprised to find that the cloud computing architectures relied on event processing techniques to manage users, SLAs and pricing for their systems.

Index

Past alphabet soups have compared CEP to (2) BI, (3) BAM, (4) BPM, (5) BPEL, (6) SOA, (7) SQL, (8) MDM, (9) ETL. Future TLAs are TBA.

Although we contributed one session (on PRR and decisions) to the RuleML conference co-hosted with BR Forum this year, I missed some of the other but nontheless significant talks. I see the keynotes are now officially published, so below are some reviews of the presentations from the web:

• Michael Kifer of F-logic fame presented on the W3C RIF Rule Interchange Format. This is a nice intro to RIF, by one of its main authors. Interestingly RIF is the logical (pun unintended) successor to much of the old RuleML research work on logic language markups, although somehow I doubt we’ll see RIF08 next year instead. My only contrary thought about this presentation is that, for the wider audience, it could have listed some of the other rule / logic languages currently in progress in the R&D communities, and how they might fit in a hierarchy of RIF dialects.
• Paul Haley of Automata presented on the need for a merged ontology for process, rules, events and state, extending the ideas he presented in the BRForum session earlier. This presentation directly addressed the CEP community, too:
  • The knowledge management handled by the BRMS community should also impact CEP and BPM…
    • For example, see TIBCO BusinessEvents’ Decision Manager (although we don’t make any specific claims around knowledge management per se, you can see where Paul was heading here).
  • The integration of rules and process is currently inadequate, with rules playing 2nd fiddle to process
    • Funnily enough there was a similar discussion at this week’s BPM Think Tank in the context of semantic BPM versus say rule representations of process (e.g. for the purposes of say verification of process steps, looking for redundancy, etc).
  • Questions whether TIBCO will build a BRMS to compete in BPM…
    • TIBCO BusinessEvents’ Decision Manager already provides a BRMS for event-driven automated processes…
  • PRR is inadequate but probably more important than RIF
    • Certainly PRR 1.0, sans expression language, is “inadequate” compared to what it should provide with a common expression language. To be fair, RIF PRD can be considered (loosely) to be PRR compatible (and mappings should be straightforward), and hopefully they will remain in sync. But PRR could help bridge the semantic gap - for example be used in both CEP and BPM (and SOA) models.
  • Natural language or spreadsheet metaphors are critical
    • So far, TIBCO BusinessEvents’ Decision Manager supports the latter.
  • There is a need for semantic BPM and semantic CEP
    • Researchers tend to agree here, but the customer benefits still need to be proven. Another PhD topic?
  • When will major BPM/CEP/rule vendors have ontologies?
    • Arguably only TIBCO fits in this group (with IBM and Oracle being only “minor” and “too new” in some of these categories). And we are probably more interested in customer / domain ontologies rather than cross-computing-platform ontologies. So far.
• Ben Grosof of Vulcan presented his latest knowledge representation project, SILK. I didn’t get to speak to Ben about the importance of time and events in knowledge representations, and these are not really covered in his presentation, but I got the impression Ben does understand this.
• David Luckham of Stanford presented on CEP. David probably should have done a keynote for the main Business Rules Forum rather than just present to the rule markup community. Oh well, maybe next year…

The other presentations are up too. I won’t go through all of them, but in particular note Mark Linehan’s talk on an ontology for time (and thence an ideal basis for a business-level CEP language), and Harold Boley and Ben Craig’s personal agents (using distributed rule systems like BusinessEvents allows). Mark Linehan won the best paper award, justifiably, for his work on mapping SBVR rules (i.e. business policy constraints) to UML classes and OCL (i.e. object constraints) - which will be interesting to compare with Microsoft’s research work in this area.