Next from the Distributed Event Based Systems conference: the CQL tutorial by Dieter Gawlick and Shailendra Mishra, covering Oracle’s version of a continuous query language [*1]. What was interesting was their defense of the claim that “your database server can do your CEP” [*2], supposedly reinforced by the observation that “you get all the (usual) DB management functions for free” [*3]. The traditional 3-tier, database-persistence-as-a-core-service, architectural design pattern is of course perfectly fine for, er, moderate-to-high latency projects with stovepipe services requiring minimal changes over their lifetime. Indeed, many current CEP applications augment, rather than replace, such existing architectures. But presumably you also have to consider run-time performance effects [*4] on your operational data store. Also some CQL examples display impressive contempt for the KISS principle - being of such complexity that they will undoubtedly frighten small children as well as experienced DBAs. Maybe Oracle will plan some nifty design tool for their CQL implementation, and if so, this would surely be a good topic for some future DEBS conference…

Of course, “real” ESP and CEP engines also support continuous queries - TIBCO BusinessEvents included; hence at least one answer to Hans Gilde’s recently asked question on why rule-based (monitoring) systems do not include some SQL-based Event Processing Language  is: actually some already do).

Notes.

[1] Probably, at some point, EPTS will need to publish a comparison of all the different CQL implementations and their semantics. After that, we can discuss”standardization”…  one gets the impression that Oracle would prefer to claim that their implementation *is* the de facto standard, but unlike their SQL implementation they likely don’t have anywhere near the installed base for CQL to justify such a claim.

[2] Note that a continuous query is not the same as CEP. A continuous query running against a high-latency database, for example, may just buy you a more efficient trigger representation.

[3] Presumably, not literally…

[4] I saw these cartoons in a blog recently and thought their similarity to CQL / database issues to good to pass up!

Continuing our posts from the Distributed Event Based Systems conference: the next tutorial was Opher Etzion from IBM Labs on CEP Patterns [*1]. Or more accurately, a notation for event processing components, which seemed more generic than the Snoop notation from the previous session, and possibly a good fit for matching to a process framework like OMG BPDM [*2] [*3].

Opher, I noticed, used the term “Event Processing Agent” to describe a logical event processing task, perhaps made up of several EP operations. Hmmm, I’ve seen the term “Agent” in CEP context before…  such as last week when Jim Odell presented at the OMG meeting on the OMG Agent Metamodel and Profile for agent-based programming (listing CEP amongst its uses). We at TIBCO talk about (large granularity) Event Processing Agents (which collaborate as multiple distributed processes, possibly using differing techniques such as rules, rules + states, queries, etc), but otherwise our agents have similarities to Jim’s definition. Oracle, in the following tutorial, seemed to share the same usage as TIBCO. So perhaps the taxonomy / terminology should look something like:

• Event Processing (EP) Operator is-used-in EP Element
  where Element is a logical CEP task consisting of 1 or more CEP operations
  for example, Operators might be  a query part, a rule condition, a SNOOP operator
  for example, Elements might be a rule, ruleset, query, set of queries
  • EP Element is-used-in EP Agent
    where Agent is an autonomous or semi-autonomous process containing EP Elements
    for example, an Agent might be a set of EP Elements together with event interfaces (and any necessary controls) to define a processing unit
    • (and maybe EP Agent is-executed-in EP Engine)
      where Engine is an executable deployment container for EP Agents
      for example, the appropriate CEP executable used to execute 1 (or more) EP Agents
      and where a System may include a set of distributed Engines and their associated Event infrastructure

Although “EP Agents” were defined by Opher, I don’t recall “distributed EP agents” being discussed (although “distributed EP” is somewhat orthogonal to “event processing element”, with the caveat that latency modeling might be needed for highly distributed CEP systems). Distributed CEP requirements are already a subject of discussion (for example, see rule engine advocate James Owen’s recent comment on the need for distributed processing for Homeland Security problems). Such requirements are also, of course, already being addressed, such as by some high-scalability rule-driven CEP vendors…

Notes

[1] Opher had to use someone else’s laptop, with the effect that his PPT displayed logical operators using some very pretty Microsoft symbol font symbols. The effect made his notation look much more interesting though!

[2] … although it may well be the case that BPDM will need “tuning” to accomodate continuous event processing and “non-traditional BPM” concepts. In addition, this may or may not be an ultimately useful exercise (except for those who need holistic views of their enterprise process models).

[3] <Rant> I was searching for a suitable link for OMG BPDM that would make sense to the CEP community, but didn’t fiind any that expressed my view (/hope) of what BPDM represents, or suffered inaccuracies like equivalence to XPDL, that SBVR business notation equates to process rules, or that BPDM is used (implication: directly) by business users. Of course, existing BPDM descriptions are focused on selling the idea to the BPM community (although the RFP implies a wider audience including the EP community). </Rant>

Occasionally it is good to attend an academic [*1] conference in one’s field, and the Distributed Event Based Systems conference is considered one of the main events for CEP (although it includes a major dose of pub-sub / middleware topics too). Held in Rome, Italy, the conference organizers had a few challenges (such as flakey WiFi, and a lack of air con in the first venue that made for the novel experience of seemingly having conference sessions in a sauna) but the fascinating location (near the Colloseum) made up for any minor inconveniences.

Sharma Chakravarthy from the University of Texas kicked the tutorials day off by covering the Snoop representation for event operations. This seemed a neat way of modeling (some) aspects of event processing, including the fact that a composite event can represent multiple events which define a duration, making temporal operations tuple-based (i.e. with a start and end timestamp). This leads to a specific event-notation (with maybe some similarities to ORM), and “interval-based semantics” for composite event operators (which look like they might be overly complex for typical use cases and CEP users). Snoop seems to have been around for years (see this 1993 reference!). I have to say that the references to Active Databases made for a good history lesson, but also made the tutorial seem outdated (e.g. as an exercise, consider how you might relate the terms “active database” and “distributed event”).

The Snoop info was interesting (for anyone who hasn’t seen it before), but I was hoping for more on patterns and uses for combining ESP with CEP, per the tutorial title (”Events and streams, harnessing and unleashing their synergy”). Indeed the session started with the basic concept of processing streams before combining their resulting high level events - certainly a CEP design pattern which is of particular interest to TIBCO BusinessEvents users.

Cruel relief was provided during Sharma’s “EP operations are not database operations” comments [*2], which for some reason seemed to annoy the heck out of the Oracle guys…

Notes

[1] DEBS included a number of industry presentations, but many of these represented “R&D” updates from vendors. For a more commercial conference, but with less technical detail, consider the Gartner event later in the year. Many CEP and EP vendors were present (NetZero, Streambase, Oracle-the-database-storing-events, Oracle-the-Weblogic-app-server-version-for-events, WareLite), as well as vendors from other technologies such as BRMS/BRE (ILOG), ERP (SAP), RDF-store (Franz), etc.

PS: googling for DEBS2008 or DEBS08 will likely hit a number of, er, unrelated and possibly work-unfriendly sites. Use the search term DEBS 2008 CEP for a better success rate.

[2] Sharma had an excellent comparison slide on “DBMS” vs “data stream management systems”. This went through their respective attributes, which looked approximately like:

1. persistent relations vs transient streams
2. ad hoc vs continuous queries
3. random vs sequential access
4. unbounded disk store vs bounded memory store
5. only current state vs history / arrival order critical
6. low update rate vs high data rate
7. no real-time services vs “real-time” requirements
8. precise results vs approx results
9. transaction mgmt vs no transaction mgmt (e.g. rollback)

To show how current CEP tools might fare in this comparison, lets see which attribute best applies to TIBCO BusinessEvents: 1. both, 2. both, 3. both, 4. both, 5. both, 6. high data rate, 7. “real-time”, 8. precise results, 9. no “transaction management”. Not bad (and you could argue that transactions are just managed “differently” in an EDA).

Although pundits might debate whether EDA (and thence CEP) is a part of the SOA paradigm (or not), there is no doubt that CEP can be used to make SOA services more useful [*1]. One such area is in managing SOA performance and SLAs (the SOA equivalent of Business Activity Monitoring or BAM), as exemplified by products like TIBCO Service Performance Manager. I was reminded to comment on SPM following an interesting blog by Todd Biske on the merits of taking a lifecycle view of SOA service development (not just “slam dunk” deploy-and-forget) on the TUCON SPM podcast [*2]. TIBCO SPM is, of course, a CEP application built on top of TIBCO BusinessEvents.

Notes

[1] Interestingly, one of the keynotes at TUCON (by Margam Sundararajan from Citigroup) explained how a CEP application was used to help justify the SOA strategy. One of their applications, using TIBCO BusinessEvents, was deployed first in order to get a quick business ROI.

[2] The AllState application discussed in this podcast by Anthony Abbattista is also a TIBCO BusinessEvents CEP application.

From OMG last week, it looked like the BPDM versus BPMN debate was “beginning to end” (although pundits are finding plenty to comment about: for example see EDS’ Fred Cummins’ comments, and Bruce Silver’s response) [*1]. The latest news is that the debate seemed to be mostly settled in favor of a pragmatic solution (which will presumably be announced at their next meeting in Sept 08).

From a CEP perspective, BPMN represents “simple-event flows”, and could in theory be used as the starting point for a set of standard semantics for continuous event processing (and, by extension, complex event processing or CEP). But simple “process flows” are to business processes like “decision trees” are to business rules - a very useful representation, but not in any way the only, nor necessarily the best, way to represent every process. There is a good reason why BPMN is commonly associated with “workflow” (modelling human-oriented business processes) [*2]. There is also a good reason why BPDM is potentially a very useful cross-process (including CEP) metamodel (and ergo why it should not be directly tied / restricted to just BPMN) [*3]. Conrad Bock’s (NIST) BPDM tutorial at the OMG meeting was well attended, and indicated that BPDM, as intended, shows scope to be a generic process metamodel that could be used to relate different process modeling styles, including continuous and complex event processing. Definitely something for EPTS / OMG / CEP researchers [*4] to look into in future.

Meanwhile, in the real world, we are finding (1) an event-driven approach to modeling enterprise (inter- and intra-departmental as well as B2B etc) processes and (2) the use of executable event-driven models to drive workflows (as commented earlier) are being recognised as very useful capabilities. At TIBCO, for example, the use of standard modeling constructs (concepts / classes, state models, production / inference rules [*4] and queries) and best practices (event-awareness, decision management) with a high-performance distributed execution engine allows for dynamic process definitions that can effectively drive BPMN/BPM (iProcess etc) workflows.

As a heads-up, the relationship between CEP and BPM is a round table topic at the BPMI Think Tank later this year [*4].

Notes

[1] For the record, TIBCO’s BPM team already supports the defacto BPMN persistence mechanism (WfMC’s XPDL), and is a member of the BPMN2 submission team that includes IBM, SAP, Oracle, etc.

[2] Consider a large decision tree where you need to insert a new rule that affects more than 1 decision path. Oops, you now have a maintenance problem (did you update all necessary decision paths? when this rule changes, will you find all the necessary paths? etc). BPMN and orchestration diagrams / activity diagrams all suffer from the same representation scalability problem, which is why declarative forms of behavior are often needed (e.g. alongside or to augment orchestrations).

[3] BPDM was designed, originally, to handle all types of processes, not just BPMN orchestrations, and including event-driven ones. It may yet be considered by EPTS for potential CEP metamodel standardization …

[4] It seems that BPDM might be compatible with existing and proposed CEP and CEP-relevant standards (such as PRR), and even Opher’s proposal for an Event Processing (EP) meta-language (research topic / future standard). BPDM provides a general structure for organizing components of processes, including an emphasis on events, but it remains to be seen if it can handle (or be readily extended to handle) declarative process models, continuous process models, etc.

[5] Disclaimer: TIBCO are chairing said round table.