Enterprise Architecture covers, by definition, enterprise-wide modeling of both business and IT systems, and therefore should accomodate Complex Event Processing. One interesting blog shows *one* way of doing this - modeling events, event patterns and situations to “trigger processes” which can be represented as “IT services”. The conundrum here is that the “trigger” (for example, implemented in a CEP engine), is in itself a kind of IT service…

Note that one web site’s definition of Complex Event Processing claims an even closer relationship between CEP and EA, declaring CEP to be “a strategic mechanism used in designing an enterprise architecture that is based on analyzing multiple events which can have varied impact on the technology implementation of the business processes in a business enterprise and deriving an appropriate action optimized for efficiency.” To which I can only add: beware web editors that strip out punctuation!

[A friendly edit of this definition might read: CEP is "a strategic mechanism that may be used in designing an enterprise architecture and that is based on analyzing multiple events, which can have varied impact on the enterprise by deriving an the appropriate actions optimized for efficiency. CEP technology may be used for implementations of the business processes in an enterprise."

But even this edit certainly still needs work...]

With the recent news that one of the 2 main pure-play BRMS vendors has got itself acquired, it might be a good idea to catch up [*1] on the differences between Business Rule Engine (BRE) and Business Rule Management System (BRMS) software components with their equivalents used in Complex Event Processing:

Business rules vs Decisions

If you compare, for example, the Business Rules Group definition of “business rule” [*2] with the types of rules we automate in CEP or BRMS tools, you find that the former mostly equates to constraints on the business, and the latter are types of behavior (as in: IT process implement behaviors, or decisions). BRMS tools manage decision-type rules that are in turn derived from “business rules for business people” (or “source rules”). The different philosophies are apparent in the 2 OMG standards, SBVR (covering business constraints) and PRR (for system behavior) [*3].

A Complex Event Processing system usually defines business rules as declarative decisions (in, for example, a rule-driven CEP system), or maps the rules to flows or procedural language constructs. Sometimes a CEP system just defines facts (such as a continuous query  indicating some interesting complex event), or presents the complex events (such as in a dashboard) to support some manual decision-making process.

Business rules relate to Events

Quoting from Ron Ross’ book [*4]: “Processes connect to rules via events … which can fire one or more rules”.

Quoting from Dave McComb’s book [*5]: “A more likely successor [to explicitly called rule engines] will be a business rules engine that operates on message traffic. ”

So these experts (in the related fields of business rules and semantics) seem to agree on the close connection between business rules and events: a corollory is that moving business rule definitions (and management thereof) closer to the source event would be “A Good Thing”. Until recently there has been no mention from the BRMS vendors of the Complex Event Processing space; the only BRMS vendor comment so far seems to provide a somewhat defensive response by implying that BRMS are for rich rulesets and CEP is for fast events. In the enterprise world, of course, you often need both rich rulesets and fast event handling. “Rule management” is as much required for the low-latency rules as for the high-latency workflow / BPM rules (as usually served by the BRMS vendors). Combining the 2 is clearly going to be an interesting evolution, already started with the TIBCO BusinessEvents rule-driven CEP product.

BRMS standards?

One interesting caveat to the propagation of rule management to Complex Event Processing, is the lack of appropriate standards at the rule management level. You can use SBVR to specify business constraints, and derive the likely events from the term and fact model (such as new car rental agreement, new customer, rental car return, etc). But you also need to map them to appropriate processes. For the most part, customers are delighted enough with the agility they get from rule management without demanding cross-vendor standards. Yet. But standards are often (as commented at a recent event) a prerequisite for federal etc adoption.

BRE Performance

… is, like in the Complex Event Processing field, a somewhat “hot topic” among certain Business Rules Approach fans. Although the vendors in this space tend to be quite relaxed on this - performance is almost always “sufficient”. And this is true, as far as rule engine performance goes. The main problem for the BRE vendors is that they get blamed for transaction performance, when the fault really lies in the data marshalling needs and the app server: there is not much point in shaving 10% off your rule transaction time of 40ms, if you are reliant on a database transaction which costs 400ms. This is where CEP-based rule execution can be surprisingly effective over traditional BRE approaches, as CEP systems generally avoid database transaction costs.

Notes:

[1] See earlier posts on this topic: Differences between a BRE and a rule-driven CEP engine (part 1) and (part 2).

[2] “A business rule is guidance that there is an obligation concerning conduct, action, practice, or procedure within a particular activity or sphere.”

[3] Both SBVR and PRR rules are “representations” of business logic - so both can claim to define “business rules”, although naturally SBVR notations are meant to be more “business friendly”. Usually PRR is used to explicitly define “decisions” including event-based decisions and rules.

[4] See Business Rules Concepts, 2nd Edition (thats the one with the green cover - google books only has the old version), Ch2, pp34. Note that in Ron’s view, you apply events to business (constraint) rules which in turn can generate events to indicate violations. Such as the rule “each customer has 1 and only 1 account manager”, which is applied to events such as newCustomer, deletedAccountManager, setCustomerAccountManager, and can generate appropriate exceptions when the constraint is not satisified.

[5] See Semantics in Business Systems, AppA, pp297.

I was surprised to see that, in the SOA world, there is some debate as to the usefulness of the Enterprise Service Bus. One reason for this might be the evolution of the term “ESB”, from enhanced middleware to service container framework (a.k.a. Enterprise SOA Bus?). Or possibly the anti-ESB lobby take a narrow definition of SOA (e.g. hierarchy of orchestrated synchronous services), rather than those who include Complex Event Processing within the definition of SOA (e.g. any combination of event-driven, synchronous or asynchronous, independently managed or developed contract-defined services).  In the latter case, which includes Event Driven Architectures, ESBs / message buses are a very common / de facto [*1] means of communicating events to, and between, Event Processing systems - indeed “ESB” could be an abbreviation of “Event Service Bus” for the CEP community.

One of the complaints about ESBs seem to be their proliferation and the need to manage appropriate gateways. I would have thought that System Architects would be happy to partition their message buses and domains, just as network specialists manage physical networks with bridges and routers based on configuration needs. Probably the main problem with ESBs is that they don’t fit into some architect’s pure WS-oriented view of the world - but I would suggest this is the architect’s “problem”, not the ESB’s.

Notes:

[1] In TIBCO BusinessEvents, for example, we include default channel adapters for TIBCO RV and TIBCO EMS (which is JMS).

David Luckham has posted a nice summary of the concepts (and use cases) for complex events and complex event processing [*1]. This complements his short CEP history series (parts 1 and 2) [*2].

Notes:

[1] Now, if only someone would use this as a basis to rewrite the Wikipedia entry on CEP...

[2] David mentions Discrete Event Simulation in Part 1 as an example forerunner of CEP. I recall as an example of this some “intelligent rule-based processing” inside some large Monte Carlo simulations in the ’80s, simulating processing of simulated events…

An interesting post on Tim Bass’ CEP blog [*1] describing Blackboard Systems, which is an established term from the era of AI research for “distributed knowledge systems” that co-operatively solve problems. Tim and I have previously mentioned blackboards and blackboard systems in the context of Complex Event Processing (CEP), but the passage of time has meant that “blackboard” is more significant for implying “distributed shared memory” [*2] in a CEP context, rather than just co-operating threads or agents looking at a shared database or memory structure [*3]. Distributed memory is a requirement we see for scalable, high-throughput event processing beyond what you can fit into a single machine’s (or JVM’s) memory space.

A general progression for “CEP system complexity” on how the system handles memory is:

• in-memory only, with persistence for reliability / restore operations
  = small, fast, independent CEP or Event Stream Processing (ESP) applications
• single-machine, multi-process (for example using multiple cores), sharing the same memory
  = small-medium, pretty fast, with a restricted number of co-operating processes
• multi-machine network of processes (exploiting control as well as data events across the network):
  • independent memory models
    = where the problem area can be partitioned without side effects: multiple parallel identical processes (for performance)
  • shared-memory models (usually using some cache technology)
    = where the problem area is large and inter-dependent, requiring inter-dependent or co-operating processes (for solution complexity) (as well as allowing for parallelism for performance).

CEP frameworks can generally support all these models (out of the box as for TIBCO BusinessEvents, or with various amounts of custom development work). Of course, the last model (multi-machine network with shared memory) is the interesting one for “Blackboard System” types of architectures (i.e. cooperative CEP agents working against a shared information model and event store, possibly under the control of a Master Control Program / Agent).

Other useful references are:

• IITB course material on Blackboard Systems
• Naval Research  sponsored paper on Blackboard Systems

One suspects the “blackboard systems” domain and terminology is overdue some updates thanks to developments in the Complex Event Processing space.

Notes:

[1] Disclaimer: Tim is an ex-colleague and runs a vendor-independent blog on aspects of CEP.

[2] Blackboard systems historically used a single memory model (i.e. multiple threads or processes using a single machine’s memory model). But the interesting aspect for CEP is not that event processing agents can create new events to be used by other CEP agents (which is pretty much de facto CEP runtime behavior), but that the memory model can exist across multiple machines (i.e. can be distributed).

[3] This old paper even suggested that blackboard systems’ reign in AI research was curtailed by rule systems’ use of independent rulesets operating on a shared working memory - i.e. standard rule engine behavior. Rule-driven CEP engines like TIBCO BusinessEvents can certainly operate this way, with “independent” declarative rulesets cooperating on a problem. This approach is more difficult if you can represent your CEP or ESP solution only as a “flow diagram”, as you are explicitly fixing (non-declaratively) the interoperation of the CEP processing elements.