Over the next few weeks James Odell, known for his work on OO, UML and agents, will be offering some thoughts on agent technologies on this CEP Blog. Over to Jim…

The notion of event processing agents (EPAs) is quickly becoming a topic that more and more organizations are considering. We’ve been building systems that employ event processing without agents for quite some time. So, why agents and why now?

One of the primary reasons is that we are no longer in the era of mainframe computing, when both companies and applications were typically command-and-control oriented and organized in vertical silos. With the combination of the Internet, fiber optics, and PCs, the business and technology playing field has been flattened. No longer primarily top down, it has changed to more side by side-as individuals, small groups, and organizations interact around the world. In other words, the events around us are based on complex web of activity that is less command and control and more horizontal connecting, collaborating, and competing.

When the events among requesters and providers are few and change is not an issue, agents are not needed and conventional IT techniques can be employed. However, when the links require complex, dynamic binding and are subject to rapid change, agent-based approaches should be considered. Prior to the demand for global collaboration, we developed a centralized controller to ensure that all interactions would be appropriately managed. Now, this is no longer possible. The world is too big and a central bottleneck would paralyze the effort. As a result, the access techniques now require a more horizontal style of interaction-rather than one that is centrally administered.

Agent technology is a primary enabler to support this new direction. In fact, without agent technology, our current technology will not scale to support the ever-increasing global interaction. More importantly, it will enable us to create and support a whole class of IT applications and approaches that we previously could not have developed-this especially includes complex event processing.

In the next installment, I will address the question: “What is the relationship between agents and events?”

Nice to see a paper on IBM Developworks on some of the theory of “event processing”, authored (amongst others) by the Chair of the Event Processing Technical Society, IBM Labs’ Opher Etzion, and fellow EPTS Reference Architecture team member Catherine Moxey. The paper is of course somewhat of a “broad brush” view of event processing, as indeed you might expect from a team representing such a diverse range of event-handling software as Tivoli, CICS, MQ and so forth from the broad (and, from a TIBCO perspective at least, heavyweight ) IBM software stack.

Some thoughts:

1. The paper implies a need to model, in any significantly large event processing system, the types and roles of the event processing agents and how they interact. Some such agents will be tightly coupled (such as co-operative rulesets in a TIBCO BusinessEvents inference agent), whereas others will have more identifiable interfaces and roles (such as separate inference and query agents in a distributed TIBCO BusinessEvents application). Tables 6 and 7 for example are almost metamodel definitions for an event processing agent.
2. Possibly a better name for the paper would be “A Conceptual Model for Event Processing Networks“, given the emphasis on event pathways through processing agents.
3. IBM Developerworks (like this blog) has an unattributable “scoring mechanism”, and shows today that one reader was unimpressed enough to profer a low score - but why? I also can’t explain why this is described as aimed at an “intermediate” audience, as the principles seem basic and well explained…

Thanks to Dr Mark Darbyshire from TIBCO’s Vertical Solutions group for the link to an interesting paper on a “Policy Appliance Reference Model” by K A Taipale, otherwise referred to by the (obvious abbreviation) term “PARM”.

The paper describes PARM as: …an enterprise architecture for information sharing and knowledge management … based on policy appliances (technical control mechanisms to enforce rules) interacting with smart data … and intelligent agents… to reconcile, enforce, and monitor agreed information management policies for information security, data quality, and privacy protection across heterogeneous information sharing systems and networks.

As seems usual, the last sentance is a summary of the description and reads best: [PARM] supports policy-based information management processes through rules-based processing, selective disclosure, and accountability and oversight.

Although PARM is not directed at event processing, it should also be relevant (if not more so) to “data in motion”. Indeed it has many similarities to a CEP-based solution available on request from Dr Darbyshire’s team at  TIBCO that exploits TIBCO BusinessEvents to provide a service gateway with embedded policy management, currently deployed for high-performance telco problems but certainly suitable for handling SOA governance in SmartGrid, government, finance and transport type domains too.

Work on the EPTS Reference Architecture is moving forward, with excellent contributions from the event processing industry and academia (EPTS members can observe the work-in-progress on the EPTS working group wiki). One interesting angle of discussion revolves around the compatibility and overlap of “event processing” with conventional “data processing” technologies, especially when it is possible to view “events” as “data” and process them as such. Of course, critics of CEP will counter that all event processing tools simply re-use existing data processing techniques (such as extended versions of SQL engines, or inference engines, or neural nets etc…), and to some extent they are right - what’s new is the idea of re-arranging these techniques to exploit higher-performance CPUs and larger amounts of available memory for in-situ event pattern detection in everyday business problems.

Defining a generalized event processing (and complex event processing) architecture that covers all use cases, even of a single (albeit ambidextrous) tool like TIBCO BusinessEvents, is rather challenging. There are a number of different architecture patterns possible, covering different functional and performance requirements. One generalized effort is included here, based on TIBCO’s work for customers in this area:

The various tiers can be described as:

Tier 0 (not displayed):  The sources of events could be anything from SCADA control systems, RFID detectors, BPM or ERP systems, or whatever.

Tier 1: Event channel (or transmission medium). In TIBCO’s world this is usually EMS (topic-based or queue-based) or RV message buses, although it could also easily be some other middleware mechanism.

Tier 2: Event PreProcessor. Optional. Handles basic event filtering, and roles such as concept enrichment (when the event is just adding information to an existing event object), concept creation (the event is transformed into an event object, or concept in BusinessEvents), and forwarding to local and/or distributed memory (a.k.a. event store, for processing elsewhere).

Tier 3: Distribution Agent(s) or Event Store. Optional - used for high-performance, high volume, fault-tolerence and large-scale CEP. Provides storage for events (and event objects) required for the event patterns representing complex (aggregate, abstract, derived) events - where these event patterns may occur over periods of time. Needed (in conjunction with Tier 4…n) where the volume of events, or amount of event processing, exceeds the limits of a single process node’s memory or CPU limits. Used also when information needs to be replicated across nodes to support fault tolerance and failovers. For performance reasons this is often some kind of high-performance cache or data grid (as is the case of BusinessEvents’ cache component), but it could also be a common-or-garden file-oriented database… in TIBCO BusinessEvents we support both approaches.

Tier 4…n: Event Processing Agent(s). In the simplest case, a single EP Agent can attach to a channel and process events (2-tier). Or it can involve a Preprocessor either in-process or as a separate agent (2.5-tier). Or be used with the cache (4-tier). Ultimately these can be organised as a hierarchy of EP services, with these services themselves constituting multiple, load-balanced, identical agents. These EP Agents define event patterns (through rule conditions, state transitions, and queries) and communicate either directly to event channels [Tier 1] or, more often, through the event store [Tier 3]. Large workloads can be distributed through queries (i.e. pull model) or through shared event / event object updates (i.e. push model). In BusinessEvents, EP Agents can be one of 2 base types: inference agents (for rules and states), and query agents.

Tier n+1: Services. Optional. Existing SOA services may also contribute data to the event processing agents, either as-required or cached in the (event) store to be used and updated as needed. Typically service invocation (which could also be a BPM workflow invocation)  involves a call out to some mechanism like TIBCO BusinessWorks, and possible some TIBCO Adapter.

Tier n+2: Persistence.  Optional. When the event store [Tier 3] is using high-performance cache there is a need to provide back-up persistence for tasks like archiving, start-up load and shut-down un-load, and standard analytics. You might also have operational data stores you need to query directly (rather than through the services layer [Tier n+1]) to enrich event objects for processing. If you want to visualise all your events over time to identify new patterns you would typically use something like TIBCO Spotfire against this layer, and perhaps S+ Miner to extract some rules to update some Tier 4…n agent with…

Jim Sinur at Gartner has posted his views on CEP on his blog. Jim comments on “event discovery” (a.k.a. detection?) via rules, patterns (maybe meaning streaming or continuous queries), process (maybe meaning process exceptions or manual detections), algorithms, and “context”. For the latter he proposes a connection between CEP and “Complexity Theory” which according to Wikipedia is either Chaos Theory or maybe Complex Adaptive Systems (”complex, self-similar collection of interacting adaptive agents … focuses on complex, emergent and macroscopic properties of the system”). That one is for the lab, methinks.