… was the subject of today’s Forrester-IBM webinar on distributed cache technology, with both Forrester and IBM citing CEP and EDA as users for this technology, amongst others. The overriding driver for this tech being eXtreme Transaction Processing, which we might just refactor as eXtreme Event Processing for the purposes of this blog!

One minor quibble: John Rymer of Forrester did the introduction and during so classified the cache market as .NET, Java and NoSQL camps, with TIBCO placed in the Java camp. This might seem a fair classification of a complex market area, but of the 2 relevant TIBCO distributed cache offerings:

• TIBCO BusinessEvents, although Java-based, is more accurately described as a CEP product that embeds a distributed cache - it wouldn’t normally appear on a vendor list of distributed cache technologies;
• TIBCO ActiveSpaces is more accurately described as a data grid, but has .NET, C and Java interfaces. I’m sure other caching / data grid products have similar multiple interfaces - after all the client is just “an interface” to the cache /  grid.

Spare a thought in passing, though, for the OODBMS guys. Amongst this buzz about data grids and caching, I notice the Forrester blog is reporting that the Progress guys (disclosure: a TIBCO competitor in some areas) are now considering their ObjectStore OODBMS a “legacy platform”. Plus ça change, perhaps.

Thanks to Dr Mark Darbyshire from TIBCO’s Vertical Solutions group for the link to an interesting paper on a “Policy Appliance Reference Model” by K A Taipale, otherwise referred to by the (obvious abbreviation) term “PARM”.

The paper describes PARM as: …an enterprise architecture for information sharing and knowledge management … based on policy appliances (technical control mechanisms to enforce rules) interacting with smart data … and intelligent agents… to reconcile, enforce, and monitor agreed information management policies for information security, data quality, and privacy protection across heterogeneous information sharing systems and networks.

As seems usual, the last sentance is a summary of the description and reads best: [PARM] supports policy-based information management processes through rules-based processing, selective disclosure, and accountability and oversight.

Although PARM is not directed at event processing, it should also be relevant (if not more so) to “data in motion”. Indeed it has many similarities to a CEP-based solution available on request from Dr Darbyshire’s team at  TIBCO that exploits TIBCO BusinessEvents to provide a service gateway with embedded policy management, currently deployed for high-performance telco problems but certainly suitable for handling SOA governance in SmartGrid, government, finance and transport type domains too.

One of the interesting requirements for XTP/XEP (as mentioned in the previous blog posting) is that such performance (and the need for fault tolerance / failover) requires co-operation across multiple systems, ideally generalised in a multi-agent architecture [*1]. The problem here is usually in re-factoring to change performance characteristics as the project develops, to optimize the architecture to the performance needs, available compute nodes, and network (as well as changes to event throughputs).

One solution is to take the agent-based approach, where agents can be re-configured (through model parameters) very simply. Agents can:

• execute declarative production rules “as required”, removing most of the need to specify / re-engineer (/ debug etc) “entry” and “exit” points;
• use queries to partition data (/event objects) and process loads to separate agents;
• rely on shared distributed cache to provide a “co-operative model” of events and data for processing.

The re-factoring / re-architecting process is, to some extent, as simple as specifying which rulesets are assigned to which agents, and defining more or fewer agents as required:
a. Revise named agents (aka agent types and roles)
b. Re-allocate rulesets (or queries) to appropriate agents
c.  Deploy (or re-deploy, with appropriate care) in numbers required.

So the agent modeling aspect is to map the event processing elements first to agent base types (e.g. inference or query [*2]) and subtype (e.g. event preprocessor, event router, event/data processor, batched event/data processor,  etc)  and verify the event flow / data flow is complete and without contention. In this way you can quickly re-architect a TIBCO BusinessEvents application from single-node to multi-node, high performance to extreme performance…

Notes:

[1] Agent representations and modelling is currently the subject of an OMG standard, AMP, currently under development.

[2] CEP agents in BusinessEvents come in 2 forms: inference agents and query agents, roughly equating to CEP agents and Event Stream Processing agents…

Defining what is “eXtreme” is a difficult topic. For example, if I am simply “processing” simple events as incoming data with no correlation requirements, I have a significantly easier job than if I am doing complex event processing, abstracting higher level events over time from incoming events.

Here is an interesting example that was being discussed last week: during an on-site application test, an enterprise event processing application (using TIBCO BusinessEvents) was comfortably processing events at their arrival rate of ~750 eps (events per second) when a “testing glitch” caused the EMS (JMS) queue to back up; when the offending component (i.e. agent) was corrected and restarted (in situ) the CEP application maxed out its CPUs to clear the queue at a reported ~5K eps. This 5K figure compares quite well “as a number” with other complex applications (as opposed, maybe, to event stream processing of simple events), although probably it will be up to someone like the EPTS Use Case Working Group to make some classifications here.

My guess is that “eXtreme Event Processing” or “XEP” (in which we really mean “extreme CEP” involving correlation across events, XML processing, some number of business rules, large number of fields per base event message, 24+ hour retention, fault tolerance, and so forth) probably starts at ~1000 events per second. But ultimately its not just a matter of “how fast”, but also “work done”…

I see from Tim Bass’ blog that Brian Husted of Pyxis has published a paper titled “Event Driven Grid Computing“. This is regarding a US DOD SOA application that exploited “CEP to enable scalability for SOA“, handling tasks like intelligent routing, governance, monitoring and complex workflows.

Brian comes across as pretty enthusiastic about the use of the TIBCO BusinessEvents event processing platform, for example commenting “… a small number of people (2 or 3) can turn out big things with BE”. Hopefully he’ll find time in the near future to write about some of his other BusinessEvents applications.

Notes: See also this past post on the topic of CEP, EDA and SOA.