Complex Event Processing: a technology evaluation check-list
Posted by Paul Vincent
One of the challenges for organisations investigating event processing technologies is “how do we know what is important”? Obviously everyone will have different views on this. Nonetheless, as TIBCO has the most experience in CEP across multiple domains and the largest customer base for a CEP technology tool, it seems to make sense to present a sample evaluation guide that shows the typical considerations and features needed.
First, a few comments. TIBCO deals with enterprises who can have sophisticated requirements around their complex event processing needs. TIBCO customers generally use CEP for operational intelligence applications covering anything from business activity and event monitoring to automated intelligent business processes. Hence TIBCO solutions handle high throughput (not just latency) and payload size (business messages can get very large), whilst providing the fault tolerance and resilience capabilities that 24×7 organizations require. But every class of CEP application has different requirements: for example an algorithmic trading solution might prioritise ultra-low-latency over other features like failover-support and logic and payload complexity. Ergo, for every use case the “example scoring factor” provided below will need to be adjusted to match the feature’s importance to the use case.
| Feature | Detail | Reason | Example Scoring Factor |
|---|---|---|---|
| 1. Event Channels | A. Specific channel types required | Example: JMS, MQ-Series, HTTP… | 10 |
| B. Custom channels | Example: custom event type for SCADA | 3 | |
| 2. Modelling | A. Event models | Event metadata, hierarchy, inheritance | 10 |
| B. Event object / concept models | Object history, hierarchy, inheritance | 10 | |
| C. State and status of complex events | State model and flow | 8 | |
| 3. Processing Elements and Expressions | A. EventConditionAction rules | Basic event filters, aggregations | 10 |
| B. Inference rules | Intelligent event processing | 6 | |
| C. Continuous queries | Event streaming aggregations, statistics | 6 | |
| D. Temporal regular expressions / logic | Concise pattern recognition | 6 | |
| E. Specialist analytics and statistical functions | Capabilities for pattern discovery, possibly in conjunction with developer interface (see 7) | 6 | |
| F. Other context support | How to handle context across time, such as GVs etc | 6 | |
| G. Extensibility | How to add custom functions / algorithms / analytics? | 6 | |
| 4. Event augmentation | A. Database access | JDBC read/update etc | 4 |
| B. SOA access | Invocation of existing services | 6 | |
| C. BPM / workflow access | Invocation of manual steps | 3 | |
| 5. Stateful processing and management | A. Event storage in-process | In-memory support | 8 |
| B. Event storage out-of-process | Low latency cache and/or high latency backing store | 8 | |
| 6. Platform capabilities | A. Scalability via distributed processing agents | For large processing loads (logic and/or event size and/or rate) | 8 |
| B. Scalability via distributed event cache | For supporting event patterns over processes, resilience | 8 | |
| C. Failover / resilience features | Hot standby and event handling if agent fails | 8 | |
| D. Hot deploy / event-based updates | Hot deploy of new rules, queries or patterns, or dynamic update through events | 8 | |
| E. Operations and Management | UI and/or reports to monitor performance and/or adapt capabilities | 8 | |
| F. Performance supporting features | Components to minimise latency like preprocessor, algorithms etc | 10 | |
| G. Deployment environment support | Desired or preferred hardware/OS/cloud platforms | 8 | |
| 7. Development Environment | A. Standardised GUI tool | Example: Eclipse-based | 8 |
| B. Verification tools | Logic verification such as diagram generation and reports | 6 | |
| C. Validation tools | Unit and bulk test capabilities | 6 | |
| D. Debugging | Local / remote debugging capabilities | 6 | |
| E. Team features | SCCS / project repository / team development | 4 | |
| 8. Business interface | A. Business Event / Rule / Decision Management | Management (editors, impact analysis) for business-controlled event processing aspects | 6 |
| B. User Workflow / control | Control and security for business updates to event processing | 6 | |
| C. Event monitoring dashboard | Display of events, and complex events such as KPIs | 6 | |
| 9. Project considerations | A. Track record | Numbers of Customers / users in same domain | 6 |
| B. Availability of SIs and development partners | Available assistance if needed | 2 |
One final checklist or scorecard item cannot be added to the list above, but is at least as important as the technology factors. This is the client-vendor relationship, and I was reminded of this when another CEP vendor tweeted recently about how they had been gaining business partly due to another vendor’s “aggressiveness” in business dealings. So, just like everywhere, caveat emptor…. and maybe add a higher score to 9.A. above! 
As usual, comments, feedback and suggestions welcome.
Tags: evaluation checklist
Complex Event Processing (CEP) | 04 March 2010 |
9 Comments
Other Links to this Post
RSS feed for comments on this post. TrackBack URI
By Opher Etzion, March 5, 2010 @ 07:23
Hi Paul. Good work in collecting all criteria, my main issue with scorecards that they are based on the assumption that there is a universal scoring, as some of this criteria may or may not apply to certain application, also it is sometimes the case that an event processing system need hybrid of capabilities, and it is difficult to define exact scoring. I believe that the way forward will be to apply a collection of building block that can easily work together so each customer can pick what it needs, and also extend without problem, as applications evolve.
cheers,
Opher
By James Taylor, March 5, 2010 @ 17:03
Paul
A nice list but I think you are missing a key element in the business interface - impact analysis. Business users, in my experience, don’t want to test or verify they want to see the impact of a potential change in business terms.
JT
By Hans, March 6, 2010 @ 06:38
Would you not say that impact analysis is highly dependent on the use case? Or is there a general kind of impact analysis that’s not coming to my mind right now?
By Paul Vincent, March 7, 2010 @ 14:29
Hi Opher - thanks for the advice - yes, I will simplify this to just GVs (Global Variables) “etc” as there are many ways of doing a global context…
Cheers
By Paul Vincent, March 7, 2010 @ 14:52
Hi Hans, James: I would claim that “impact analysis” in Event Processing could be viewed as 2 different activities:
- monitoring and prediction (what will happen downstream of incoming events)
and
- “what if” analysis of such monitoring and prediction (normally covered by “simulation” in CEP).
The first (monitoring and prediction per se) is usually a pretty common CEP role. We are trying to predict what will happen based on event patterns occuring. Examples would be predicting electricity grid transformer failures, or manufacturing process quality test requirements (both TIBCO BusinessEvents use cases). I think the industry is at an early stage of providing “business tools” to handle the complexity of the development and specification of the temporal pattern logic required for CEP…
The second, simulation, is “relatively easy” in simple use cases (like the test facilities in TIBCO BusinessEvents Decision Manager, and in many other BRMS tools), but likewise more difficult to achieve effectively for event sequences involving time and current state, especially for a business user. Indeed, outside of simple “event simulation” components in CEP tools, I’m not aware of any business suite that does this effectively today. Probably you would need to automate support for some technique like Combinatorial Testing.
On the other hand, we do see occasions where test cases are generated by rules (and separate rules than those providing the CEP!) to try and provide good coverage - indeed this is used in some of the TIBCO BusinessEvents provided examples. Overall, test-based development and the use of techniques like event generators for testing I would consider as “best practices”!
Cheers
By Tina Groves, March 8, 2010 @ 11:06
The criteria seems fairly complete though latency appears missing (implied in the scalability criteria?). I agree with Hans that capabilities related to application management such as impact analysis, auditability and so on warrant inclusion. The scoring, as you say, is relative to the use case and, I would add, target environment. “CEP Opportunity Analysis and Assessment” by IDC’s Fleming and Silverstein (Dec 2008) does a nice job describing how priorities shift according to the use case.
Thanks for sharing.
By Paul Vincent, March 8, 2010 @ 15:01
Hi Tina: Thanks for the feedback:
- On “latency” and other performance metrics, of course everyone “has good performance” :). Then again, I didn’t specify this was a “paper evaluation checklist” so although I should do a “software trial checklist” later I will add performance to the platform component of this list.
- I would map “application management” to “operations and management”; “impact analysis” / “auditability” map somewhat to verification and validation (see the earlier comment response to Hans and James…). On the other hand, it is up to end user organizations to assess their need here, so I will add that in under business interfaces.
Cheers
By Keith Dunlop, August 24, 2010 @ 13:56
Invitation to register for Sybase CEP Webinar on September 14th, 2010
…
By Paul Vincent, August 26, 2010 @ 02:22
Thanks Keith - but we don’t really want the TIBCO CEP blog comment system to be used to advertise other vendors’ events!!! Nontheless, thanks for the invite and good luck.
Cheers