CEP versus ESP - an essay (or maybe a rant)
Posted by Paul Vincent
A colleague asked for some comments on the terms “CEP” and “ESP” - and in particular why, if a customer already had a stream processing product, they might still need some alternative or non-stream technology for certain other event processing duties…
1. There is growing industry consensus on the “CEP” and “ESP” terminology. “Event stream processing” is for things like market data streams (usually a high ratio of event throughput vs numbers of queries), whereas “complex event processing” is a superset term that also covers event-by-event processing and aggregation (e.g. on potentially out-of-order events from a variety of sources - often with large numbers of rules or business logic).
Sources:
a. The EPTS Glossary by most of the CEP vendor community covers these terms. In particular it defines Event Stream Processing as the processing of in-order events.
b. The “inventor” of the term CEP, David Luckham, provides an excellent differentiation on his web site.
c. Analyst comments on this differentiation by Bloor .
d. 3rd party blog comments by Opher Etzion , by Jack vanHoof , and by Tim Bass .
2. Generally “stream processing” processes event streams as “sets” and does “set type” operations, typically using “continuous queries” (SQL-type queries that operate over time and buffer windows). The wider “complex event processing” term additionally covers other mechanisms like ECA rules, production rules, and so forth. By definition, ESP products are therefore also CEP products in the same way that both Edsels and Bugattis are all cars.
While streaming query engines are very useful technologies they are not optimal for all CEP problems - in particular they have different characteristics regarding, for example, comparing out-of-order or out-of-stream events, applying decisions and reactions to event patterns, and so on. For this reason multiple CEP language classes have evolved, typically described as queries, rules and procedural approaches (to event pattern detection). This is also why TIBCO BusinessEvents explicitly supports state models and production rules as well as continuous queries, and is architected to allow additional Event Processing Languages to be added in future if desired.
Sources:
a. TIBCO Blog on Queries based on TIBCO’s coverage of ESP. Other ESP vendors are commented on in another post.
b. EPTS Language Working Group is defining some differentiations here but have not published their work yet: we blogged on a preview at a recent conference.
3. Because ESP is but a specialised subset of CEP, some people may have tried to make “ESP” synonymous with “CEP”. This remains a very grey area as while event stream processing using queries is a very useful capability (i.e. event set handling and detection), so is event-by-event handling and correlation using, say, inference rules.
So… in most enterprises there are usually multiple use cases for multiple types of CEP that are best handled by multiple paradigms (such as specialist ESP, event-driven business processes, rule-driven event processing, event-based business rules, event-driven analytics, etc). One should no more expect a large company to rely on a single CEP paradigm as it would on a single computer hardware technology.
Complex Event Processing (CEP), Queries | 21 August 2009 |
26 Comments
Other Links to this Post
RSS feed for comments on this post. TrackBack URI
By Hans, August 22, 2009 @ 09:16
Argh.
Anyone making a serious effort to sort CEP from ESP products will find nothing but confusion. They will go back and forth forever, with one side claiming you “can’t do this with that technology” and the other side showing that you can.
The strange (to me) thing is that you have such clear differences and terminology at your disposal, but you still rely on these horrible old terms.
If you removed all your references to ESP, you would actually come out with a much more clear message:
Both stream and non-stream approaches are subsets of CEP. There are clear differences between the approaches and the only way for anyone to make an educated foray into CEP is to examine both approaches in their context of their goals.
Engines that focus on streams as a primary concept will, by nature, express business logic using stream concepts like filtering, storing/retrieving, aggregating, joining, buffering/sorting, etc.
BE uses some stream concepts, but focuses on expressing business logic as rules and states. Many programmers and most non-programmers prefer this way of expressing business logic. Anyone currently working only with stream concepts should at least look at whether their logic would be a better fit to a rules and states approach.
Although there are various claims about performance of one approach or another, as with most software, customers overwhelmingly find the performance question to be highly subjective to the particular use of the product. It is simply not possible to genuinely prefer a particular approach on performance grounds, without digging a little into the requirements and business goals.
Argument complete! And all the follow up questions will be about logic capabilities, ways of thinking about performance and overall CEP strategy. Not useless talk about what is CEP, what is ESP and whether this is a subset of that.
By Paul Vincent, August 23, 2009 @ 05:44
Hi Hans - yes, I guess the main point is that there are various event processing problems (and technology solutions), and ESP != CEP. On the other hand, tools can be optimised for ESP or EBE (we need an acronym for event-by-event…), do either or both, etc.
Cheers
By Hans, August 23, 2009 @ 07:51
I would go a step further and say that either no tool is ESP, or almost every tool is ESP.
If you interpret ESP as “requires events to be in order” then every tool can easily process events that are out of order, and thus no tool is ESP.
If you interpret ESP as “uses a buffer to store events that were out of order, so that they may be processed in order” then every tool is ESP. In particular, BE is ESP since it uses a buffer in this way.
By Paul Vincent, August 24, 2009 @ 04:49
Hi Hans - in sofar that both farm tractors and garbage trucks have 4 wheels, then CEP engines can all “handle” out of order events, one way or another. But that is not the point. I really wouldn’t want to sow a field with a garbage truck…
The main thing to bear in mind is that there are various paradigms for (stateful/complex) event processing. These paradigms are suited to different event processing needs. Query-based set handling is well suited to ESP / ordered event streams, and less so for unordered disparate events.
The vendor markets seem to reflect this - stream processing vendors concentrating on (stream-oriented) algorithmic trading, whereas TIBCO CEP is distributed around (event-by-event) logistics, telco, finance middle office, etc use cases. This is by no means a coincidence!
Cheers
By opher etzion, August 24, 2009 @ 07:59
Hi Paul. I tend to agree with Hans that making distinction between the labels ESP and CEP is not a useful thing to do in practice, the reason is that all products now call themselves CEP, and the term ESP was actually introduced to describe Apama, which does not really do stream processing in the way that the database community sees this term, I prefer to say that there are different programming styles to do event processing. It is useful to make distinction between products that process events in an “event-at-a-time” mode vs. those which process events in a “set-at-a-time” mode. From theoretical expressive power point of view, there may not be significant difference. The difference may be in - ease of use, and ability to optimize for certain types of applications. My guess is that we’ll see everybody converging into hybrid approaches, where both processing modes can be implemented, under the same roof.
cheers,
Opher
By Paul Vincent, August 24, 2009 @ 08:30
Hi Opher: for sure we are talking about
1) applications (eg streams of events with concerns mostly about interrelationships within the stream) that are often best suited by set paradigms (eg time window queries),
2) the Event Processing Language paradigms themselves (inference or production rules, ECA rules, queries etc).
On the validity of the term “ESP”: I don’t think it will go away, and its possibly a useful description of a subset of CEP.
Having said that, probably it is up to vendors like TIBCO (who provide distinctive EPLs for rules and queries) to show best practices for when to apply which to what use cases… that is certainly an idea for a future post!
Cheers
By Hans, August 24, 2009 @ 09:10
Well, we agree that there are big differences in the approaches.
In the distant and foggy past, David started this ordered vs. unordered thing. Then someone said that stream engines must be better at ordered events and backed this up by stating that algorithmic trading mostly deals with ordered events because market data streams are ordered. This was picked up by Bloor and many others as fact, but unfortunately whoever said it had no experience with either a stream processing engine or algorithmic trading.
You may notice a distinct lack of actual users complaining about problems with out of order events in stream engines. In fact, none of the sources quoted in your post ever used a stream engine, nor do they quote or even mention talking to anyone who has.
The whole thing became an urban legend, where everyone “knows” that ESP engines are for processing in-order data, but no one can locate an actual user to back this up.
Rather than the ability to handle unordered events, I would say that the biggest difference between stream engines and something like BE is in how the EP logic is exposed to the user. Front office finance developers seem to like stream processing engines because (a) they are used to translating business logic into data processing algorithms and are happy to do this for the sake of performance and (b) trading systems are highly focussed on specific tasks and it’s usually easy to tell in advance what are the possible input events and what derived events you may get from them.
Most business logic is not focussed in this way and for an application composed of dozens, hundreds or thousands of decisions, it is much easier to declare if/then or when/then conditions than to worry about how the events are progressing from left to right through a graph. Also most developers (people in general) would rather think about business logic than data processing algorithms, even if the result is a few milliseconds of delay.
And so, the most common complaint (from experienced users) about stream engines is that the logic quickly becomes hard to understand.
This is an actual difference between the approaches and provides a clear reason why the mixture approach is clearly the best for most applications: in a mixture approach, one uses the logic style that makes sense for their particular problem, which reduces the chance of a mismatch between the requirements and the way that logic is expressed.
So yes, there are huge differences between approaches, but it’s got little to do with ordering of events.
By Paul Vincent, August 24, 2009 @ 14:13
Hi Hans - yes, ordering of events isn’t a big deal. Which is why I use the term “set based” for Event (Stream) Query Languages…
Interestingly the “hard to understand” scalability is often more a procedural / declarative issue… fitting a new rule / pattern into a large procedural flow diagram can be very problematic… rather than a query vs rule thing, IMHO.
Cheers
By Hans, August 24, 2009 @ 20:00
Paul, I think set vs event-at-a-time is maybe a little better. But you will find few (if any) real stream processing customers complaining about this difference. That’s because stream processing products mostly also allow for a range of event-at-a-time processing.
However, your point about scalability into many rules is well made. Stream products are mostly not quite procedural (with one exception) but still, it is generally harder to fit a new rule/pattern into one of these engines than into a production rule system. Of course there are exceptions, but I think that by far the code complexity/scalability/understandability issue is the biggest problem currently faced by stream processing engines.
By Paul Vincent, August 25, 2009 @ 07:28
Hi Hans - you are right, I haven’t heard any complaints (but I wouldn’t expect to either) - on the other hand, one wonders if there isn’t an element of “sticking through thick and thin” with a particular technology choice - when you have a round peg there appear to be many round holes etc - and equally this also applies, no doubt, to the ECA-based and inference rule-based EPL providers!
PS: Interesting comment on the (perceived) differences between streams and other event types on Charles Young’s blog at http://geekswithblogs.net/cyoung/archive/2009/08/24/134276.aspx - where he looks at MS StreamInsight versus the events coming out of BizTalk…
Cheers
By Paul Fodor, August 27, 2009 @ 07:17
The main difference that I see between stream processing languages and complex-event processing systems is expressiveness.
Most ESP systems are based on automatas (finite state machines), so one needs to know the entire automata to generate some complex patterns,
while systems based on rules/patterns to define complex events, the intermediate nodes/events are represented as rules and are combined automatically to construct event workflows and trigger actions (including using recursive event definitions, negation, etc.).
Stream processing systems cannot define complex events and use them in building other complex events, but, usually, they can just query (SQL like). For large worlds, one cannot see/represent the whole automata, so representing only the building nodes and letting the system to build the workflow/model has higher complexity (and can be slower than ESP (because of complex rule combination, keeping explicit intermediate events, indexing of rules, etc.)).
However, it mustn’t be the case that CEP systems are always slower than ESP systems because one can think of complex scenarios where CEP systems may perform better (e.g., processing out-of-order events, re-ordering, etc. (for instance, lets consider an example of re-ordering: a(X)*b(X)*c(X,1)->d(X), that is if an event a is detected and is followed by an event b and an event c with the same first argument, but a constant second arg (like a filter) (and a low probability to happen: 1/1000 c(_,1)), a fixed automata would follow the path for all events a,b (and do many copies of the environment, etc.), while an optimized rule might wait for c first and then check the history for a and b).
Technology ESP CEP
formalism automatas rules
By Paul Vincent, August 28, 2009 @ 06:20
Hi Paul - usually query languages are combined with some other mechanism (either explicit or implicit) to sequence queries together to provide “complex event” building blocks. And many query languages allow you to assert new combined (complex) events from query results. However, your points on the difficulties of out-of-order and re-ordered events are taken (usually via query language extensions in the case of the ESP tools).
) and you can't do aggregations via inference rules (OK, you can, but it might be messier).]
[Note that TIBCO's approach is to provide different Event Processing Languages for the different processing needs - eg state model / inference rules and continuous queries. You can't do inferencing via queries (well, I suppose you could
Cheers
By Paul Fodor, August 28, 2009 @ 10:24
Hi Paul Vincent,
I totally agree with your points above.
> usually query languages are combined with some other mechanism (either
> explicit or implicit) to sequence queries together to provide “complex event”
> building blocks.
Yes, querying one (e.g., well-founded semantics, production rules systems (PRD)) or multiple models (e.g., stable models, answer-set programming) has no support for processing events or streams. But, many works on complex event processing (CEP) extend or use these rule-based semantics for: processing events, finding complex events and performing updates.
My argument in my previous post was that, in general, CEP systems are usually based on some rule-formalism, that is defining complex events though rules, while event stream processing (ESP) usually are based on automata theory. I’m not saying that all CEP systems use rule-based formalisms or that all ESP (or SQL-like stream processing systems) use automata theory (although I never seen any that didn’t).
I think that there are advantages and disadvantages for both directions. CEP systems tend to be more theoretical complete (i.e. define event consumption policies, models, state transitions, etc.), while ESP are usually faster.
> And many query languages allow you to assert new combined (complex) events
> from query results.
I think you lost me here. What do you mean by “assert new combined (complex) events from query results”?
There are different ways in literature to model events in logic programming systems: using temporal logics, asserting events as base relations and using incremental view maintenance to find new composed relations. In PRD systems events are also modeled as base facts and the PRD system computes the model after some number (one or more) of events are received.
> However, your points on the difficulties of out-of-order and re-ordered events
> are taken (usually via query language extensions in the case of the ESP tools).
Yes, out-of-order is usualy modelled as rules.
> [Note that TIBCO's approach is to provide different Event Processing Languages
) and you
> for the different processing needs - eg state model / inference rules and
> continuous queries.
> You can't do inferencing via queries (well, I suppose you could
> can't do aggregations via inference rules (OK, you can, but it might be messier).]
Very good. I partially agree with you here that aggregates in rule-systems might be more “messier”, complex and difficult to define) then some implicit understanding of aggretates in automatas. Still, clear definitions of aggregate operations are necessary and will be defined in time (for instance, almost all LP model theories have aggregates for queries, but it took over two decades of research for this task and it’s still an open area).
Regards,
Paul Fodor.
By Hans, August 28, 2009 @ 16:24
Paul Vincent - It’s possible that people are not complaining because they are too busy working around the flaws in the product. But
Paul Fodor - Could you give examples of what you consider to be “CEP languages”?
By Paul Fodor, September 1, 2009 @ 11:01
Hi Hans,
> Paul Fodor: Could you give examples of what you consider to be “CEP languages”?
Yes. It is quite a new area, so there are a lot of experimental systems - most of them still in research. Maybe the best way to tell you about these languages is to mention one paper I saw in this year’s VLDB which enumerates many systems for CEP and ESP: Complex RFID event processing, by Fusheng Wang, Shaorong Liu, Peiya Liu, VLDB09 (see related works: the references 26-32 are about rule-based complex-events in active databases (more precisely, ECA (event-condition-action) rules), while 33-41 are about stream-processing and SQL-like stream languages).
I can also add other systems if you need: CEP rule-based languages: Prova, Drools Fusion, IBM’s AMIT, Etalis, Esper; or event-stream processing with automatas: Cayuga, Aurora http://www.cs.brown.edu/research/aurora .
Anyway, keep in mind that this is my own classification of event processing systems (CEP with rule-systems vs. ESP with automatas), and everybody uses these two terms (CEP and event-stream processing) interchangeably (and some systems just mention both to cover all the bases http://esper.codehaus.org/esper-3.1.0/doc/reference/en/html_single/index.html
Regards,
Paul.
By Paul Fodor, September 1, 2009 @ 11:08
> It is quite a new area, so there are a lot of experimental systems - most of them still in research.
I cannot emphasize enough that these are pretty new systems, so expect that they might contain bugs and lots of debugging. Even if the development is serious and the systems are pretty mature, they still cannot be 100% reliable.
By Paul Fodor, September 1, 2009 @ 11:15
Hi Hans,
> Paul Fodor: Could you give examples of what you consider to be “CEP languages”?
Yes. It is quite a new area, so there are a lot of experimental systems - most of them still in research. Maybe the best way to tell you about these languages is to mention one paper I saw in this year’s VLDB which enumerates many systems for CEP and ESP: Complex RFID event processing, by Fusheng Wang, Shaorong Liu, Peiya Liu, VLDB09 (see related works: the references 26-32 are about rule-based complex-events in active databases (more precisely, ECA (event-condition-action) rules), while 33-41 are about stream-processing and SQL-like stream languages).
I can also add other systems if you need: CEP rule-based languages: Prova, Drools Fusion, IBM’s AMIT, Etalis, Esper; or event-stream processing with automatas: Cayuga, Aurora http://www.cs.brown.edu/research/aurora .
Anyway, keep in mind that this is my own classification of event processing systems (CEP with rule-systems vs. ESP with automatas), and everybody uses these two terms (CEP and event-stream processing) interchangeably (and some systems just mention both to cover all the bases http://esper.codehaus.org/esper-3.1.0/doc/reference/en/html_single/index.html
Regards,
Paul.
By Paul Vincent, September 2, 2009 @ 01:58
>> And many query languages allow you to assert new combined (complex) events
>> from query results.
>I think you lost me here
Sorry: consider this:
implicit declarative approach: queryA asserts eventA which is an input to queryB
explicit imperative approach: queryA is modeled as providing the filter to queryB
In the declarative approach, eventA can be used by any (or none) subsequent queries. The “connector” between queries is eventA.
In the imperative / connection diagram approach, the “connector” is the modelled link. This makes it easier to draw and understand, but adds problems in maintenance and scale…
Cheers
By Kurt Rudahl, October 11, 2009 @ 21:39
Paul Fodor (Sept 1) says “see related works”. Where is this list to be found?
Thanks
By Paul Vincent, October 13, 2009 @ 06:15
Kurt - I assume Paul is referring to a list in the paper he references at the ACM portal which by his description covers “rule-based complex-events in active databases (more precisely, ECA (event-condition-action) rules), …[and] about stream-processing and SQL-like stream languages”.
It’s quite a common mistake (and I have no idea if the paper authors are making this mistake) to assume “rules” relate only to “databases”. Of course, one can take the meaning of the term “database” to be any system holding and controlling data (ie any system), but nontheless, a CEP technology like TIBCO BusinessEvents is never really described as an “active database”; it is however rule-based (as in production rules, covering ECA rules as well as inference/knowledge rules), and includes a distributed in-memory “database” (/cache/event store).
For a list of “event processing languages” check out the EPTS Languages Working group - see Ophers blog here - and also this previous blog post…
Cheers
By Paul Fodor, January 15, 2010 @ 10:02
Hi,
I know that I reply after such a long time, but I hope this can help others that read this blog. Paul Vincent is right about what I meant above.
Regarding the topic of rules version SQL oriented CEP, I now think that the most important difference is their expresive power. Rules can represent recursive complex events. For instance, a simple definition for transitive closure is represented in the equations below. Depending on the default consumption policy this transitive closure event can be triggered: for all reachable events in the “unrestricted” consumption policy or for the direct and one indirect reachable events in the “recent” consumption policy.
I checked CEP languages and I found the following: Aleri and Esper are based on SQL and are inheritably non-recursive, Apama is a procedural language and StreamBase is a graphical language, both restricting CEP to non-recursive complex events, Etalis is rule-based and recursive. I don’t know if Tibco can represent that.
reach_event(X,Y) <- edge_event(X,Y).
reach_event(X,Y) <- reach_event(X,Z) ’seq’ edge_event(Z,Y).
I think I am right, but this is how I see this different in approaches in CEP.
Paul.
By Alex Kozlenkov, January 16, 2010 @ 01:23
I think Paul Fedor has mentioned a very significant factor to consider for this comparison. SP operators are fairly static, the pattern is either detected or a timeout or other termination occurs. In reactive frameworks in
(a) functional languages,
(b) rule-based languages like Etalis and Prova,
(c) workflows based on process algebras,
“event detection” is much more dynamic, and allows for potentially infinite recursion, non-determinism, and agent-style protocol handling (as, for example, required by the Flower Delivery use case forthe upcoming Opher Etzion and Peter Niblett’s book on EP).
I am discussing a concrete recursive example here:
http://prova.ws/confluence/display/EP/Lifecycle+of+reaction+group+instances
and this is an example of a recursion:
server_2(Market,Limit) :-
@group(g1)
rcvMsg(Market,Protocol,From,update,price(Price1))
[Price1 .lt. Limit],
@group(g1) @not @timeout(1000)
rcvMsg(Market,Protocol,From,update,price(Price2))
[Price2 .gt. Limit],
server_2(Market,Limit).
By Paul Fodor, January 16, 2010 @ 18:23
Hi Alex,
I had a discussion with Dr. Etzion about recursion and there are two distinct components of the event processing language: the event processing agents (EPAs) with inputs and outputs and the event processing flow/network (EPN) (directed graph of EPA nodes).
He said that recursion can be applied by the fact that an output event from an EPA can also go back as an input for the same EPA.
I guess, with my background in logic programming, I’m trying to see it as a single whole declarative program with a model theory (behind the operational semantics, e.g., “take the output of one EPA and plug it as an input in another”) and maybe better integration with logic programming.
Up to what I understand, one has to build the event workflow, while, in real applications, there can be many types of events and the network can become too big to be built manually. In that case it would be better if events are detected without building the workflow.
I also see other drawbacks of the flow approach. Triggering an event in such a network is very similar to the operational semantics of Prolog. There is research for over 40 years in top-down evaluation and Prolog systems are highly optimized (the Warren Abstract Machine WAM). It looks to me that re-inventing a very similar process is not necessary. I know that Prova does it bottom-up, but that is also based on techniques in logic programming.
I will take a look at your example. Thank you for sharing it.
Paul.
By Paul Fodor, January 20, 2010 @ 14:19
Hi,
Trying to continue our discussion above I got interested in EPN and EPAs and searching for this topic on Google I found Opher’s hierarchy of EPAs
http://epthinking.blogspot.com/2008/04/on-event-processing-agents.html
Interesting is that just a few days ago we were talking about a related subject by email. Our specific subject was recursion in event processing. His point was that this is supported by loops in the event processing network (EPN). However, I was not convinced that recursion is not part of the event processing agent (EPA) (and depends on its expressive power). Researching more on the subject I started to think of reasons for changing this architecture (better integration of EPA and EPN, declarative dynamic workflow instead of fixed structure, etc.).
First of all, although I don’t agree with fixed workflows, I agree with the overall idea of an Event Processing World/Web of agents (a specific case can be a fixed network, but it shouldn’t be limited to that). There is definitely value in EPW by having separate processing entities in the form of agents communicating in a common environment (the world). This makes sense in the Semantic Web area where are distributed “namespaces” and agents in the entire Web.
Second, I see CEP agents as full agents with enough expressive power so they can compute any kind of complex events.
Regarding the hierarchy of EPAs, I have a few remarks:
-”simple event processing” EPAs - filter and routing,
Filter means selection with some condition on the original events, while routing is part of the event processing network (EPN).
-”mediated event processing” EPAs - enrichment, transformation, validation
“Enritchment” means using the internal database to enrich the input events with some extra information; transformation is both selection on some condition on the arguments and projection of some arguments of the original events. Validation is selection on some condition and maybe a mechanism of exceptions and alarms.
I’m not sure I can see the distinction between “simple event processing” EPAs and “mediated event processing” EPAs.
-”Complex event processing” EPAs - pattern detection
EPA pattern detection is detection of sequences of events and other patterns (based on time, filters with static conditions or conditions from the database, etc.)
Up to here EPAs were definitely “stateless”. However, “recursion” presumes that agents are stateful, or, at least, have access to their outputs and compute some extra events or a fixpoint of its own derived events. That is, the relations that define new complex events are recursive. The recursive example that we discussed is:
reach_event(X,Y) <- edge_event(X,Y).
reach_event(X,Y) <- reach_event(X,Z) ’seq’ edge_event(Z,Y).
I think this still follows the definition of CEPs:
“Complex Event Processing Agents is a software artifact that receives an event cloud or stream or collection of events or a single event, does some computation on these event, and produce one or more events as an output.”
However, this might also be a different category or EPAs:
-”Stateful Complex event processing” EPAs or
-”Recursive Complex event processing” EPAs
Personally, I prefer a hierarchy based on the expressive power in the class of “Complex event processing” EPAs. In logic programming, recursive programs are a completely different category of programs then non-recursive programs. Moreover, when one has negations (classical and default negations) the hierarchy becomes much more complex, e.g., stratified programs, non stratified programs, locally-stratified programs and not locally-stratified programs, with diverse logic semantics: well-founded semantics, stable models, etc.
-”intelligent event processing” EPAs - prediction, decisions…
Just to finish the hierarchy, I think that the last category is just about integration of CEP with other semantic constructs: “prediction” IEP EPAs are CEP EPAs with underlying meaning of events (semantics like in the Semantic Web and knowledge representation areas), while “decision” refers to integration with event-actions rules (up to my understanding). The point is that this last category is regardless to the issues discussed above.
Finally, the point of this comment is to see if the issues mentioned above are valuable or not, and that I believe in a better overall integration of the whole architecture for complex event processing and maybe in a different hierarchy of EPAs.
I hope this post also helps other people interested in the same problems discussed above. I also believe that more people involved and dissemination of ideas usually helps progress or just to filter out valuable or useless ideas.
Paul Fodor.
By Paul Vincent, January 21, 2010 @ 16:56
Hi Paul - usually agents are more semi-autonomous processes - and therefore we expect (and indeed see) EPAs covering various combinations of event processing functions (in the real world). Check out the EPTS work on Reference Architectures for some of the functional descriptions carried out in event / complex event processing.
Cheers
By Paul Vincent, March 2, 2010 @ 02:58
Tim Bass added his (somewhat predictable
) commentary on this discussion in his blog in this post.. See also Charles Young’s first comment for a riposte.
Tim did mention (in a comment in that post)
Suggestions that “statistical methods” are an alternative to rule (and query) based CEP are misleading: statistical techniques are often tested inside rules, and data for statistics can be generated through queries. In TIBCO BusinessEvents, for example, statistics for trend monitoring are provided in the rule language, and these operate against “histories” of concept values. For more complex statistical techniques (Markov models, neural nets, et al) then one can add in the full power of TIBCO Spotfire S+ analytics.
In short, any claim that any one particular representation for CEP - whether analytics / statistics, or rules and queries - is “best” or “true CEP” is simply irrelevent: the real issue is how to use any or all these techniques to best advantage to solve particular problems. And the selection of methods (for your event processing agents) will vary depending on whether you are detecting cyber attack events, or providing situation awareness of a business process, or tracking customers or products, and so forth.
Cheers