CEP vs. BRE - A TIBCO TTL (Top Ten List)
Posted by Alan Lundberg
My colleague, Paul, got lots of… let’s call it, “feedback” regarding his post on the impending demise of the standalone Business Rule Engine (BRE) Market. It seems there are lots of folks out there who feel quite passionate about the subject, so I thought I would continue, albeit from a different angle and relate it back to the CEP and BusinessEvents for comparison.
So… taking my cue from David Letterman and with a tip o’ the hat to Paul, here are the:
Top 10 reasons why TIBCO BusinessEvents (BE) beats a simple Business Rule Engine + JMS layer (remember, no wagering please)
10. BE is Standards-based (for concept/class models, state models, rule models etc)
9. BE requires no app server or RDBMS (for lower cost, and quicker deployment)
8. BE provides multiple options to extend to other event channel types (for flexible complex event processing)
7. BE has Rule / decision management (for business control of software services)
6. BE takes a co-operative agent approach (for co-operating components and event processing services)
5. BE supports high scalability (for parallelizing applications and eXtreme Transaction Processing)
4. BE supports queries as well as rules (for dynamic facts)
3. BE supports State Models as well as rules (for case management, entity lifecycles, etc)
2. BE is designed as a stateful approach (for saving temporal information between messages)
and the number 1 reason is…
1. Real-time event-driven support already built in to the BE rule engine (for efficient Event-Driven Architecture use)
Complex Event Processing (CEP) | 19 August 2008 |
By Peter Lin, August 19, 2008 @ 15:33
Of the 10 items, none of them are unique to Tibco business events. JESS and JRules have equivalent functionality in the rule engine. For me, the differences are in the tooling. JESS doesn’t focus on tooling, but it has reactive mode. JRules provides several ways to author and manage rules and it also provides some event capabilities.
My bias perspective is the CEP world has a lot to learn from BRMS and from pattern matching algorithms. On the reverse, the BRE world can learn a lot from CEP engines.
If I had to guess, it’s more likely the two will converge.
By vincent, August 21, 2008 @ 01:31
Peter is technically partially correct in that several of the 10 points Alan makes are also available in other rule engines. For example:
10. Arguable - for example PRR and RIF are still in development, although BE’s state model is a UML standard and others ruleflow mechanisms are not BPMN based.
9. Most BREs deploy via app server (although test instances won’t need that). BE is its own service container. Advantage BE, unless you really want a web service deployment (ie avoid EDA / CEP).
8. BE has built-in support for 2 external channel types, all other BREs presumably have to have any added programmatically. Advantage BE.
7. Rule and decision management is usually associated with BRMSs which are part of the BRE proposition. BE’s is not a unique feature.
6. Infrastructure for co-operating agents using other BREs needs to be programmed manually (eg shared blackboard). Advantage BE.
5. High performance and scalability is offered usually via the app server used by other BREs, and requires additional programming eg for load balancing. Advantage BE (where its automatic by default).
4. Currently few (if any) other BREs support an SQL/OQL mechanism. Advantage BE (with the caveat that other BREs have extended rule languages that cover some query language features, such as inner joins in JRules).
3. Few (if any) other BREs support state models explicitly. Advantage BE.
2. Most BREs support stateful operation, but not shared persisted failover-supported working memory. Advantage BE.
1. Most BREs are designed for transactional use, and don’t incorporate event semantics like TTL and event consumption explicitly. Advantage BE.
On pattern matching algorithms: BREs mostly use developments of Rete, as does BE. But BE is also extensible to other pattern matching mechanisms (as are other BREs presumably), and can (as an example) support SQL/OQL type continuous and static queries for certain types of patterns.
But we fully agree that the BRMS and CEP spaces have much to share. Watch this space!
By Peter Lin, August 21, 2008 @ 19:44
#10 JRules used UML in previous releases for modeling business object model (BOM), but that was poorly recieved by the users. JRules, Blaze and Drools all provide rule flow, but they don’t use BPMN or UML. Having used UML and worked at IBM in the past, not using UML is a good thing in my mind. Blaze and JRules currently support XML Schema for business models. PRR hasn’t been released yet, but I hope it will soon. RIF is DOA in my mind.
#9 JRules, Jess, Blaze, and Drools can all deploy standalone or in an appserver.
#8 Jess doesn’t explicitly define external channels. I don’t think the latest version of blaze supports external channels, but it does provide adapters, which is functionally equivalent. Likewise JRules doesn’t provide pre-defined channels out of the box, but it does provide API to build your own.
#6 Jess is used in many research agent projects and provides API for calling one engine from another instance. AFAIK blaze doesn’t support that out of the box. I don’t know if JRules provides that in the latest release. I know jrules provided agent functionality in the past, but most of that was removed the last few years. Drools doesn’t provide agent support out of the box.
#5 I know jrules and drools are both exploring data grid integration. Now that IBM bought iLog, I’m sure they will integrate object grid with jrules for fault tolerance. The most scalable approach is distributed RETE, which no product supports today. I should state that said and I filed a patent for distributed pattern matching in 2004. In 2006, I did a proof of concept integration with my engine Jamocha and Coherence. Said and I came up with the idea of integrating Jess with coherence back in 2003, so the idea is old.
#4 When FI bought RulePower, they acquired Relational Logic Translation, which has Dr. forgy listed as an inventor. JRules has fastpath, which is functionally equivalent. Back in 2000, Said and I explored the same idea, so the idea is old. Jess 6 provides API for querying a database out of the box. There’s still a lot of work needed in this area.
#3 I agree many BRE products don’t focus on state models. To date, blaze, jrules and drools focus on rule flows, which is a type of state diagram. It’s not a general purpose state modeling tool, so BE has an advantage there.
#2 I agree. blaze doesn’t support that in the latest release. JRules has been exploring it, but they haven’t released that capability. I know Oracle has integrated their rule engine with coherence, so they’ve done it. Oracle’s engine is a license of JESS.
#1 AFAIK, JRules provided event capabilities out of the box. Jess has provided reactive mode since version 5, but it doesn’t provide a standard syntax for expressing TTL. I hope my temporal logic extension for RETE helps push things forward. I’ve sent my paper to blaze, jrules, ernest, drools and the architect of BE at tibco. I hope some of the ideas in my paper get adopted by commercial engines and push things forward.
By vincent, August 22, 2008 @ 03:09
I think Peter is mostly proving Alans point. Just to clarify further…
#10. OK, some people don’t like standards!
#9. But very few regular BRE’s are deployed using their own container without an external app server. There is a reason for this, and why CEP-oriented BREs don’t…
#8. An important point here: things like DB Adapters (etc) for external sources of data (and runtime data transactions) are totally different conceptually from event channels (and event-driven rule execution). The former are important, though (such as the BE DBConcept - Database-derived concept). The only similarity is that both (data and events can) map onto the underlying data model for rule execution.
#6. Note that most BRE’s could invoke another rule service (but that is far from sufficient to support an agent-based approach). The point is that the shared distributed memory model (shared Working Memory) is out-of-the-box in BE.
#5. “Both exploring” and “will integrate” implies that this is future for other BREs, but currently included out of the box with BE. [Note that it is quite possible that a conventional BRE customer has done a custom integration with a high performance cache or data grid technology (like you thought of doing). Check out their Wall St customers...]
#4. BRMS’s use queries in rule management for verification etc, but Alan was referring to *runtime* continuous query support. A BRE could delegate this to a DBMS (or ESP tool), at the expense of performance (e.g. see 8 above) or complexity.
#3. No, “state model” is not the same as a “ruleflow”. They *can* be used for similar purposes (orchestration), but have different purposes and semantics (e.g. the action being in the state transition vs ruleflow task). For example: you can inherit a state model, have multiple state models active against current objects, use a state model for case management, etc. [Really, a ruleflow is a process diagram, although few BRMSs use standard BPMN notation or provide workflow extensions. Also there is a strong overlap with a Decision Graph, a type of decision metaphor].
#2. I hadn’t heard / seen any web references to Oracle and cache - it will be interesting if this is available (for Oracle customers) and adds another internal CEP competitor to the existing Oracle CQL / BEA tools.
#1. Some BRE’s have a notion of events, which is not the same as being “real-time” or CEP-capable. Also, Temporal Rete was presented at DEBS this year (by an ERP vendor but not apparently using their own BRE…).
So overall, Alan’s points still stand.
By Peter Lin, August 22, 2008 @ 03:30
thought i’d clarify on #3. I agree state model is different than rule flow. I would consider ruleflow a subset of state model.
#2 Oracle hasn’t publicly said anything about it. it was mentioned on the jess mailing list. There was work done on DJess a few years back with shared memory.
Do you have a link to the temporal logic paper given at DEBS? here’s an old paper on DJess http://www.waset.org/pwaset/v4/v4-18.pdf , which uses a shared memory approach. It’s not as robust as integrating coherence with JESS.
The next logical step with scaling RETE is to distribute just the indexes and partition the data. I’ve actually discussed this with Cameron and the coherence team in the past. A partitioned cache scales much better than replicated cache. I believe cameron has stated this numerous times in presentations over the last 2 years.
If tibco wants to license distributed RETE, said and I would be happy to consider any offers.
By Peter Lin, August 22, 2008 @ 03:43
Matthais used drools to do his research into temporal logic extension. I don’t know if he cites my paper or not, but I’ve had discussion with him on the topic. Here is a blog entry about his work at drools blog
http://blog.athico.com/2008/01/interview-with-matthias-groch-drools-as.html
I’ll have to ask him for his paper.
By vincent, August 22, 2008 @ 11:44
Hi Peter:thanks for the reply.
State: In terms of capabilities and usefulness, yes I would agree a ruleflow is a subset of a state model.
DEBS paper: sorry I didn’t see a link.
Rete scaling: we’re quite comfortable with BE’s shared-WM approach so far, but thanks for the offer! You should present findings on this at James Owen’s RuleFest or RuleML08 latter this year.
Note to Alan and Chris: we should blog some more on distributed Rete and State for performance / scalability (across event volumes as well as numbers of rulesets).
Cheers
By Peter Lin, August 22, 2008 @ 16:22
Since BE uses Coherence to share the memory across multiple agents, I would expect it to scale roughly the same as coherence. In my blog, I go into details on the trade-offs of shared memory versus partitions memory.
Shared memory is good for failover and redundancy, but it isn’t ideal for distributed reasoning. Ideally, the system should divide and conquer and only replicate the indexes. Each rule engine instance then joins on indexes, without needing the fact. This reduces the work for each engine, memory requirements, network traffic and makes pattern matching parallel. To the best of my knowledge, no one else has taken this approach with pattern matching algorithms or Rete.
I must say the discovery was purely accidental on my part. I spent 3 years researching distributed reasoning and one night had an “Aha” moment. Said tabet and I spent a year looking at existing literature and the current state of art in 2003. Once we realized no one else thought of the approach, we filed a patent for it. Patents are evil, but we would be fools for not filing a patent.
As I mentioned in previous comment. I spoke with the coherence development team about distributed Rete and most of them agreed with me paritioned working memory would be much more efficient than shared memory. I believe Cameron and the core coherence developers would be able to shed insight on it. The funny thing is, once I explained the patent to the coherence guys, their response was “that makes complete sense. I’m surprised no one else thought of it.”
By Peter Lin, August 22, 2008 @ 16:43
I thought I’d mention this paper published in 2006 in the P2P space by Reaz Ahmed.
http://bcr2.uwaterloo.ca/~rboutaba/Papers/Conferences/noms2006.pdf
Their approach is similar with one major difference. They replicate all the indexes, whereas my approach replicates just the beta node indexes. Also, their approach doesn’t use index joins, instead they use queries to find partial matches in other remote nodes.
If you want an invite to my blog, send me an email woolfel AT gmail DOT com.
By Peter Lin, August 23, 2008 @ 08:18
Hi Paul,
For those interested in the DEBS08 paper you mentioned, ACMQueue has it. Here is the URL
http://portal.acm.org/citation.cfm?id=1385989.1386008&coll=GUIDE&dl=GUIDE&type=series&idx=SERIES10714&part=series&WantType=Proceedings&title=AICPS&CFID=77485512&CFTOKEN=18609565
Now that I’ve read the paper, the approach described in the DEBS08 paper isn’t as efficient as my design. The paper also doesn’t provide a detailed description of the implementation algorithm. In contrast, my paper on temporal logic extension for RETE provides a detailed description of the temporal distance calculation and an abstract description of the implementation. I also point out important issues and considerations, which the DEBS08 paper does not address.
The biggest challenge with supporting temporal operators is restricting the syntax to avoid non-sense rules. The paper doesn’t address this issue, so there is still a lot of work needed before a coherent temporal modal for Rete is realized.
By Chris Bird, August 29, 2008 @ 04:00
I have been reading this paper…
An Abstract Semantics and Concrete Language for Continuous
Queries over Streams and Relations
Arvind Arasu Shivnath Babu Jennifer Widom
Stanford University
farvinda,shivnath,widomg@cs.stanford.edu
at the following URL
http://dbpubs.stanford.edu:8090/pub/showDoc.Fulltext?lang=en&doc=2002-57&format=pdf&compression=&name=2002-57.pdf.
This is clearly not new stuff, but I am curious if there is any real relevance here. It seems intuitively sensible that some kind of treatment of real time streams with SQL like language could be appealing. Especially as there will often (I suspect) need to be a join over stream data and relation data.
What does anyone else think? Is the approach feasible? Is it just research? Can we envisage products?
By vincent, September 1, 2008 @ 00:57
Shared vs Partitioned (working) memory: actually I believe you need both (so the data grid approach used by BE allows for this). For CEP, where scalability to load (/low latency) is required, controlled shared and non-shared distributed memory is a required. The sort of controlled distributed reasoning we tend to need is multiple disparate agents sharing a blackboard model.
Automatic scaling of Rete (for v large rulesets) is an interesting idea but I’m not sure there is any version of Peter’s algorithm in production anywhere. Note that large rulebases are usually formed (in business applicaitons) by rule generators, and often such rules can be analysed and reduced in volume (per model-driven engineering, in the transformation from CIM to PIM - using MDA nomenclature), to reduce the load on Rete.
It would be interesting to see what use cases there are for v large rulesets and compare approaches…
By vincent, September 1, 2008 @ 01:00
BE Shared Memory: just a note to confirm that although BE (>v2) ships with Coherance to handle shared memory, it is architected to potentially support other types of data grid too.
Such data grids are an increasingly interesting alternative to expensive centralized database transactions, especially for CEP and knowledge-based applications.
By vincent, September 1, 2008 @ 01:05
Temporal Rete: this is an obviously interesting area of research. I wonder if any such engines are in use anywhere?
For BE, the temporal aspect is covered through events (and in particular, timer events), and the production rule language is designed for technical users. So an Event Processing Language that included temporal constructs could map to events AND rules, as opposed to just temporal rules, following the model-driven engineering approach.
Obviously this might change in future, and the nice thing about the shared memory approach used in BE is that different types of engines can share the same data (eg Rete, Common Logic, neural net, etc).
By vincent, September 1, 2008 @ 01:08
Chris: thanks for the link (although it didn’t work for me - I found another one, also an ACM link and a slideshow version!). For the most part I understand these are older papers from Stanford - we’ll have to do a CEP geneology chart at some point!
Continuous queries (item#4 in Alan’s Top 10 List) form the heart of many of the niche CEP tools (and are an optional engine type in BE). In BE, of course, you can use Rete to reason over the changing result of a continuous query (i.e. co-operating paradigms for CEP). This sort of cooperative problem solving can be very useful in complex processes (/complex processing of complex events!).
Cheers
By Peter Lin, September 1, 2008 @ 06:49
Hi Paul,
thanks for taking time to respond. From the description, it sounds like BE supports collaborative agents, since they share the working memory. The partitioned memory approach was inspired by requirements said and I saw in the past. For small datasets less than 1-2 million using shared memory should be sufficient. I say “should” since the rules will affect the number of partial matches and impact the performance. I’ve seen cases where a dataset of 500K facts produced 500Mb+ of partial matches. Developers have to be careful about how they write rules and avoid exponential memory consumption as a result of partial matches.
Coherence supports shared and partitioned configuration, so using coherence doesn’t necessarily mean BE supports the distributed pattern matching I described. Shared memory design means n nodes repeat the same process n times, which is a waste of CPU cycles, IO and memory. It would be like having n people re-assemble the same puzzles n times. Distributed pattern matching with partitioned data is like dividing a puzzle into n sets and have each person work on just their set. From first hand experience researching and implementing Rete, you can’t bolt on the distributed pattern matching I describe as an after thought. The design of the engine has to detach the indexes from the Working memory elements. If the indexes are tightly coupled to the WME’s, the engine can only use shared memory.
In terms of temporal Rete, my rule engine jamocha implements the extensions I describe in the paper. I believe the architect of BE has already read my paper, so he maybe able to fill you in. If you have any questions, feel free to ask. For the record, my temporal logic extensions are open and I have no plans to patent them. My hope is that people use the temporal logic extensions.
I believe my temporal logic extension and Distributed Rete extension can be applied to different types of forward and backward chaining engines. That would include leaps and other discrimination networks. In my mind, shared memory is orthogonal to pattern matching algorithms. The catch though is how an engine implements it. The simplest way is to use Distributed HashMap, but has scalability limits. It’s the same reason Oracle and Db2 replicated indexes across partitions.
By vincent, September 1, 2008 @ 12:38
Hi Peter:
Yes - BE supports collaborative agents (either via event passing and/or shared memory / data grid).
For large data sets one would use different agents with different views of the blackboard / shared memory area (i.e. with access to different classes) and apply apply layered reasoning (different depths of knowledge / rulesets) to the events and event histories. The idea is to try and avoid the problem you tried to solve - i.e. here is a very large set of data to apply to a very large set of rules, all in one go, with a mechanism that allows for real-time complex event processing alongside more knowledge-rich, slower rule processing, to collaborate on solution. “Divide and Conquer” etc etc.
Note also that with CEP, I only want to process an event once, but I might want to process it against all other available data. The same rules are processing *changed* data (i.e. typically the new event, and any assertions / changes from the other processing agents). So there is no “waste of CPU cycles”, and low latency is preserved with a small overhead of synchronizing data updates across WMs (as of course there will always be *some* overhead)…
There are a number of large customers using the BE distributed rule engine architecture already, and we hope to get permission to name some of them later.
Thanks for helping with the ideas on how to explain this material, which we’ll probably expand in a specific blog post later.
Cheers
By Peter Lin, September 1, 2008 @ 17:54
From the description, it sounds like BE recommends partitioning rules. Having used that approach in the past, it’s a powerful technique. That technique only goes so far and at some point, it runs into serious scalability issues. Not all problems can be solved with that technique and it requires quite a bit of expertise to do. I would argue very few people have that skill, so implementing it is costly from a “man hours” perspective. Also there are many situations where partitioning rules is impossible and impractical. I’ve done quite a bit of research and tried to push those approaches to the limits.
For the scenrio in the response, rule partitioning makes sense, but there are many other scenarios where that approach won’t work. Using the distributed Rete said and I invented, the engine takes care of distributing reasoning dynamically. This lowers the learning curve and makes it easier for developers. That doesn’t mean it’s easy, but it makes it easier and solves many scalability issues.
[[Ed note: duped comments combined]
By vincent, September 2, 2008 @ 00:20
Hi Peter:
There are 4 use cases that BE allows:
A - shared memory, shared rules = replication for throughput
B - different memory, shared rules = data partitioning to handle data volume
C - shared memory, different rules = co-operative agents working against the same information
D - different memory, different rules = independent agent.
Your use case appears to be case B; most BE users today are probably doing A (remember most TIBCO customers are large enterprises). The power of this architecture is in the combinations of A-D that you can use in the same distributed application. The design approach is to use C to handle application complexity, and D in a different agent set to provide processing of large volumes of information, and then A for necessary performance / failover support as required for B and C agents, with D probably providing internal BAM monitoring / checklisting.
Note that the event-driven approach also provides a slight paradigm shift over conventional rule engine usage, which assists with understanding the needs for partitioning…
Cheers
By peter lin, September 2, 2008 @ 05:12
thanks for the additional information. Actually, distributed Rete can handle all those cases. Where it accels is with A and C. The key benefit is the cluster manages the memory dynamically and avoids the cost of replicating all the data when shared memory is used. If a rule engine can use index joins to pattern match, why send all the extra data? In my mind, B is a subset of A. For B (different memory, shared rules), one can do that today with any of the existing products without using coherence or a data grid. Of course it may not be fault tolerant, but it’s quite common to deploy the same rules to multiple instances and use load balancing to distribute load. Likewise most of the commerical Rete engines can handle D today. From a technical perspective, B and D are closer to load balancing than distributed pattern matching with shared memory.
Imagine how much faster BE would be if the IO and memory usage was reduced by 4x or 10x. Replicating the full working memory is ideal for fault tolerance, but suboptimal for increasing pattern matching performance and throughput.
I agree an enterprise solution needs to provide fault tolerance, but how a system achieves that result will vary. Some approaches are more efficient than others.
By vincent, September 2, 2008 @ 14:48
Hi Peter: you asked:
“If a rule engine can use index joins to pattern match, why send all the extra data? “
Remember for event processing, the “extra data” is an event at a time plus any inferences, so this could well be less than re-sending index information around.
“Replicating the full working memory is ideal for fault tolerance, but suboptimal for increasing pattern matching performance and throughput.”
Of course, application reliability and uptime is as important to our customers as performance. Especially resilience in *distributed* systems.
“Replicating the full working memory is ideal for fault tolerance, but suboptimal for increasing pattern matching performance and throughput.”
Ah - but the point is that the WM is partitioned too!
As I said earlier, performance will depend on the use case. We need to get PRR or RIF out of the door so we can have some decent benchmarks to prove this stuff in action (across different frameworks).
Cheers
By Peter Lin, September 2, 2008 @ 19:20
I’m not sure I understand the following statement
“Remember for event processing, the “extra data” is an event at a time plus any inferences, so this could well be less than re-sending index information around.”
To a Rete rule engine, the inference would be some fact, which may be an event. From a Rete algorithm perspective, it doesn’t make any difference. Assuming the engine provides proper temporal logic and constraint support, replicating just the beta node indexes is much more efficient than replicating all facts, which will result in each engine reproducing the alpha and beta memories. To be clear, when I say beta node indexes, I’m referring to hashed beta memories.
In the situation where the rules do not have joins, the cluster basically runs in share rules, partitioned memory mode.
[Edited: removed condescending statements]
By vincent, September 3, 2008 @ 01:37
Hi Peter,
“replicating just the beta node indexes is much more efficient than replicating all facts”
Which is certainly true if you are replicating all facts on every (rule execution) cycle, as the Rete index represents a “compiled form” of the conditions dependent on those facts.
But in BE (’s Rule Engine) you are:
(a) not replicating the entire WM on every cycle
(b) only replicating certain WM “as needed” (and even then, note that “notification of change” <> “replicate WM fact”, and that not all WM will need to be shared anyway)
(c) due to the event-at-a-time nature of continuous, complex event processing, we are likely to be replicating small numbers of facts in any cycle anyway.
So any assumption that the
Performance(replication(index)) < < Performance(replication(WM delta))
is, at best, very use case and implementation dependent - and in the case of CEP, where the WM delta for shared facts may be very small, this might not be the overhead some might assume…
[Note: I don't want to undermine Peter's work on Distributed Rete - developing a distribution mode for Rete indexes, handling the issues of managing rule (Rete) updates, etc, are all complex problems - is very interesting, although probably with high levels of dependancy on middleware latency like any other distributed rule solution, and probably unproven in a CEP application.]
In any case, thanks for the dialog as it certainly helps to explain / position how BE works!
By peter lin, September 3, 2008 @ 17:13
thanks again for taking time to respond. I’m enjoying the discussion. During my lunch break I looked at BE 2.1 user manual and came across the following paragraph from page 185.
When the Instance.getByExtId(extId) function is invoked, the engine first
looks in working memory and the Rete network for references to the instance. If
none are found, then the engine gets the instance from the cache and asserts it into
working memory. If the entity instance is not found in working memory or cache,
your application logic determines what to do next, for example, create a new
instance of that entity type.
From my understanding of how BE works, data is divided into concept instances and event data. The concept instances “can” reside in coherence outside of BE rule engine. The event data enters BE through channels. What BE calls concept instances sounds functionally similar to initial facts in expert systems. That data tends to be relatively static, so it makes sense to put it in an in-memory database. I can’t tell if the event data is also stored in coherence. Is event data also stored in coherence? The paragraph seems to imply developers are responsible for partitioning the data and asserting it to each BE engine instance. Is that a correct interpretation?
In the interest of exploration, I thought it would be good to explain the approach I used to implement fault tolerance with Rete. In Jamocha, the alpha and beta node memories use HashMaps. When I implemented a proof of concept fault tolerance with coherence, I changed my collection factory to return coherence HashMap, instead of java.util.HashMap. This has several benefits for those familiar with fault tolerant Rete. The first is that one engine in the cluster performs the pattern matching. The other active engines sharing the same rules and facts have their node memories updated through coherence. This avoids asserting the same facts in multiple active engines sharing the same rules and eliminates redundant rule activation. The second is that if the fact doesn’t create any partial matches, it never replicates the node memories, since there’s nothing to replicate. In contrast, asserting the same facts in multiple rule engines will cause each engine to go through the pattern matching process.
For the record, this is basically what Oracle did with JESS. I’ve been blogging about this specific topic since 2005, so several people in the field are aware of these issues. The redundant activation issue is quite old and predates my own research. When I designed jamocha, I specifically chose to use Maps for this reason. The other benefit of using Maps for the node memories is it makes it easier to implement distributed Rete.
Does BE integrate coherence Maps for the node memories, or is it simply using coherence as an external datastore?
By vincent, September 4, 2008 @ 05:05
Hi Peter:
My comments apply mostly to the current shipping version of BusinessEvents, which has extended its distribution mechanism over the version you have docs for.
“the engine first looks in working memory …”
Since BE2.2, the strategy per class is set by the developer, for in-memory only, cache+memory (default), and cache-only (i.e. explicit retrieval) modes. This is for fine-tuning and performance reasons, obviously.
“data is divided into concept instances and event data”
Event objects are really just a special type of object (and with appropriate event classes). Events are characterised by their Time To Live (how long they live in the Event Processing Agent, whether rule-driven or otherwise), and what “consumes” them. The latter is important because the registration of “consumption” can be used by the middleware layer to determine a successful “delivery” of the event. If the system has a catastrophic failure before event consumption, the event can be consumed by another agent instead.
“data tends to be relatively static, so it makes sense to put it in an in-memory database”
The in-memory role is primarily for reduced latency. CEP is about comparing new events with past events, so the latter must be easily (and speedily) accessible, whilst be capable of being updated as needed with information from the new events.
“asserting the same facts in multiple rule engines will cause each engine to go through the pattern matching process”
In the general sense, yes, but remember that raw / external events are only ever delivered to a single agent (for processing, and thence assertions / updates to any shared facts).
“Does BE integrate coherence Maps for the node memories, or is it simply using coherence as an external datastore?”
The latter, as sharing Rete node information would not be useful for other Event Processing Agent types (ie non-Rete) that are contributing to the solution.
[Incidentally, for usage advice and commentary with BE, end-users and partners should head over to TIBCOmmunity's BE discussion forums.]
Cheers
By peter lin, September 4, 2008 @ 08:14
Thanks for taking time to respond. I appreciate it. The information is quite useful.
I look forward to see where Tibco BE heads in the future.
By Ken Cottrell, August 24, 2009 @ 18:29
Here’s a question for the CEP industry experts…. how many of the other vendor CEP packages offer state models? I would think that state models that provide deep context of the real world are one of the most important sources of events, if not the most important. Your comments please.
By Hans, August 24, 2009 @ 19:51
Ken, I just happened to be browsing the comments and I saw your question about state models. To save the TIBCO guys from having to respond about their competitors capabilities, I will respond.
Amazingly, BusinessEvents is the only product to offer good state models for tracking entity states. I have never understood why.
SQL-like vendors offer something that Esper calls patterns. This is a way of expressing something like “event A before event B but not before event C”. This kind of thing is very useful, but provides only limited control over states.
No other CEP product allows you to define a state model for an order (for example) and have the system track the order through the states.
Some vendors let you code your own state machine in a script-like language, but I would not consider this to be the same thing.
By V.Arun, January 21, 2010 @ 07:38
Whether we like it or not, unfortunately BE does not have competetive UI which can be used by business users easily.
By Paul Vincent, January 21, 2010 @ 16:53
Hi Arun - depends what are you expecting business users to do - developing CEP applications is probably out of scope, but maintaining then (such as certain event patterns or decsision rules) can make sense. Hence BusinessEvents Decision Manager makes perfect sense - although as a RCP client it is aimed more at the business analyst types who would also download and use BusinessStudio… more on this topic in a few weeks…
Other Links to this Post
links for 2008-08-27 » Smart (Enough) Systems, the blog — August 27, 2008 @ 08:03
RSS feed for comments on this post. TrackBack URI
Leave a comment