Overview

Around a decade ago, it was possible for a single, precocious developer to build an entire SOA-like platform single-handedly. Granted, it was not an easy task to construct a service container and surround it with robust connection management, application security, transaction management, and reliable messaging, but it was at least an attainable goal. It was a proud achievement to build a toy application server or MOM platform in one’s basement.

Those were simpler times. Today’s SOA suite is far too vast for one person to build, and uses computer science that is more advanced than a basement hacker can contend with. In the first place, today’s services—as I argue in my new book SOA Cookbook (Packt, 2008)—are designed as processes rather than simple method calls. Building an SOA process engine means brushing up on, or learning from scratch, Finite State Automata, Petri Nets, and, perhaps, the Pi Calculus. Additionally, these processes use business rules to decide on next actions. Building the required rules engine presupposes proficiency in forward-chaining rule definitions, the RETE algorithm, and, possibly, genetic algorithms. Toy SOA is a relic of the past; today’s SOA requires a large team of very smart people.

Still, for all its might, today’s SOA is merely reactive. An SOA process provides a business function that reacts to requests. If the function is requested ten thousand times, the process is run ten thousand times. It is not in SOA’s mandate to make sense of trends, or to find suspicious happenings, across those ten thousand invocations. SOA’s job is to be a workhorse, serving the requests. It is not an anomaly watchdog or a slice-and-dice business analysis tool. Still, many enterprises seek just such a capability.

Not surprisingly, SOA vendors are starting to add Complex Event Processing (CEP) to their solution. CEP listens on the bus for the same events that pass in and out of SOA processes, but—unlike SOA—is concerned mainly with the detection of patterns that emerge across events. It does this, as we’ll discuss, by applying event-pattern matching rules to the events. CEP thus relies on both SOA’s event bus and its rules engine. Not only does CEP fit in the SOA architecture, but also provides the watchdog capability that business stakeholders find so valuable.

CEP and SOA

CEP isn’t a new technology but is only now gaining traction in the marketplace. David Luckham wrote the definitive book on CEP (The Power of Events, Addison Wesley), which was published in 2002. Over the years several academic implementations have been developed, notably Stanford’s Rapide and Cornell’s Cayuga. Numerous CEP startups failed in the early days of CEP, during the rise and fall of dotcom in the late 1990s. It was to help market one of these companies, ePatterns, that Luckham started work on his book. The company didn’t pan out, as Luckham reminisces in his article “A Short History of Complex Event Processing Part 3: The Formative Years.” (http://complexevents.com/?p=553), but the book lived on, becoming the bible of events.

Until recent years, the most established commercial implementations came from small vendors, such as Coral8 and Streambase. But within the past few years the major SOA vendors—who had previously kept arms-length from CEP, doubtful of its market potential— have started to incorporate CEP into their stacks. IBM, for instance, recently acquired Coral8. TIBCO aggressively sells its CEP tool, BusinessEvents, in its SOA and BPM deals. Oracle, thanks to its BEA acquisition, offers a product called “Oracle” CEP.

The combined SOA/CEP offering is encouraging, because CEP’s value comes not from its intrinsic value but from its contribution to an overall solution. SOA vendors, in a sense, have been in the “overall solution” business for years. They promote the idea that business architecture should be built on services, and they provide a platform on which to build those services. But what’s always been missing was a tool to watch the services, to make sense of what’s happening operationally.

What CEP and SOA have in common is events. Both technologies use events, but for different purposes. SOA processes use events to drive control flow. An SOA process is started by an event, and during the course of its execution waits for further events to propel it forward. Events in SOA, in effect, force process transitions. Most SOA processes not only receive events but also send events. When a process sends an event, another process receives it. Elaborate choreographies arise when the processes of multiple organizations engage in conversations of events.

CEP, by contrast, is a rules engine that uses events to trigger rule evaluations. CEP is constantly listening on the SOA bus for events. As we explore in more detail below, by using event pattern matching rules, CEP is able to infer causal connections between seemingly disparate events. CEP can also aggregate a set of inconsequential low-level events into a single, business-meaningful complex event. In stock trading, for example, a customer’s purchase of shares is the combination of four events (keeping it simple):

1. The customer’s request to the broker to buy the shares
2. The broker’s placement of the order
3. The result of the order, including the price at which the shares were bought
4. The broker’s response back to the customer.

When CEP detects these four events, it publishes a complex event—BuyOrderCompleted, let’s call it—back to the bus, where interested listeners may pick it up. The following figure illustrates the sequence of steps; this design is discussed in more detail below.

CEP can also detect breakdowns in the buy order. For example, if the broker fails to respond back to the customer in a timely fashion, CEP can easily spot this and publish a BuyOrderBrokerResponseLate event. CEP can also spot anomalies that span multiple orders. For example, it can detect suspicious broker activities like a buy-ahead, in which the broker, when directed by a customer to buy shares, buys his own lot of shares first, and sells them after placing the customer’s order. (The broker makes a nice profit from this furtive trade; the customer’s order, when large enough, drives the price up, so when the broker sells, he comes out ahead.) It would be fiendishly difficult to build a detection mechanism for buy-aheads into SOA processes; CEP, as a watchdog off to the side, is much better equipped.

This trade monitoring example is one of many CEP use cases. CEP use cases arise naturally from SOA use cases. If events move through the bus to drive services and processes, there is bound to be some important business reason to infer complex events from them. Indeed, many CEP use cases require SOA. Fraud detection, for example, which checks for anomalies in transactions on an account, must reside on an SOA platform with processes in place to perform transactions on accounts. The fraud detection rules must understand the structure of the account events and the protocol governing how they are exchanged.

CEP in the SOA Stack

The following figure shows where CEP fits in the stack offered by major SOA vendors.

The three layers of the operational stack, shown on the right side of the diagram, are the Enterprise Service Bus (ESB), the Orchestration/Service Engine, and the BPM engine. The Orchestration Engine runs SOA processes, and it uses the ESB, which acts as a full-featured message broker, to send and receive events to its partner systems. Processes that involve both system and human work are split between the Orchestration Engine and the BPM Engine. A workflow process running on the BPM Engine invokes SOA processes in the Orchestration Engine to perform system actions; an SOA process running on the Orchestration Engine uses processes in the BPM Engine to delegate human work. You can refer SOA Cookbook (Packt, 2008) for an expanded discussion of the stack.

CEP and its cousin Business Activity Monitoring (BAM) are shown as pieces off to the side. They are the not part of the core operational SOA platform, but rather provide an important business monitoring capability. BAM, in most implementations, is a summary view of business processes in the system. Like CEP, it observes the progress of operational processes, though it handles this data somewhat differently. BAM’s purpose is to create a rolled-up, read-only data view. CEP, in contrast, tries to infer complex events that can trigger follow-on actions. Both CEP and BAM funnel (as the figure shows) have large volume of events from the operational stack, reducing all that chatter to a smaller set.

The following table maps this stack to vendors. We consider the four major SOA vendors: IBM, Oracle, BEA, and TIBCO. BAM is omitted from the table; CEP is our focus.

The operational stack supports a distributed architecture in which the activities of client applications and partner services, both internal and external to the organization, are coordinated by orchestration processes. Clients and partners communicate with these processes through the ESB. Internal connections typically use MOM queues to access the bus; external connections use SOAP over HTTP. The orchestration processes, besides coordinating partner activities, also interface with backend systems (databases, mainframes, and so on) and use BPM to delegate manual work to human actors. The following figure illustrates this architecture.

CEP consists of a set of rules that listen for events from the bus. (A publish-subscribe messaging infrastructure allows CEP to get these events without disrupting the operational processes.) CEP rules, as we discussed above, infer complex events from these operational events. They send complex events back on the bus on orchestration queues, where, as we discuss further below, they are picked up and handled by CEP-aware orchestration processes.

CEP must be sufficiently scalable to handle the high volume of events on the bus. There is more to this than sheer processing speed. CEP, like most rule-based systems, is stateful. It keeps its state (i.e., its current set of asserted facts) in a datastore called Working Memory. For performance reasons, Working Memory should be kept in physical memory, and paged out to disk when necessary.

Using an RDBMS instead of memory would prove to be simply too slow and only a few commercial databases can handle updates at the volumes of events on a bus anyway.

This memory should be fault-tolerant, given the business criticality of CEP. Such an implementation is full of the sort of thorny performance design issues in which SOA vendors have the greatest expertise.

BAM works somewhat differently from CEP. Orchestration and BPM processes have state, and BAM uses this state, combined with business data from backend systems, to present a consolidated view of the state of the processes. For example, rather than tracking just order processes, BAM might also track the orders themselves, as they appear in the backend order system. In the previous figure, we labeled this piece as BAM/BI (for business intelligence), because most BAM implementations by themselves lack the ability to incorporate application data. Combining process state with application data often requires the combination of BAM and some sort of BI or analytics tool.

Whereas BAM merely presents a view of the system, CEP, by creating events that it has inferred from its observations, serves as an active participant. SOA platforms with CEP are self-healing; CEP watches what’s happening operationally and sounds up when it detects a significant occurrence.

The dotted line in the previous figure from CEP to BAM suggests that complex events created by CEP could be incorporated into the BAM dashboard. Thus, the BAM dashboard for the trading application could show statistics on completed orders or buy-aheads per broker. The mechanism by which CEP notifies BAM of complex events is implementation-specific.