(For more resources related to this topic, see here.)

What is event processing?

In the world around us, every second of every minute of every hour, the human brain is bombarded with a limitless number of things that happen either at the same time or sequentially, or in a totally and seemingly erratic way that may not make sense immediately but as more of these things happen, we can start to understand their relevance and importance.

For example, we hear cheering in the distance, we see balloons flying in the air, music starts to play, police cars and trucks appear pulling brightly covered trailers with puppets and people waving on them, followed by ambulances, and today’s date is July 4th. Individually, these events could mean anything, but together? It’s probably an Independence Day Carnival Parade!

Our brain can easily determine this fact in the blink of an eye” and while not overly simple to define in computing terms, we could describe a “Parade Event Pattern” as follows:

One (or more) police cars + followed/preceded by, or adjacent to + one (or more) carnival trucks + followed/preceded by, or adjacent to + one (or more waving people) + followed/preceded by, or adjacent to + one (or more emergency vehicles) + where music can be heard + and today’s date is 4th July

Your brain is not restricted to sending information and just waiting until there is a response, or forced into following a series of fixed steps to get something done. As with this example, it is able to take the events happening now, their relevance to additional external factors such as today’s anniversary date and understand a “parade” event pattern.

So as you learn more about Complex Event Processing, we focus on how this technology can take continuously flowing, never-ending information, from a potentially unlimited number of different places, and immediately understand how it relates to things happening right now and in the very near future, commonly known as Real-Time Situation Awareness.

Relating this to a business in computing terms

The problem now in the world of computers is the proliferation of data. Information arrives from many different systems, in vast quantities, at different times, at different speeds, some of importance now to certain other systems, people or processes, and some stored for later recovery and determination. Why the proliferation now?

There are many issues involved, but here are just a few major ones:

• The cost of computer power and sophisticated environmental sensor devices has become less expensive

• Networking capacities increase and become more intelligent

• The many different functional computing silos (finance systems, manufacturing systems, sales systems, and so on) are broken down, rewritten, enabling processes that can span more and more business demands

• New computer solution demands expand beyond the enterprise to include partners, customers so more and more data sources and other inputs are brought online

• Computing technology architectures such as Service Orientated Architecture (SOA) becomes increasingly successful, resulting in an ever more elaborate ecosystem of re-usable services

• A Big Data explosion, a term now used widely for information that arrives in high volumes, with extreme velocity, and in a wide variety of mostly unstructured formats emanating from social media sites, cell phones, and many other sources

• A growing demand from businesses that expect their Information Technology (IT) teams to respond to market situations much more effectively in real time

As we evolve and the complexity of these systems “pour” more and more huge volumes of information at computer applications, we are reaching a “tipping point” where traditional point-to-point or request-reply-based solutions of the world break down and become unmaintainable and not extendable.

A company business can be influenced instantaneously from things (events) that can happen, not only in the “cozy” understandable world within its own environment but also from activities (events) from beyond, such as from “the Internet of things”—realtime sensor device that can measure and report on a multitude of situations, including “the impending danger from a sudden rise in temperature in a food storage facility” or “the global positioning system location of a shipping container which is having an unauthorized opening with movement detection sensed from within”.

Immediate impact to a company’s business can also come appear “out of nowhere” emanating from a change in global business conditions indicated from the everexpanding social media outlets, for example, Twitter, instant messaging, and so on. Millions of people at the same time can all comment on the poor condition of a new product, highlighting an immediate need to change a product design. This will inevitably affect profits and will probably significantly affect the value of the business. So companies are now inevitably being manipulated by a wide range of both understood and misunderstood events.

In the past, probably going back over 15 years ago, business applications have had to conform to the methodologies, structure, and interfaces from the then available computing technologies (such as databases) where information must be inserted and statically placed. Only after this can users then analyze and respond. Traditional JEE Application Servers were generally implemented, expecting a client application to send an initial request and will then only process that request through, in most cases a significant amount of logic code, before it can respond back to the client. While these technologies enable, and will continue to provide benefit in more batch-orientated, less real-time approaches, newer lower latency and faster in-memory middleware products are now available.

Event-Driven (Architecture) based systems are intrinsically smarter, or better “equipped” to handle these types of situations, processing an entire business infrastructure as events that can be immediately interpreted and handled, spanning across the many departmental “silos” such as finance, manufacturing, and sales. These types of systems are also context aware and execute when they detect changes in the environment or business world, rather than occurring on a predefined (nightly) schedule or requiring someone to initiate an execution.

As the problems associated with Big Data grow substantially over the coming years in terms of the capture, management, and the ability to process the information within a tolerable amount of time, Event-Driven technologies (specifically Complex Event Processing) can provide Fast Data capabilities to apply a greater level of “intelligence” and decisioning to the originating data streams much closer to the “point of occurrence”.

So the benefits of an Event-Driven technology approach is to turn that proliferation of data into real-time knowledge by firstly representing events (things that happen from anywhere) in standard ways, providing an ability to factor out events, route events, filter events, aggregate events, and correlate events intelligently, so that in most cases fragmented events can be evolved into holistic, solid, understandable business events, enabling the business to better view, control, and adapt to situations relatively instantaneously.

Use case: A solution for customer problems

So how are Complex Event Processing Platforms used now to solve business problems? Certainly over the past few years, this technology is being used across most, if not all, of the different types of industries.

The financial services capital markets companies are using this technology for real-time algorithmic trading and real-time risk management types of solutions. As the stock markets stream their endless financial instrument data with values which can instantly fluctuate, there is an ever growing need to effectively handle this huge volume of information, understand its impact and potential risk, and then react as quickly as possible. The better the capability to evaluate and predict the consequences of the information, and the quicker the ability to respond to the results of this analysis, the more successful the business and the more money that can be made with less exposure to business risks and threats. This type of real-time trading information can be usually visualized using heat maps and scatter charts.

In the Electricity industry, customers are using the Complex Event Processing (CEP) platform for many new types of applications, which include Smart Meter, Smart Grid, and outage detection monitoring solutions. Sophisticated Demand Response (DR) solutions bring together system operators and the power generation companies, who contract with energy management and monitoring companies to provide energy usage load reduction services on demand. These technology companies that are using CEP-based applications contract with commercial and industrial businesses that are large consumers of energy, whom agree to curtail energy usage on demand. Streaming event devices are installed at client locations to measure energy usage and, in some cases, proactively control the load using continuous energy demand and usage data at minute or, even second, intervals. The generated profit revenue received from system operators is then passed back to the clients, relative to the number of associated load reduction dispatches.

Handling real-time events has a long history in the telecommunications industry, such as those generated by the various devices on the network, events from mobile phones, or perhaps streaming Call Detail Record (CDR) events indicating the time of calls made and whether some of these calls failed. Complex Event Processing platforms provide the technology for many new applications and solutions in this domain. As in other industries, Event-Driven platforms have a broad base of possible implementations. Some businesses have created powerful network management and monitoring solutions, which can detect hardware failure-related events continuing over certain time periods, or situations where equipment has not been issuing events for some time and in these circumstances alert messages are distributed and escalated.

In the context of an enterprise-level mobile telecommunication IT infrastructure, there are many different applications coming from many different suppliers. When the overall performance is not immediately meeting expectations, it’s not easy to identify which component is the offending issue in the supply chain. Therefore these next-generation management and monitoring applications (based on Complex Event Processing) provide the capabilities to show the complete, holistic “picture”, providing full visibility to the situation of a business through flexibility and fully integrated features, enabling agility for the infrastructure to react quickly to changing scenarios, and providing full operability enabled by a solution designed to meet business needs.

A very powerful capability of Complex Event Processing platforms which is being leveraged in the Transportation, Telecommunications, and Public Sector domain is real-time integrated spatial analysis.

A business can use this technology in applications where there is the need to monitor the movements of its assets and resources. Using, for example, GPS (global positioning systems) the movement patterns of someone, or something can be tracked in real time as it passes through boundary points (such as security checkpoints in an airport) to identify its route and, to some extent, predict where this person or object may subsequently move next. Also, this capability can be used to analyze a current position and its relationship to geofenced areas. A geofenced area being the definition of a geographical shape (polygon) defined or declared by a series of spatial coordinates.

When a resource gets near, inside, or enters and exits the geofenced area, various actions can be immediately performed, such as a warning message of an imminent exposure to a dangerous natural disaster, or offering a big discount on a second coffee at the person’s current location or soon to be, position, based on his or her current movement pattern.

First Responder emergency services solutions can use integrated spatial technologies to not only monitor a fire or hundreds of simultaneous fires, but also dynamically track the movement on the fire, affected by weather conditions (wind) or igniting hazardous materials. These types of systems can evaluate immediately the relevance, importance, and applicability of all of the related assets (fire engines, police vehicles, and so on) close to these areas. For example, if a fireman does not move in certain number of seconds when close to a fire, this could indicate a serious life threatening situation.

There are many other types of business solution implementations using Complex Event Processing platforms that range from online retail monitoring systems, real-time data center infrastructure management, fleet vehicle transportation monitoring, traffic flow monitoring with variable toll charging and speed control, oil fields and rig monitoring/automation, and a host of real-time sensing device opportunities, where these devices can monitor the environment inside shipping containers, or air pollution situations. The scope and different type of applications that can now benefit from using Complex Event Processing technologies are evolving just as quickly as the world is changing, with a growi ng need to predict and pre-empt and in, some cases, prevent situations from even happening.