EAI – The Broader Perspective
No one should have (or will) ever dared to build a ‘Single System’ which will take care of the entire business requirements of an enterprise. Instead, we build few (or many) systems,and each of them takes care of a set of functionalities in a single Line of Business (LOB). There is absolutely nothing wrong here, but the need of the hour is that these systems have to exchange information and interoperate in many new ways which have not been foreseen earlier. Business grows, enterprise boundaries expands and mergers and acquisition are all norms of the day. If IT cannot scale up with these volatile environments, the failure is not far.
Let me take a single, but not simple problem that today’s Businesses and IT face – Duplicate Data. By Duplicate Data we mean data related to a single entity stored in multiple systems and storage mechanisms, that too in multiple formats and multiple content. I will take the ‘Customer’ entity as an example so that I can borrow the ‘Single Customer View’ (SCV) jargon to explain the problem. We gather customer information while he makes a web order entry or when he raises a complaint against the product or service purchased or when we raise a marketing campaign for a new product to be introduced or … The list continues, and in each of these scenarios we make use of different systems to collect and store the same customer information. ‘Same Customer’ – is it same? Who can answer this question? Is there a Data Steward who can provide you with the SCV from amongst the many information silos existing in your Organization? To rephrase the question, does your organization at least have a ‘Single View of Truth’, if it doesn’t have a ‘Single Source of Truth’? Information locked away inside disparate, monolithic application silos has proven a stubborn obstacle in answering the queries business requires, impeding the opportunities of selling, not to mention cross-selling and up-selling. Yeah, it’s time to cleanse and distill each customer’s data into a single best-record view that can be used to improve source system data quality. For that, first we need to integrate the many source systems available. Today, companies are even acquiring just to get access to it’s invaluable Customer information! This is just one of the highlights of the importance of integration to control Information Entropy in the otherwise complicated IT landscape.
Figure 1. The ‘Single Customer View’ Dilemma
So Integration is not an end, but a means to end a full list of problems faced by enterprises today. We have been doing integration for many years. There exist many platforms, technologies and frameworks doing the same thing. Built around that, we have multiple Integration Architectures too, amongst which, the Point to Pont, Hub and Spoke, and the Message Bus are common. Figure 2 represents these integration topologies.
Figure 2. EAI Topologies
Let us now look at the salient features of these topologies to see if we are self-sufficient or need something more.
Point to Point
In Point to Point, we define integration solutions for a pair of applications. Thus, we have two end points to be integrated. We can build protocol and/or format adaptors/transformers at one or either end. This is the easiest way to integrate, as long as the volume of integration is low. We normally use technology specific APIs like FTP, IIOP, Remoting or batch interfaces to realize integration. The advantage is that between these two points, we have tight coupling, since both ends have knowledge about their peers. The downside is that if there are 6 nodes (systems) to be interconnected, we need at least 30 separate channels for both forward and reverse transport. So think of a mid-sized Enterprise with some 1000 systems to integrate!
Hub & Spoke
Hub And Spoke Architecture is also called as the Message Broker. It provides a centralized hub (Broker) to which all applications are connected. Each application connects with the central hub through lightweight connectors. The lightweight connectors facilitate application integration with minimum or no changes to the existing applications. Message Transformation and Routing takes place within the Hub. The major drawback of the Hub and Spoke Architecture is that if the Hub fails, the entire Integration topology fails.
Enterprise Message Bus
An Enterprise Message Bus provides a common communication infrastructure which acts as a platform-neutral and language-neutral adaptor between applications. This communication infrastructure may include a Message Router and/or Publish-Subscribe channels. So applications interact each other through the message bus with the help of Request-Response queues. Sometimes the applications have to use adapters that handle scenarios like invoking CICS transactions. Such adapters may provide connectivity between the applications and the message bus using proprietary bus APIs and application APIs.
Service Oriented Integration (SOI)
Service Oriented Architecture (SOA) provides us with a set of principles, patterns and practices, to provide and consume services which are orchestrated using open standards so as to remove single vendor lock-into provide an agile infrastructure where services range from business definition to technical implementation. In SOA, we no longer deal with single format and single protocol, instead we accept the fact that heterogeneity exists between applications. And our architecture still needs to ensure interoperability and thus information exchange.
To help us do integration in the SOA manner, we require a pluggable service infrastructure where providers, consumers, and middleware services can collaborate in the famous ‘Publish — Find — Bind’ triangle. So, similar to the integration topologies described above, we need a backbone upon which we can build SOA that can provide a collection of middleware services that provides integration capabilities. This is what we mean by Service Oriented Integration (SOI). Gartner originally identified Enterprise Service Bus (ESB) Architecture as a core component in the SOA landscape. ESB provides a technical framework to align your SOA based integration needs. In the rest of the article we will concentrate on ESB.
Enterprise Service Bus (ESB)
Roy Schutle from Gartner defines an ESB as:
“A Web-services-capable middleware infrastructure that supports intelligent program-to-program communication and mediates the relationships among loosely-coupled and uncoupled business components.”
In the ESB Architecture (Refer Figure 2), applications communicate through an SOA middleware backbone. The most distinguishing feature of the ESB Architecture is the distributed nature of the integration topology. This makes the ESB capabilities to spread out across the bus in a distributed fashion, thus avoiding any single point of failure. Scalability is achieved by distributing the capabilities into separately deployable service containers. Smart, intelligent connectors connect the applications to the Bus. Technical services like transformation, routing, security, etc. are provided internally by these connectors. The Bus federates services which are hosted locally or remotely, thus collaborating distributed capabilities. Many ESB solutions are based on Web Services Description Language (WSDL) technologies, and they use Extensible Markup Language (XML) formats for message translation and transformation. The best way to think about an ESB is to imagine the many features which we can provide to the message exchange at a mediation layer (the ESB layer), a few among them is listed below:
- Addressing & Routing
- Synchronous and Asynchronous style invocations
- Multiple Transport and protocol bindings
- Content transformation and translation
- Business Process Orchestration (BPM)
- Event processing
- Adapters to multiple platforms
Service Aggregation in ESB
ESB provides you the best ways of integrating services so that services are not only interoperable but also reusable in the form of aggregating in multiple ways and scenarios. This means, services can be mixed and matched to adapt to multiple protocols and consumer requirements. Let me explain you this concept, as we will explore more into this with the help of sample code too.
In code and component reuse, we try to reduce ‘copy and paste’ reuse and encourage inheritance, composition and instance pooling. Similar analogy exists in SOI where services are hosted and pooled for multiple clients through multiple transport channels, and ESB can do this in the best way integration world has ever seen. We call this as the notion of shared services. For example, if a financial organization provides a ‘credit history check service’, an ESB can facilitate reuse of this service by multiple business processes (like a Personal Loan approval process or a Home Mortgage approval process). So, once we create our ‘core services’, we can then arbitrarily compose these services in a declarative fashion so as to define and publish more and more composite services. Business Process Management (BPM) tools can be integrated over ESB to leverage service aggregation and service collaboration. This facilitates reuse of basic or core (or fine grained) services at Business Process level. So, granularity of services is important which will also decide the level of reusability. Coarse grained or composite services consume fine grained services. Applications that consume coarse-grained services are not exposed to the fine-grained services they use. Composite services can be assembled from coarse-grained as well as fine-grained services. To make the concept clear, let us take the example of provisioning a new VOIP (Voice Over IP) Service for a new Customer. This is a composite service which in turn calls multiple coarse grained services like ‘validateOrder’, ‘createOrVerifyCustomer’, ‘checkProductAvailability’, etc. Now, the createOrVerifyCustomer coarse grained service in turn call multiple fine grained services like ‘validateCustomer’, ‘createCustomer’, ‘createBillingAddress’, ‘createMailingAddress’, etc.
Figure 3. Service Composition
Java Business Integration (JBI)
Java Business Integration (JBI) provides a collaboration framework which provides standard interfaces for integration components and protocols to plug into, thus allowing the assembly of Service Oriented Integration (SOI) frameworks. JSR 208 is an extension of Java 2 Enterprise Edition (J2EE), but it is specific for Java Business Integration Service Provider Interfaces (SPI). SOA and SOI are the targets of JBI and hence it is built around Web Services Description Language (WSDL). The nerve of the JBI architecture is the NMR (Normalized Message Router). This is a bus through which messages flow in either directions from a source to a destination. You can listen to Ron Ten-Hove, the Co-spec lead for JSR 208 here and he writes more about JBI components in the PDF download titled JBI Components: Part 1. JBI provides the best available, open foundation for structuring applications by composition of services rather than modularized, structured code that we have been doing in traditional programming paradigms. A JBI compliant ESB implementation must support four different service invocations, leading to four corresponding Message Exchange Patterns (MEP):
- One-Way (In-Only MEP): Service Consumer issues a request to Service Provider. No error (fault) path is provided.
- Reliable One-Way (Robust In-Only MEP): Service Consumer issues a request to Service Provider. Provider may respond with a fault if it fails to process the request.
- Request-Response (In-Out MEP): Service Consumer issues a request to Service Provider, with expectation of response. Provider may respond with a fault if it fails to process request.
- Request Optional-Response (In Optional-Out MEP): Service Consumer issues a request to Service Provider, which may result in a response. Both Consumer and provider have the option of generating a fault in response to a message received during the interaction.