The Integration strategy provides the technical strategy that will be used to connect service providers and partners that are part of the quickly growing digital business ecosystem.
Strategic goals and drivers include:
- Technology and product implementation flexibility
- Development community support and ease of use
- Connect internal and external service providers to internal and external service consumers
- Partner management
- Centralized policy definition and enforcement with minimal impact to service providers and consumers
- Security and protection for both services being exposed to consumers (outside – in) and services being consumed from providers (inside – out)
- Insight, monitoring and advanced analytics
- Fault tolerant, high availability, multi data center deployment
- Continuous integration / continuous deployment automation
- Conformance with enterprise architecture and technology standards
For most companies the following integration strategy provides a good starting point to enable their digital business efforts.
Application and Integration Architecture
Our digital business technology architecture will be based on modern web oriented architectures.
We will use microservices segmented into business capability driven domains to build our applications.
All new services will be implemented as REST interfaces with application/json content types.
Swagger will be used to document the REST contracts.
The REST URI is resource based, has only nouns and no verbs, uses plural resource names, and includes resource IDs. Content is transmitted to the service in the request body, which includes query strings and JSON.
Services will use standard HTTP methods.
Services will use standard HTTP mime types for request and response content.
Services will use standard HTTP status codes to communicate success and errors to the client.
Composite services that require data and assets from multiple domains and providers can be implemented using a business process management framework or as a composite microservice using the microservice development framework. See the “Composite Service Implementation Framework Selection” section for more details.
Application and integration architecture has evolved over time. It started as data integration using batch jobs and ETL, transitioned to real time enterprise application integration using messaging middleware with interfaces and adapters, and then transitioned to SOA using web services. Now the industry is moving from service oriented architectures to web oriented architectures based on microservices and web APIs. This evolution is being driven by the same things driving the API economy: mobile, cloud, and the internet of things (IoT).
The Open Group defines an individual microservice as “a service that is implemented with a single purpose, that is self-contained, and that is independent of other instances and services.” Adrian Cockcroft defines a microservices architecture as a service-oriented architecture composed of loosely coupled elements that have bounded contexts. Domain driven design deals with large models by dividing them into different bounded contexts and being explicit about their interrelationships.
Microservices are used to decompose monolithic applications into independent functional components. These components can then be used to assemble applications in an agile and flexible manner. Microservices are more of an aggregation of several contemporary practices and patterns than a technology or a standard. Martin Fowler identified the following key characteristics of microservices:
- Componentization via services
- Organized around business capabilities
- Products not Projects
- Smart endpoints and dumb pipes
- Decentralized Governance
- Decentralized Data Management
- Infrastructure Automation
- Design for failure
- Evolutionary Design
Once applications are decomposed into microservices integration between them is simply a matter of one application calling a microservice exposed by the other application.
A common theme in “Domain-driven Design: Tackling Complexity in the Heart of Software” by Eric Evans (2004), “Software Fortresses: Modeling Enterprise Architectures” by Roger Sessions (2003), and “Designing IT for business” by Jurgen Laartz, et.al. (2003) is the decomposition of large IT systems, applications, teams, and models into domains to reduce complexity and gain business capability alignment. In microservices “2) organized around business capabilities” also strongly promotes the grouping of microservices into business capability based domains. Logically, a business capability based domain approximates a single product. Examples of business capability based domains are member, provider, payer, utilization management, etc.
Applications within a domain are business capability product based. They are managed and governed by a single DevOps team. The microservices within the domain are decomposed and structured to best serve the needs of that domain. Although microservices are “independent of other instances and services” they are coupled within a domain because they are part of the same product.
However, applications and microservices must remain loosely coupled across domains. An application or microservice in one domain should never directly call a microservice in another domain. Each domain has its own operations and governance, and because of this, disrupting changes may occur to services in one domain without prior notification to consumers in another domain if the services are directly linked.
Digital Business and APIs
In the modern digital economy APIs expose internal data and business logic over a Web oriented architecture to API consumers.
Kristin R. Moyer, vice president and distinguished analyst at Gartner, says that “the API economy is an enabler for turning a business or organization into a platform.” Christy Pettey, also from Gartner, further explains the API economy as “a set of business models and channels based on secure access of functionality and exchange of data. APIs make it easier to integrate and connect people, places, systems, data, things and algorithms, create new user experiences, share data and information, authenticate people and things, enable transactions and algorithms, leverage third-party algorithms, and create new product/services and business models.”
Analyst Craig Burton postulated the following “Five Axioms of the API Economy”:
- Everything and everyone will be API enabled.
- APIs are core to every cloud, social and mobile computing strategy.
- APIs are an economic imperative.
- Organizations must provide their core competence through APIs.
- Organizations must consume core competences of others through APIs.
Internet business models are also a major driver of the digital business and the API economy. There are a lot of internet business models but they fall into three primary categories. They are public, partner, and internal. In the public model APIs are consumed by external unknown and loosely affiliated developers to build custom interfaces to your business data and assets. In the partner model APIs are consumed by business partners, including suppliers, providers, and resellers to extend market reach and reduce costs through tighter partner integration. In the internal model APIs are consumed by internal developers within your organization to gain speed and agility.
As you can see, APIs provide opportunity for revenue growth, brand expansion, market expansion, new channels, cost reduction, and improved efficiency. However, when making business data and assets available through APIs new risks, such as security, privacy, and trust, also arise.
API Management and Service Gateway
We will use an API service gateway to expose sets of services from multiple internal domains and external providers as a coherent API to internal and external consumers.
The API services will be exposed as REST interfaces with application/json content types.
The API service gateway will consume application/xml content types from the event bus and publish application/xml content types to the event bus. When combined with JSON – XML content transformation and HTTP/REST transport bridging this will allow the publishing and subscribing of events using the service gateway API.
Avoid complex content transformations in the API service gateway. Complex transformations usually violate the “smart end-points and dumb pipes” principle of microservices.
All externally originating service calls will go through the API service gateway. All internal calls to external providers will go through the API service gateway. All internal cross-domain service calls will go through the API service gateway. Service calls within a domain aren’t required to go through the API service gateway.
All internally originating service calls that are targeted to external partner services will go through the API service gateway. This is also referred to as reverse gateway functionality.
Authentication and authorization security policies will be enforced by the API service gateway.
Throttling policies, for both incoming and outgoing service calls, will be enforced by the API service gateway.
API usage details and key performance indicators will be monitored using advanced analytics.
This decision is based on the following considerations:
- The granularity of APIs provided by microservices is often different than what a client needs. Microservices typically provide fine-grained APIs, which means that clients need to interact with multiple services. The service exposed by the API may need to get data from and write data to multiple services at the same time.
- Different clients need different data. For example, the desktop browser version of a product details page desktop is typically more elaborate then the mobile version. The data format may change between services (transformation – the ability to manipulate data within the mediation exchange).
- Network performance is different for different types of clients. For example, a mobile network is typically much slower and has much higher latency than a non-mobile network. And, of course, any WAN is much slower than a LAN. This means that a native mobile client uses a network that has very difference performance characteristics than a LAN used by a server-side web application. The server-side web application can make multiple requests to backend services without impacting the user experience where as a mobile client can only make a few.
- The number of service instances and their locations (host+port) changes dynamically
- Partitioning into services can change over time and should be hidden from clients
- Services might use a diverse set of protocols, some of which might not be web friendly (transport bridging – the ability to change the service transport)
API service gateways are designed to provide security, policy enforcement, and throttling to protect downstream systems. They are designed to provide superior security capabilities, without impacting on performance. They also decouple the API from the service implementation, using concepts such as service end-point virtualization and mediation, to provide flexibility and agility in both the API and the implementation.
Positions Service Discovery and Routing
The API service gateway will expose the API services at well-known and consistent URIs.
The API service gateway will serve as the service registry for the target services using a third-party registration pattern.
The API service gateway will route the API services that are exposed at the well-known URI to the registered target services.
There are two (2) main patterns for service discovery and routing. They are client-side and server-side.
In the client-side service discovery pattern the client is responsible for determining the network locations of available service instances and load balancing requests across them. The client queries a service registry, which is a database of available service instances. The client then uses a load‑balancing algorithm to select one of the available service instances and makes a request.
In the server-side service discovery pattern the client makes a request to a service via a service gateway/router/load balancer. The service gateway queries the service registry and routes each request to an available service instance. The API service gateway hides the routing fluctuations and complexities between a client application and the microservices that power it. As with client‑side discovery, service instances are registered and deregistered with the service registry. There are two (2) main patterns for service registry. Services can either self-register their availability or a third-party can handle the registration.
This decision is based on the following considerations:
- client application developers expect to be able to retrieve an API at a consistent URI
- prevent external application developers from discovering details of internal services
- one-API for all internal and external developers
- loosely couple client and target service
API Service Versioning
The REST services exposed by the API will use the HTTP accept header to communicate the content version. This is also referred to as content negotiation. If there is no version in the accept header then assume the latest version, this allows the browsing of the latest version of the API.
Resources with the same name that are part of different industry standards may be different resources. In this case it is valid to include the version of the standard in the URI to differentiate the two resources.
The primary alternative to these recommendations is the URI approach. The URI approach puts the version number in the URI. The URI approach violates the original intent behind REST and HATEOAS. The primary justification for using the URI approach is that it is easier to browse APIs using a web browser without the accept header. However, this can also be achieved by assuming the latest version if the accept header isn’t provided.
Microservice Development Framework
We will adopt a Microservice Architecture (MSA) for our development practices and patterns.
We will use a distributed microservice implementation platform which includes Enterprise Service Bus (ESB) capabilities.
Distributed Enterprise Service Buses (ESB) are used when mission critical core business processes have to be quickly integrated across one or more enterprises and provide high availability, reliability, and performance. (See Enterprise Service Bus Discussion below.)
Modern enterprise service buses are distributed, cloud and container capable, and use the same contemporary practices and patterns as microservice architectures. In addition, they provide distributed code-free integration and orchestration across multiple COTS systems and technologies using web native standards and technology-specific plug-ins. Their support for distributed and event-based messaging provides reliability and availability when integrating with distributed, occasionally disconnected, systems and services.
The Enterprise Service Bus is an architectural pattern that includes service design and implementation patterns, mediation/transformation patterns, and messaging patterns. It supports virtualization and management of service interactions between communicating participants. It acts as an intermediary for connecting services between service providers and requesters in a service-oriented architecture. It is a flexible connectivity framework that facilitates reliable and secure system integration while reducing the number, size, and complexity of application interfaces. Instead of interacting directly, services communicate through this service connectivity framework.
Enterprise Service Buses (ESB) have had multiple architectures over time and this has resulted in confusion. ESBs started out as Enterprise Application Integration (EAI) brokers and had an application server/central hub architecture. The ESB not only managed the service components, it also managed the communication between the service components. Within the communication functionality the platform provided mediation capabilities such as message routing, choreography, and transformations. This created dependencies and coupling between application service components and the ESB platform. Even though the system was decomposed into multiple service components, most implementations ended up acting like a monolithic architecture with all of the change management issues because of the coupling with and reliance on the large ESB platform. The effort involved with installing, configuring, and managing the ESB platform required dedicated middleware IT teams and product expertise. These platforms also couldn’t innovate fast enough and became “heavy” compared to emerging architectures.
ESBs next moved a distributed agent architecture without the need for a central hub. These architectures have adopted microservice architecture patterns and practices. They also moved the mediation capabilities from the communication pipes to within the agents to match the microservice approach of “smart end-points and dumb pipes.” Each of the agents can run independently of each other or coordinate through a shared event and data store. These agents expose both application service endpoints and management endpoints for distributed access and control. The largest downside to these products is that their management framework is proprietary.
Today, modern ESBs are not only distributed but have also been decoupled from the management platform. They have kept the best of the ESB concept while removing the proprietary bits that have caused operational and change management issues for IT infrastructure teams and data centers. Instead of proprietary management platforms they now consist of a container friendly lightweight runtimes that can be managed via systems such as Kubernetes.
Composite Service Implementation Framework Selection
This section provides guidance on selecting the most appropriate implementation framework for a composite service that requires data and assets from multiple domains and providers.
Things that need to be considered to make a selection:
- Are their human users involved in the orchestration?
- What is the purpose of the composite service?
- Who owns the composite service? Is it a single business department?
- Does the service capability map to a single domain, even though it may interface with multiple domains and external providers?
- Are any of the providers commercial off the shelf systems, industry standard products, or industry standard technologies?
- Are there different transports, protocols, or formats being used across the multiple providers?
- Will the service be asynchronous and long running?
Business Process Management Framework
If the orchestration requires input from human users during its execution (as opposed to at the start or end of the orchestration) then this would be the most appropriate implementation solution. This also includes front-end development frameworks and platforms, such as Salesforce.com.
Microservice Development Framework
If the composite service doesn’t require input from human users during its execution and will be reused by multiple consumers then this would be the most appropriate implementation solution. It may also be the preferred solution if it requires integration capabilities that are outside the scope of the business process management framework.
Most composite services that call cross-domain services align with a primary domain and can be packaged with that domain. However, in some cases, a composite application domain may be required to reduce cross-domain package coupling.
API Service Gateway
Because of the limitations listed below we do not recommend using the API service gateway to implement composite services.
As discussed in the previous section on API management and service gateways, in addition to providing security and protection for the underlying implementation services, API service gateways can also perform the following tasks:
- Service end-point virtualization (location transparency)
- Service composition (routing and orchestration)
- Protocol mediation
- Message mediation
However, there are technical and practical limitations on how much should be done within the API service gateway.
- Message transformation must not violate the “smart endpoints and dumb pipes” rule of microservices. This means that while message model changes are okay it isn’t okay to change message content.
- API Gateways can be used for simple composite services, but do not support Business Process Execution Language, and are not suitable for long duration composite services.
- API Gateways are stateless and cannot maintain transaction state.