Ranger4 DevOps Blog

Breaking Environment Limitations with Integration Testing - Part 2

Posted by Andrew Cullum on Tue, Mar 17, 2015 @ 16:03 PM

Part 2 of 3

Rapid integration testing is a key to delivering frequent, high quality software. But, environment availability is often a limiting factor. In Part 1 we took a look at the limitations on environments, this week we will be looking at techniques to resolve the bottleneck.

Techniques to resolve the bottleneck

There are three techniques that smooth out the issues with integration test environments and promote their availability: Environment reservation, environments as a service, and service virtualization. Each technique solves different parts of the problem.

Environment reservation

The simplest strategy is to actively schedule and manage environments. This is usually the responsibility of a release manager. Integration environments are treated as precious resources and allocated to release testing based on release priority and distance from release date. Modern release management tooling, such as IBM UrbanCode Release, can provide formal environment tracking, scheduling, and conflict detection, but spreadsheets are still commonly used.

Benefits

Clearly delineating which releases can use an environment and when gives development and test teams needed predictability and maximizes the value derived from limited resources.

Limitations

While environment reservation helps ensure that limited resources are used well, it doesn't provide more environments or assist with environment inconsistencies.

Environments as a Service

The ability to request a test environment tailored to your application and have it provisioned and configured in a few minutes is extremely powerful. Cloud technologies (public or private) are combined with an environment pattern engine like UrbanCode Deploy with Patterns to spin up environments, configure them, and retire them when needed.

Benefits

Using environments as a service solution drastically reduces the labor of setting up test environments. Solutions that also update the configuration in place keep maintenance under control, while improving environment fidelity to production. Overall, teams can get the integration environments they need when they need them. Environments as a service technology should be a cornerstone of your integration testing strategy.

Limitations

Cheaply creating environments tends to encourage more environments. Hardware expenses can be a concern, especially with very large integration environments. Environments built on top of relatively cheap cloud resources help reduce setup costs. Because environments are both easy to create and retire, there is less inclination to hold onto seldom used environments. Instead, you can reclaim the resources when unused and spin the environment back up if needed.

Because this strategy is built on cloud and virtualization, many of the "precious" components do not fit neatly into the environments as a service strategy. They either need to be shared by multiple provisioned environments or virtualized.

Broken components from other teams can be a concern, but if each big team has their own integration environments and use automated deployments of only known good versions of other components, they can use environments as a service for isolation.

Virtualize services

Service virtualization replaces some of the components in the system with "stubs" or "mocks". Mocking is an approach that has been in use for a long time. Developers write a service that functions like a full service and tests against it. For example, if a stock quoting service provided by a third party charges per transaction, a developer might create a stand-in service that has the same API but always returns the same value for testing. Service virtualization tools like those in IBM Rational Test Workbench streamline the process of creating these stubs, managing where they are running, and how they perform.

Benefits

Service virtualization provides a clean way to handle the "precious" components. Stubs can stand in for components that charge per use or are unique (mainframes, expensive middleware, or appliances).

Stubs can also stand in for components from other teams. This has three key advantages:

  • A degree of isolation. You have an integration test environment that stays useful even when another team breaks their stuff.
  • In resource use. The server capacity required to run those other components isn't needed anymore.
  • A stub can stand in for components that aren't complete yet, giving you access to integrated testing scenarios earlier in the lifecycle.

Limitations

Tests are more relevant when you test against the real stuff rather than a stub. The same isolation capabilities that keep another team from breaking your work, also defer testing of the fully integrated system. Deferring testing has a cost because it slows feedback. Service virtualization also doesn't help with managing the environments of the components that are present.

In part 3 we will be looking at a realistic scenario that brings the techniques together.

Topics: Service Virtualization, IBM