Ranger4 DevOps Blog

Breaking Environment Limitations with Integration Testing - Part 3

Posted by Andrew Cullum on Fri, Mar 27, 2015 @ 18:03 pm

Part 3 of 3

Rapid integration testing is a key to delivering frequent, high quality software. But, environment availability is often a limiting factor. In Part 1 we took a look at the limitations on environments, in part 2 we looked at techniques to resolve the bottleneck and this week we will conclude this 3 part blog by looking at a realistic scenario that brings the techniques together.

A realistic scenario that brings the techniques together

The fictitious example of a major system called Marketplace shows how to use the tools together. Marketplace is made up of many pieces.

  • 60 web services that are somewhat tightly coupled. Four teams each own 15 services.
  • Mainframe components contribute to 20% of transactions; the components rarely change and are owned by another team.
  • The front end website, in front of the services, is owned by the dot-com team.
  • Data feeds from 2 third parties are used (via web service). One is metered on transactions, the second is not.

The Marketplace release team had one large Integration Test Environment (INT) and a Performance Testing Environment (PERF). Each of the six teams now has a small test lab where they can test some of the components, but they cannot test any integrated scenarios. Integration testing is on the release schedule and release management has governed access to the INT and PERF environments.

Team level integration environments

To drive developer productivity, the Marketplace organization decided to ensure that each development team had their own integration test environments and the ability to spin up extra environments if they want to test extra code lines or if development spikes.

  • Environments as a Service (EaaS): The EaaS tooling spins up a copy of the application using the company's internal cloud. Only enough virtual machines for that dev team's services, and the latest working copy of the UI is used.
  • Service virtualization: Services from the other web service teams and the mainframe are virtualized. The metered third party service is virtualized, but the other feed is used live.
  • Environment reservation: For visibility, the reservation system in the Release Management tool knows which environment hosts work for which release. However, since there is no shortage of environments, team level environments are not reserved.

In the end, each development team has a number of small, cheap environments that use service virtualization heavily. Although they are able to test their components within the larger system, they are isolated from other teams. These other teams might break them or they worry that they will break the other teams' components. However, the heavy utilization of virtualization means that integration issues across services will not be found immediately.

Release level integration environments

System integration testing environments for each release are provisioned. These are mostly complete and only use minimal service virtualization. To enter this environment, changes must have successfully passed a robust set of automated tests in the team level environments.

  • Environments as a Service: EaaS can provision many environments that tend to be long-lived:
    • A test environment for patches to the current release
    • A heavily used environment for the upcoming release
    • An occasionally used one for big development efforts coming later
    The EaaS is mostly responsible for keeping the environments in line with the correct infrastructure.
  • Service virtualization: Only the mainframe and the metered web service are virtualized.
  • Environment reservation: These environments are big, and therefore expensive in terms of hardware. Environment reservation is used to watch for instances when extra environments might be needed and to minimize them if possible. Like the team environments, this system is primarily used to ensure that everyone uses the right environment for the right release.

Because of the heavy integration testing that occurs in the team level environments, changes that interfere with testing in this environment are very rare. For the manual testers, these environments are where they spend their time, and they benefit from a combination of high availability and always being on the latest good code.

Performance testing

Performance testing is largely unchanged and remains the largest environment. Service virtualization is available for both web services and the mainframe. For the mainframe and the metered service, virtualization is occasionally used during high transaction count testing. In other performance testing scenarios, stubs for both 3rd party web services are set to respond slowly to requests to validate the behavior of the application when the 3rd party providers are having trouble.

Final integration testing

The integration testing environment is used for final integration validation. Because it contains access to scarce resources like the live version of the metered service and the mainframe components, it is managed and scheduled. The environment reservation system is still used to allocate this to expected releases.

Part 1 & 2 are available here:

Part 1 - Strategies to improve environment availability

Part 2 -Techniques to resolve the bottleneck

Topics: Service Virtualization, IBM