Using a contract testing approach for CI/CD

To release enterprise software solutions, the continuous integration, continuous delivery/deployment (CI/CD) model has become very popular for software developers. By evolving from a monolithic design to a microservice architecture, an application has become a collection of loosely-coupled specific services, communicating through lightweight interfaces and protocols. By assigning work to these components, organizations can more effectively utilize more of a distributed, virtual workforce.

Although it proved successful for engineering, this approach may not be adopted by quality assurance (QA). Upwork, a leader in enterprise software development, has attempted to resolve this issue with a concept called contract testing.

Understanding the problem

Before microservices became popular, monolithic end-to-end automated tests (auto-tests) suffered from instability, causing too many false-positive failures. This quality deficiency has been attributed to two factors:

• Tests were too long and bulky, making them more likely to fail.
• Tests depended on the uptime of multiple software services, most of which were irrelevant to specific scenarios being tested.

Since CI/CD relies on rapid, quality releases, QA tests must verify an application’s end-to-end operation and individual microservices that may behave independently of each other. This calls for independent auto-tests.

Exploring a possible solution

A solution could be to create an isolated set of auto-tests limited to a specific service/page. These tests would only consider the functionality of that micro-service. There are distinct benefits of this simple approach:

• Test stability would most likely be resolved
• Redundant tests on other services wouldn’t be used
• Test containment could improve confidence in the release process

The process appears straightforward:

1. A pull request (PR) is issued from the master
2. Tests are run against the code in a sandbox
3. After passing, the validated code is merged back into the master

But this approach no longer tests the interactions between adjacent UI services. In other words, test results would be successful (green) until bugs are found later in the integrated environment.

‍

Contract testing comes to the rescue 

In CI/CD, a contract is an agreement between two adjacent services that define the user state at the transfer point between two UI services. This contract ensures that the output data of one UI service corresponds to the input data of another adjacent UI service. This relationship requires that tests are modified to validate contracted microservices’ conditions.

Let’s look at an example. QA has created an end-to-end auto-test validating two UI services (pages) representing a single test flow (assembly of test components of a system that validate business rules using a sequence of test scenarios). A logical approach would be to create two separate tests (one for each service).

Unfortunately, separate tests won’t guarantee that all interactions between the two services can be fully validated. So in addition to dividing the test in two, another verification is added, which becomes the contract. It checks that the user’s state at the first test’s exit and the beginning of the second test remains the same.

The results are in

There are significant benefits to a contract testing approach. Unmanageable, complex, monolithic tests can be replaced by several simpler, smaller, isolated micro-service tests.

Considering the relationship between micro-services, end-to-end test coverage wouldn’t have to be sacrificed either.

If, for example, an owner of an adjacent service changes something that impacts the user state in another adjacent service, that change would need to be reflected in both adjacent tests. Otherwise, the tests would fail (turn red) because of the mismatch of user states.

The importance of communication and collaboration

This contract testing approach does require an emphasis on communication between owners of adjacent services. The contracts joining these micro-services need to be discussed to create isolated tests. Specifically, QA must understand what checks and data preparations would be required to ensure test coverage among micro-services in a test flow.

It is also important to know of other contracts impacting your tests to avoid:

• Breaking any interactions between services
• Breaking other tests (especially due to contract changes)

This disciplined approach could have an even more significant benefit: collaboration. As a result, contract discussions may result in a more effective design among adjacent micro-services.

In summary

The adoption of isolated auto-tests with contracts for adjacent micro-services has shown distinct advantages at Upwork: 

• Quality of CI/CD has been enhanced to the adoption of testing micro-services
• Quicker response to changes and improved auto-test stability
• Improved maintenance and easier debugging of failed tests due to better test isolation
• Reduced need for QA involvement in the final release process—QA can focus on comprehensive testing during integration

‍