If you’ve worked within the technology industry, you might be familiar with the infamous quote from AWS CTO, Werner Vogels; “Everything fails all the time”. This is an unfortunate inevitability within the realm of modern business applications that integrate using APIs. An application malfunctioning due to an issue with the set of APIs that it (application) consumes, is commonplace and has serious implications on the users of a system. What is unfortunate though, is the fact that the application itself almost always is not the root cause of the problem. This ripple effect can be largely minimised by applying a proper API testing strategy - something we have learned while driving API centric engagements for our customers.
API testing is a crucial element of establishing a robust API architecture and API platform. It plays a large role in ensuring that your APIs work as seamlessly as possible and continue to deliver value to your business and customers over time. The process of API testing itself is intensive and thorough and consists of many different elements and stages to cover all of the bases.
In this blog, we’ll explain the various levels of API testing that we frequently implement throughout various phases of the software development lifecycle (SDLC), communicate how each level helps to ensure an air-tight system and mention some API testing tools we commonly use. Want to know more about API Best Practices - get the white paper here.
What is API Testing and Why is it Important?
Application Programming Interfaces (APIs) are a way for various applications to communicate with each other using a consistent format, designed using best practices (see our popular white paper on API best practices) and documented using standards like Open API & RAML for RESTful services or a WSDL for SOAP services). These documents (often called API specifications) work like a contract. Engineers treat these contracts as a source of truth before they attempt to integrate systems.
The role of API testing is to verify the compliance of an API to its contract. API testing should be a part of every step of the process and happens hand in hand with development work on the API layer.
The consequences of failing to use API tests and mitigate risks earlier within the value chain are substantial, as the failures don’t just impact the business itself, but the customers too, once it surfaces to production.
For example, let's assume that a standard API was exposed by an order management solution, to the fulfillment center so that it (the fulfillment application) can get the order detail and initiate shipment. If the API breaks for some reason, the fulfilment centre would no longer be able to discover new orders to process and the problem would ripple downstream to customers who have paid for an order they haven’t received.
Broken APIs cause disruption/outage for API consumers and their customers. The final result is distrust, defamation of brand and time/revenue loss. This is the reason that it’s important to test your APIs against their contracts and in every environment possible, including external portals that your API is integrated with.
The API Testing Process
Following in the spirit of Martin Fowler’s test pyramid our API test strategy can be broken down into the following pyramid structure. The key take-away is you need more testing at the component level than at the broad scope omni-channel level.
Level 0: Component / Individual Resource Level Testing
Component testing involves the testing of all API for a resource (much like unit testing for a class). A few of the commonly followed validations performed at this level of testing are:
- Ensuring that the API accepts the request payload as documented in the API specification
- Ensuring that the response follows the right schema
- Ensuring that the API behaves in congruence with the HTTP verb it is associated with, for example, a GET request is idempotent and deletion of a resource is accomplished via DELETE verb for example
- Ensuring that the right error code is returned; Ensuring that the API is performant etc.
Essentially, you are testing the API at a certain point in time which usually happens hand in hand at the beginning and throughout the development work.
For component level API testing we use a number of different tools including Spring Boot Testing (embedded in Spring MVC Framework), JUnit, REST-Assured.
Level 1: Contract Testing
While Component and Scenario level testing can ensure that a specific application resource behaves correctly, as individual and as part of a whole, they are not good at verifying if multiple systems integrate among themselves as desired.
However, when designing large systems, often two or more vendors are developing parts of the whole system independently and aren’t collaborating from day one. In such scenarios establishing and complying over an API specification or contract becomes important. Given that API specifications are static documentation it is difficult for consumer systems to track if any change in the producer system’s API breaks their own system or not. To overcome such scenarios, the producer system publishes runnable mocks/stubs for the API they build. The consumer systems use those mocks/stubs to write test cases against it. This way if ever producer changes API (i.e. the contract) - the consumer test cases break giving an early indication of possible issue.
The process of publication of mocks/stubs thus becomes continuous and guarantees that the producer is aware of the ripple effect of a change and consumer is well aware of anything they have to change on their part.
This completely eliminates the need for all services to co-exist at the same point in time and detects any issues around integration early in the process.
For API contract testing we use tools such as Pact and Spring Cloud Contract.
Level 2: Scenario Testing
APIs are individual building blocks, used to build a solution that helps to achieve business goals. In many instances, there are cases where multiple APIs are combined. Scenario testing is the process of mimicking the way external applications will use the APIs together in different ‘scenarios’ to achieve the outcome.
Typically this might look like:
- Step 1: the customer comes to your portal and selects one or more products to purchase
- Step 2: based on the product they have chosen, a promotional offer is shown back
- Step 3: final order is being created
- Step 4: payment process runs
- Step 5: invoices are generated and sent to ERP
Scenario testing ensures that the set of APIs that realises the above sequence end to end - are tested together in entirety to validate the end to end business scenario. During scenario testing we leverage a range of API testing tools including Postman, Newman, Cucumber - JVM.
Level 3: Performance Testing
This step comes much later in the API development cycle and requires the testing of performance and vulnerabilities to meet performance metrics. Performance testing ensures that the APIs are performing as per service level agreements.
For example, APIs ideally should not take more than 1 - 2 seconds as often there can be issues with timeouts if APIs don’t return data in a certain period of time. In this case, developers need to follow certain design best practices to ensure that the API performs the process faster and returns a small - finer grained payload instead of a large one (which typically takes more time to get marshalled into a response).
In some cases, certain performance metrics (such as a high payload vs. speed) counter one another. In this instance, it’s important to determine which performance metrics are to be prioritised.
- Response time
- Error rates
- Number of concurrent API users supported during peak load etc.
All of the above need to be tested at the performance testing stage. We use JMeter for performance testing.
If you've found some performance issues with your API response time, a implementing an API response cache can significantly improve API performance,
Level 4: Security Testing
Security testing is one of the most important levels of testing, where authentication and authorisation requirement for APIs are validated.
About 1% of all the APIs that an application supports, are unguarded (for example, when you change a password on Facebook you’re not authenticated and therefore the API is not guarded). However the rest all require the invoker to authenticate itself before processing the request.
In case of application, often multiple users are supported who have different roles. That way the API typically needs to validate authority of the role of the user, in addition to authentication before processing requests. These authorization mechanisms involve a detailed matrix of role vs. authority - something quite hard for a human to test diligently. Security testing of APIs shines in this space.
Security testing covers:
- Who has access to what (for example, role-based access control)
- How many times people can request access
- Measures to ensure there is no ‘eavesdropping’ or interception of data
To perform security checks, developers use an API testing method, which essentially can impersonate a person logging in or attempting to access data and reports back system breaks which can then be manually resolved. We use Postman for API security testing.
Level 5: Omni-Channel Testing
API fuels omni - channel strategy for organisations. At its core, the goal of the strategy is to serve users with consistent and personalised experiences across all digital touch points. These touch points have significantly different requirements from an API, though the underlying key information they seek may be the same for all cases.
API developers need to constantly validate that required varieties (as required by various channels) are served by the API, to ensure frictionless integration between the channel and API.
Essential considerations while developing API test cases for different channels include:
- Is the API serving required functionality correctly given the specific channel (ie human interaction is different from bot interaction for a particular function)? This should consider API usability testing for each specific channel.
- Is the API following prescribed security practices for the specific channel?
Is the API meeting the performance characteristics of the channel (ie acceptable response times may vary) ?
Omni channel strategy has profound implications on building brand loyalty. Omni channel testing ensures that organisations are able to find any issues with the underlying implementation sooner rather than later. We use a variety of tools for omni-channel testing including Selenium, Botium and Postman.
Want to learn more about API standards and governance?
Our blog "The important role of API Standards" talks about API standards and governance but also highlights other important considerations to take into account from an organisational perspective when rolling out an API program.
Whatever stage you're at, we make it easy for you to leverage APIs to deliver on your digital initiatives. Book a time to chat with us today.