Testing in Agile

Exactpro is happy and honoured to help our clients on their innovation journey. In recent years, many technology transformation initiatives within financial market infrastructures have been undertaken based on Agile methodologies. 

In this document, we outline the Exactpro team’s view on software testing services provided in an Agile environment.

What is Agile?

According to the Agile Manifesto, one of the values of Agile approach is responsiveness to change. The speed of change is crucial for any innovation project, as is the ability to continuously maintain it. The highest speed of development, aligned with the highest quality of resulting software platforms, is the cornerstone of our customers’ success and, therefore, a priority for Exactpro.

Is quality everyone’s responsibility?

Some organisations that pride themselves on adopting Agile believe the stereotype that the responsibility for product quality is shared among different team members. However, if you are in charge of a regulated organisation, the responsibility is yours – a principle strongly advocated by regulatory bodies, business leaders, and industry practitioners alike.

‘Management and control’ is a central idea of the FCA Handbook Principles (Principle 3). SYSC explicitly states that, even if regulated firms outsource their critical or important operational functions, they “remain fully responsible for discharging all of its obligations under the regulatory system”. In particular, the responsibility issues are covered in rules SYSC 8.1.1 R and SYSC 8.1.6 R, as well as in guidance SYSC 3.2.4 G.

As argued by recognised software testing practitioners Cem Kaner and Michael Bolton (Bolton 2010), the responsibility of quality assurance always lies with the management and the CEO, since it is they – and certainly not the testers – who have the authority to make decisions about quality. For systemically important regulated entities, it is not just an option. To do their job right, those in charge must execute that responsibility and make informed decisions based on independent information obtained from testing activities. 

Software engineers and software testers also have the responsibility – to do their job, development and testing, respectively, and communicate reliable and truthful information ‘up the chain of command’, as it was put in Extreme Ownership (Willink, Babin 2015, 2017). Software testing specialists are responsible for the quality of the information they provide, whereas the responsibility for the quality of the system lies with the management. Thus, it is the obligation of those in charge (1) to implement processes that ensure a continuous flow of reliable information, and (2) critically evaluate it and act upon it. If this obligation is not fulfilled, the project will fail, no matter the effort and contribution of the rest of the team.

Achieving the speed of change

The speed of development depends on the speed of the flow of information about the software being developed. Agile is not about following certain procedures, it is about being able to respond to change. As the change is a product of decision-making capability, it relies heavily on information. In true Agile, this information should be obtained as soon as possible, on a continuous basis.

What is the source of this information? The information about software under development is obtained through software testing. 

To achieve the highest quality of the end product, software development and software testing processes should run in parallel and be aligned in terms of their speed.

However, from the perspective of the information flow within the two, software development and software testing represent two antiparallel flows: 1) from knowledge to end product in software development and 2) from end product to knowledge about it – in software testing.

Testing in Agile. Achieving speed of change

The Antiparallel Structures of Software Testing and Software Development,
the diagram is inspired by (Drexler 2013)

Hence, to align the parallel processes of software development and software testing, the antiparallel flows of information within the two should be aligned as well.

Testing in Agile. Achieving speed of change

Software development and software testing: 
parallel process flows vs. antiparallel flows of information

Agility is achievable through three important actionable steps: investing in scalable platforms (both software and hardware), investing in people and, most importantly, investing in improving the development and testing processes.

Testing in Agile. Achieving speed of change

Optimising the feedback loop

The success of development depends on the speed of the quality assessment of the code: the information should be provided as soon as possible, delivered continuously throughout the development process.

A case study by Hewlett-Packard is an example of an approach to optimise the feedback loop in the Agile development process (Gruver, Young 2012). The transformation project at HP proved successful, with one of the factors contributing to this success being the continuous multi-level rapid feedback-oriented testing approach.

HP dramatically improved their development practice by ensuring that valuable information about software is communicated rapidly and in an efficient manner.

Testing in Agile. Optimizing the feedback loop

FutureSmart FW testing levels, from (Gruver et al. 2014)

Continuous feedback loop implemented through large-scale automated testing worked effectively on 5 levels of testing. Scope and feedback time increased from level to level, ranging from level L0, where smaller areas were committed and tested every 15 minutes, to level L4 covering full end-to-end regression testing performed on a daily basis. With this approach, different flows of information were received continuously: at the same time, but at different speeds.

Investing in platforms

In Agile development, one version of the system is rapidly superseded by another. Thus, it is important to get extensive information on its state in a timely manner.

Let’s consider an example, where we have a software system of version A on day 1. Let’s assume that it takes three days to explore the piece of software completely and get sufficient information about it. During testing, information about the version A is gradually accumulated, and on day 2 we have some (but not all) knowledge about it. At the same time, a new version of the system, version B, is released. What do we do now? If we continue to explore version A, we will get information about the piece of software that is no longer most recent. However, if we switch to testing version B instead, the whole information gathering process is to be started from scratch.

The best approach is, of course, not to choose between the versions A and B but explore both of them simultaneously, ensuring that the latest set of tests covers the scope not covered by preceding testing. This approach is only achievable when there is enough hardware.

In Agile development, having just one test environment is infeasible: to technically implement the continuous feedback loop on rapidly changing software, one needs multiple test environments and, therefore, lots of reliable hardware as the foundation. 

Investing in people

With a sufficient amount of hardware in place, it is possible to set up continuous checks against the system. Those checks generate a very important information flow – the flow that, once processed and analysed, is communicated to the stakeholders and used for decision making.

The checks per se are not enough, the key is to extract valuable information from the test execution data flow. Failure triage always involves manual testing – aimed at exploring the system under test and studying what went wrong. This kind of testing is called exploratory testing, and it is an important part of the feedback loop. 

A cognitive flow generated by manual exploratory testing and test data analysis is a crucial element of the ability to respond to change, which is at the core of Agile practice. That is why it is necessary to ensure this cognitive flow by investing in the human brain – the technical talent capable of understanding the system under test and setting up the balanced development/testing process.

Testing in an Agile environment – Case studies

The following case studies illustrate the points explained above and demonstrate the importance of an optimised feedback loop, sufficient hardware resources and qualified technical talent in Agile practice.

Exactpro case study 1:
Large-scale testing in a major post-trade infrastructure

Exactpro adopted a similar approach to the HP example described above, and the closest representation would be on an engagement testing one of the world’s largest CCPs (systemically important in more than one jurisdiction).

Testing was organised according to a layered model supported by different-level test libraries providing a continuous information flow about the system under test in different time intervals. 

At the first layer, there were several small test libraries of limited scope. Consisting of about a hundred test cases, such a library was continuously run against the system under test, with execution times measured in seconds and immediate feedback on its state.

The second layer was a bigger library with wider functional scope: it was run in parallel with the tests of the first layer and it took about 15 minutes to execute all the tests. As this test library contained more complex tests, it explored the system under test on a deeper level and provided more information over longer time intervals.

Testing in Agile. Large-scale testing in a major post-trade infrastructure

Finally, the third layer was represented by an end-to-end full-scope test library that generated detailed multi-aspect data flow once a day. It was the largest collection of tests taking 24 hours to execute – the “Big Button” automated test library. Though it provided the most reliable and detailed information about the system under test, it was crucial to get some of this information faster, on a continuous basis, which was achieved through smaller libraries on the first two layers, executed concurrently.

Bottom line: It is unreasonable to have just one level of testing when everything is done in a single commit. A better approach is to explore the system on multiple levels in parallel and provide feedback constantly, through several information flows delivered at different speeds.

Exactpro case study 2:
Large-scale testing for a DLT network

Supporting a major provider of distributed ledger technology (DLT) for financial institutions, Exactpro was engaged in software testing of a distributed network consisting of hundreds of nodes. The technology was developed in accordance with the principles of Agile development.

The Exactpro testing team was able to align its work with the Agile development process by leveraging th2, the company’s framework for automation in testing. 

th2 allows for multiple levels of test execution: on top of unit tests and component testing, it also provides capabilities to set up a large test environment for extensive functional coverage and pervasive load testing. The test library created by the team was run as part of an automated continuous integration, build, and test process. The test execution process involved hundreds of nodes generating huge volumes of load representing transactions all across the DLT network.

Bottom line: In this use case, large-scale pervasive load testing was achieved through a high-performance test environment as well as through test automation applied on different levels of the Agile development life cycle.

Testing in Agile. Large-scale testing for a DLT network

The common characteristics of the two examples above are their scale and extreme complexity: the complexity of functionality (operational logic of a post-trade infrastructure) in Case study 1 and the complexity of a multi-node distributed network in Case study 2. 

Both examples required the team to create comprehensive test libraries that were run against systems under test and to set up multiple test environments backed by sufficient hardware resources for large-scale functional and non-functional testing.

Exactpro case study 3:
Behaviour-driven development for a complex trading system

Quite often, Agile projects follow a set of practices commonly associated with the methodology. Behaviour-driven development (BDD) is one of such practical approaches. Although it works best as part of the software development effort, BDD is often viewed as a go-to approach for software testing activities in Agile environments.

However, when used for testing per se (as opposed to unit tests being the part of rigorous software development practice), BDD is but a number of simplifications not suitable for large-scale Agile projects. Such simplifications just do not provide a capability to create information flows that can be valuable to stakeholders to make informed decisions.

This example describes a testing approach implemented in the trading auctions area. The initial idea was to conduct testing in line with the organisation’s BDD practice. The client planned to develop all the functionality first, and only after the main development was finished, to do the testing based on automation using Gherkin. All bug fixes resulting from testing activities were to be included into one separate release.

The approach would probably work for a relatively simple software system. However, the system being built was a new trading auction model which was inherently complex: with a continuous data flow through multiple endpoints and interfaces, it could hardly be expected to exhibit deterministic behaviour with foreseeable outcomes. 

By the time Exactpro got involved in the project, the in-house team had implemented Gherkin scripts as part of the BDD approach adopted for testing. However, all the complex business logic was hidden in the statements and steps within the Gherkin Given-When-Then statements, which oversimplified/overgeneralised test scenarios and limited the depth of checks and input data permutations. While the scripts performed well at the confirmation task, they did not find defects. This can be explained by the fact that the BDD models were created by the development team and, very likely, inherited the problems of the software developed by the same team, with the same defects making their way from code into the tests.

To improve acceptance testing, the Exactpro team relied on a model-based testing approach instead. The team was able to create models replicating the system logic and generating a set of outcomes for each set of input data. These models provided an extensive coverage of the system functionality and were able to identify a number of defects, including a critical error in the system’s matching engine.

Bottom line: Model-based testing proves to be better aligned with Agile principles: models allow for testing the complex behaviour of software systems, they can be built in parallel with the development process and provide quick feedback to support Agile practices. Unlike BDD, which is only suitable for limited checks, model-based testing is more focused on delivering actionable information needed for decision making.

Development and testing in Agile: The perfect balance

Development and testing, even when implemented as two parallel process flows, can still demonstrate a certain imbalance jeopardising their value for the Agile practice.

There are two types of deviations from a perfectly aligned Development-Testing tandem.

One scenario is characterised by imbalance between software complexity and the sophistication level of the testing approach and tools, when a relatively simple piece of software is tested with a powerful test automation framework by a strong team of software testing professionals. In this case, testing activities will generate massive flows of information about software defects, so that the developers will not have the capacity to process all this data and fix the defects without impeding development rate. 

Another scenario reflects the opposite situation, when development is much faster and more efficient than the testing capabilities within a project. In this case, the speed of development will limit the testers to very small areas and they will not be capable of covering a sufficient testing scope that is needed to provide reliable information about the system.

Development and testing should be organised in such a way that they are run in parallel and are perfectly aligned in terms of their capacity and speed. Thus, to accelerate development, you need to invest in testing.

How does Exactpro fit into the Agile practice?

To identify the areas Exactpro can be instrumental in, the best way would be to look at the four Agile Testing quadrants, first introduced by Brian Marick, discussed in detail in the famous book ‘Agile Testing: A Practical Guide for Testers and Agile Teams’ by Lisa Crispin and Janet Gregory (Crispin, Gregory 2009), and further developed in the sequel ‘More Agile Testing: Learning Journeys for the Whole Team’ (Crispin, Gregory 2014). 

Agile Quadrant

Agile testing quadrants, from (Gregory, Crispin 2014)

Q1 – Technology-facing checks that are aimed at internal evaluation are part of the software development and should be performed by the development team. Unit-level and component-level tests provide the earliest feedback and are considered the core of the Agile software development practice. 

Q2 – The checks in this quadrant are more business-oriented. While still focusing on the development process, they provide important information on the system’s behaviour from the customer’s point of view and are frequently based on approaches such as BDD and specification by Example. 

The widespread industry practice is to incorporate Q1 and Q2 checks in the CI/CD pipeline, and some teams mistakenly consider this process to be a software testing exercise. 

Software is a collection of instructions and data that tell a computer how to work. The computer (hardware) is able to execute the software. Each instruction causes the computer to carry out an operation – moving data, carrying out a computation, altering the control flow of instructions or manipulating hardware. CI/CD falls perfectly under this definition. The CI/CD pipeline itself represents the code – operational software running on allocated hardware. At Exactpro, we Build Software to Test Software. The Exactpro service helps our clients to deliver their software, including the parts of it related to automated checks, delivery and deployment.

Two quadrants on the left correspond to software development and related activities can benefit from Exactpro support. Two quadrants on the right correspond to software testing and are at the core of the Exactpro business.

Q3 – Exploratory and user acceptance tests rely immensely on human expertise, which we consider to be at the core of the software testing services provided by Exactpro. This quadrant reflects Exactpro’s vision of end-to-end software testing based on the extensive business knowledge of the system under test. Q3 is built around people – one of the crucial components underpinning the Agile approach.

Q4 – This quadrant focused on the technology side and aimed at critique of the software product is associated with tools that enable pervasive automation in software testing. This activity represents the core of Exactpro’s expertise due to our strong track record of handling large-scale testing at the confluence of functional and non-functional requirements. This capability depends on yet another element of a successful Agile implementation – the platforms.


The structure of a typical Exactpro project team corresponds to the idea of a perfect alignment between the development and testing processes. This structure is best described through the analogy of a wheel rotating in parallel to the development flow, but providing the information service in the form of a continuous feedback loop.

Testing in Agile. How does Exactpro fit into the Agile practice?

The team is created in such a way that it supports and enhances the Agile practice implemented within the client company. 

First, the Exactpro team has people responsible for setting up the alignment between the software testing and software development activities and implementing the continuous multi-level feedback loop. These people orchestrate the whole testing process, from test design to test harness and test library creation. They also communicate information to the stakeholders and make sure that software testing activities are backed up by sufficient human and hardware resources.

Second, there are subject matter experts – people who are capable of comprehensive analysis of the test results and the source of knowledge about the system under test. They make sure that valuable information is extracted from test execution, analysed and communicated in the most efficient and timely manner.

The third group of people on the team is responsible for the testing environment: hardware and software resources needed to execute tests under sufficient load, testing framework deployment, monitoring of the test execution process.

All team members work together to ensure the highest development speed and help to achieve it, providing timely and informative multi-level feedback about the system under test.

Summary

To achieve the speed of innovation, the Agile development process should be perfectly aligned with the process of getting truthful and unbiased information about the software product. While software testers are responsible for the quality of the information service they provide, it is the stakeholders' responsibility to make sure that information flows within their company are implemented efficiently, critically evaluate the received objective facts, and, based on them, make informed decisions. 

In Exactpro’s view, the following are required to achieve the speed of change, while aligning the development effort with the Agile principles: 

  • the process of the continuous multi-level feedback loop 
  • scalable platforms for massive test operations
  • highly qualified software testing talent

All three contribute to your ability to respond to change in a rapid and resilient manner and obtain a competitive advantage through Agile transformation.

References

Beck, K., Beedle, M., van Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R., Mellor, S., Schwaber, K., Sutherland, J., & Thomas, D. (2001). Manifesto for agile software development. Available at https://agilemanifesto.org/.

Bolton, M. (2010). Testers: Get Out of the Quality Assurance Business! Available at https://www.developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business/.

Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Addison-Wesley Professional. 576 p.

Drexler, K. E. (2013). Radical abundance: How a revolution in nanotechnology will change civilization. Public Affairs. 370 p. 

Financial Conduct Authority (2022). FCA Handbook. Available at https://www.handbook.fca.org.uk/handbook.

Gregory, J., & Crispin, L. (2014). More agile testing: learning journeys for the whole team. Addison-Wesley Professional. 544 p.

Gren, L., & Lenberg, P. (2020). Agility is responsiveness to change: An essential definition. In Proceedings of the Evaluation and Assessment in Software Engineering. Association for Computing Machinery, New York, NY, USA, pp. 348–353. DOI: https://doi.org/10.1145/3383219.3383265.

Gruver, G., Young, M., & Fulghum, P. (2012). A practical approach to large-scale Agile development: How HP transformed LaserJet FutureSmart Firmware. Addison-Wesley Professional. 208 p.

Willink, J., & Babin, L. (2017). Extreme ownership: How US Navy SEALs lead and win. St. Martin's Press. 384 p.