Deliberate Practice of Software Testing in an Agile World

Exactpro is an open access software testing subsidiary of the London Stock Exchange Group. (The content of this white paper should be viewed in the context of its publication date – May 25, 2017. Exactpro Announced Management Buyout from London Stock Exchange Group on Jan 17, 2018)

Exactpro provides functional and non-functional testing services to the Group’s own markets, as well as other exchanges, market infrastructures, banks and financial firms worldwide. Fully dedicated to the quality assurance of FinTech software, the company employs one in eight of the LSE staff. Exactpro has grown 50-fold since its incorporation in 2009. This was possible due to some fundamental principles that have been worked out with experience and are relentlessly observed within the company.

Exactpro - Deliberate Practice of Software Testing in an Agile World

Exactpro’s primary focus is Deliberate Practice of Software Testing. These are the six principles that are implied when using this term:

  • No Trust. No Fear. No Begging.
  • Smart Automation
  • Collaboration
  • Focus
  • Listening to the system
  • Embracing defects

This White Paper is a detailed outline of these principles.

Introduction

The London Stock Exchange, along with most companies worldwide, is currently going through an agile transformation. In the process, a legitimate question arises: is there any room left for independent QA nowadays?

Exactpro - Deliberate Practice of Software Testing in an Agile World - Quality Assurance

As a result of the recent changes, clients often express requests to confine QA to scrum teams. But most of Exactpro’s clients operate on such a large scale, that the tasks do not just fall on a single scrum team, but more like a train car of teams or even a whole chain of train cars - a prison train for software testers. To retain our internal freedom in such a harsh environment, we have had to elaborate and follow some survival principles.

Don't believe them, don't fear them, don't ask anything of them

In one of his books, Alexander Solzhenitsyn, a Russian 1970 Nobel Laureate, quoted a saying popular among the prisoners of GULAG which helped them endure their daily interaction with the prison officers. It goes like this: "Don't believe them, don't fear them, don't ask anything of them". A similar latin saying - Noli credere, noli timere, noli petere - is attributed to Epicurus, an ancient Greek philosopher. Exactpro testing philosophy principles are the essence of these two combined: “No Trust, No Fear, No Begging”..

 

1. No Trust. No Fear. No Begging.

Seven years ago, Exactpro had a very small team. Now we employ over five hundred specialists. In addition to luck and hard work, there is another factor that makes us uniquely skilled and that has helped the company grow: people born in the former Soviet Union generally grew up in the atmosphere of profound doubts about everything, thus developing a mindset of mistrust. It is an ideal mentality for a software tester.

Exactpro Builds Software to Test Software using the company’s own gateways to connect to the systems. This makes independent automated testing possible. Exactpro’s testing tool Sailfish has been instrumental in providing connectivity testing for many brokers and exchanges.

Independent Automated Testing

There are hundreds of papers stating that agile requires trust. However, complex and stable systems allow for No Trust. They are built on the absence of it. No one can ensure the election of an ideal president every run. Instead, a system of checks and balances is put in place, guaranteeing that the society will remain stable, regardless of the election outcome. History has proven once and again that blind trust put into one particular leader may lead all the way up to a dystopian nightmare.

Have No Fear

A common fear among clients is having unrealistic performance testing. Have No Fear! The tests can and should be unrealistic. Limiting the validation to production-like scenarios and staying within the pre-specified volumes and throughputs will not guarantee adequate preparation for the things to come.

The tests should be run until the system breaks down. It is crucial to observe what will happen when the system is no longer able to withstand the load. Micro-bursts do happen in the real world. Some micro-scale unrealistic throughput level will be observed sooner or later. Do not be afraid to stretch the limits.

No Trust, No Fear, No Begging

Asking questions is crucial for software testing. No Begging is allowed, though. Begging is requesting the unavailable. And what’s the use of that? Neither should a plan be dependent on the things you do not trust.

What is the use of asking for a confirmation that Production has the same configuration as the test environment? This confirmation is unreliable, unless it can be validated. Why request for the tested drops to come to QA from development? There is no need to ask for that or to establish a quality gate between these two departments. Early testing is a better way to push forward, and defects are one of the best learning tools to help along the way.

2. Smart Automation

Requests to automate everything are frequent in the testing business. That, however, contradicts the concept of Deliberate Practice of Software Testing. Deliberate Practice is not just repeating something for 10,000 hours, as successful as the popular theory may be. Rather, deliberate practice means constantly stretching the limits to improve one’s skills.

Many people drive their car to and from work for many hours, but never achieve true mastery. That is because it is common to turn one’s brain off while doing repetitive things. Automating every user story out of context comes dangerously close to turning one’s brain off.

Smart Automation

Exactpro automates test libraries containing tens of thousands of scenarios. In doing so, automating user interfaces that will change in the next sprint would be a waste of time. Automate what needs to be automated. Automate smart.

3. Collaboration: Separate but Together

Another popular request on par with automating everything is to merge the QA and the software development teams. But does a firefighting robot’s operator really need to sit in the burning building where the robot is working?

Collaboration: Separate but Together

A whole other issue is that creating such a robot is a sophisticated technical process, possibly requiring a separate dedicated development team that is not embedded in any existing scrum team.

Exactpro has managed to create testing tools, such as ClearTH, that reflect complex operational life-cycles and reference data setups of the post-trade systems and execute huge regression libraries in an unattended mode.

It would make much more sense to merge QA with Business, that is why we try to maintain some presence at our client’s locations.

Exactpro client’s locations

4. Focus vs Cross-Functional Teams

Excellence is achieved through focus. When someone makes an emergency call, both the police and the paramedics arrive on scene. A policeman can offer immediate medical assistance, but any complex case requires a medic. Policemen and medics have the same purpose – saving and protecting human lives. However, they have different functions, and neither can fully substitute for the other one.

Focus vs Cross-Functional Teams

Software developers focus on building software. Software testers focus on finding failure. These are inherently different purposes. It is said, however, that the presence of a software testing team removes the ownership of quality from software developers.

If there is a literary representation of an ideal Agile team, it is, of course, a squad of Navy SEALs, masterfully pictured in the book entitled “Extreme Ownership: How U.S. Navy SEALs Lead and Win”. They truly take extreme ownership of the quality in everything they do. This superb cross-functional team can accomplish many missions autonomously. Having something like this for a software development team would be a dream come true. However, a scrum team cannot possibly be expected to have superhuman abilities: to be extremely professional, tireless, dedicated, selected through an elaborate process out of hundreds of others.

There is something that this self-sufficient “Ultimate Agile” team does while planning their operations. They always assume that the things are likely to go wrong, and additional heavy-duty firepower support from the tanks will be required. This is the way to run operations efficiently and with minimal casualties. Independent software testing is like those tanks.

Exactpro is well-equipped in terms of firepower. LSEG operates some of the most advanced and well-performing systems. The average round-trip latency is 84 microseconds. Last year, the number of transactions on a single platform exceeded 128 million per day.

Stress-testing,testing tools, non functional testing, load testing,load generation software

To stress-test such systems, Exactpro uses a set of dedicated testing tools. To reduce the hardware footprint of the load generation software, we make it very simple - like a bag of nails - but very powerful.

In addition, we use the tools that mimic real market participants and look at the system as real traders do. Exactpro’s Minirobots can spot problems missed by the load generators or data analysis tools, since their perspective is entirely different.

5. Listening to the Existing System

In the context of big data and machine learning, people often overlook the importance of the information coming from the systems that are already in production as a source of requirements. For instance, when a company migrates to a new system and has to implement connectivity with legacy platforms, the projects often rely on the written specifications describing how things are supposed to work, instead of looking at the actual production logs and the data setup. No merging is required to obtain valuable insight from the existing data.

Shsha - Exactpro’s passive testing tool - is capable of reading the wire and the log files and extracting data from them. It stores the data for processing and runs checks against it to find patterns and problems.

Shsha - Exactpro’s passive testing tool

The passive testing approach is also used at Exactpro for latency measurement across all the markets. In addition, passive testing can help to onboard clients and even run crowdsource testing.

6. Embracing Defects

Software Testing is a process of relentless learning. Should defects leaking out of a sprint be something to be afraid of? The answer is “No”. Defects are the best source of information about the system.

They

  • help everyone to understand the system better
  • help develop better software testing tools and libraries
  • are a great collaboration tool between Business, developers and QA.

The absence of defects may not be the evidence of ideal quality. It is a consequence of confirmation bias and groupthink. The presence of defects within the software development life cycle is what makes projects Antifragile.

But what if those defects were critical and leaked into Production? Production outages are like death, in terms that they are inevitable in complex technology systems. It is not a question of “if”, but a question of what to do when it happens.

It should always be assumed that the system will break. It is the only way to get a truly resilient system and avoid problems turning into disasters. To have a good life, one should never dismiss the possibility of death.

Exactpro’s Agile Manifesto

  • There is no need to confine QA to a scrum team or process. Instead, focus on individuals and interactions capable of finding defects.
  • There is no need to request all test scenarios upfront for test first development. Rather, focus on delivering working software.
  • Defects should be used as a collaboration tool, as an alternative to tracking SLAs between sprints and gates.
  • Deliberate Practice of Software Testing enables you to respond to change in a truly agile manner.