Date:
1. Exactpro: Introduction
Exactpro is a part of the London Stock Exchange Group’s Technology Division. Exactpro focuses on functional and nonfunctional testing, most of the business being in the market infrastructure space, exchanges and clearing. The company currently employs over 500 people: essentially, 1 in 8 of the London Stock Exchange Group employees works in the software testing quality assurance area.
We deliver software testing services across most divisions within the group in technical markets, post-trade and information services. We also provide open-access software testing services externally to many companies from Singapore to the USA, from Europe to Johannesburg.
2. What is disruptive technology? Disruptive vs Incumbent
“Incumbent” means working with something you already know and something you can expect to perform in a certain way. “Disruptive” technology occurs when we face new things - the things that will re-define what we already know. Incumbent testing of post-trade systems is a very complex area to begin with. Post-trade systems are massive, they have many interfaces, many distributed components and legacy systems. There are many challenges in providing Quality Assurance for such systems, starting with participant and asset classes complexity, followed by lifecycle complexity resulting in multi-day cycles, on top of risk calculations, existence of upstream and downstream systems, APIs, reports, etc.
Traditional software testing “hooked on” preconceptions about how the system under test is supposed to work. Only disruptive testing is able to reveal truly new knowledge about the system and let us learn from it. And software testing is relentless learning.
3. Disruptive Testing at Exactpro
Exactpro is a defect mining company. The goal of disruptive testing is to find defects that are unlikely to be found in testing with other methods. It is quite challenging to be able to extract hidden defects that only reveal themselves under load, under risk conditions or other atypical conditions and cases. In order to do it, Exactpro relies on a set of testing tools. One of them is the ClearTH tool that is used within the London Stock Exchange Group, as well as by external companies. It is used to cover most of the testing for LCH’s new clearing platform. The tool simultaneously connects to all possible endpoints, where necessary, to simulate various data flows. It can work as a settlement simulator, as a risk management simulator, it can control reference data and the incoming market data. The tool concurrently operates on all these endpoints and produces diverse random loads, and places the post-trade system in stress conditions. Contrary to typical load or functional testing, we have carefully studied everything that happens at every endpoint and, also, internally within the system. This allows us to identify the problems that will not occur in the process of ordinary functional validation, where tests are run one by one. This approach is used in supporting large initiatives, both in the waterfall mode and also on Agile projects.
4. Exactpro and Agile
Similar to other large financial sector organizations, London Stock Exchange Group’s technology is going through an Agile transformation, and running testing scenarios for large systems in short sprints represents many challenges. The first idea that comes to mind during a transition to the Agile mode is to try and squeeze software testing into sprints and confine all the testing to sprints. Quite frequently, it leads to “confirmation-bias” testing: the system is expected to work, and instead of trying to obtain new knowledge, the testers convince themselves that it works as expected. In Agile projects, it is important that every iteration delivers new value, and when people implement it, they frequently forget that the test library should reflect the same idea. So if no effort is spent on creating a test harness, the end result will be a test library that will not deliver value every step of the way.
The law of requisite variety states that the control system must have at least as many possible states as the system it wants to control. It means that test harness development is a separate software development process. The challenges and the effort required to create a test harness is frequently underestimated.
5. Testing Critical Post-trade Infrastructures
Performing ordinary functional testing is much like obeying the main safety rule on board a submarine: “Do NOT open the portholes when underwater”. Ordinary functional testing comes down to iterating through a finite number of scenarios to prove that the “portholes” will, indeed, not open. The number of scenarios may vary from one to over a hundred, and their sole purpose is proving that the “portholes” stay shut in this particular subset of scenarios.
Non-functional testing, on the other hand, lies in iterating through an even smaller number of scenarios - that is the general tendency across Exactpro projects - to prove that the “protholes” will not open by brute force.
Last but not least - disruptive functional testing. Its first part consists in iterating through a huge number of random diverse scenarios under load to prove that the portholes stay shut. The second part of disruptive testing is opening the “porthole”.
Disruptive functional testing is the only way to ensure that your system is not only ready for what you expect, but it is also potentially ready for the unexpected. And when new disruptive technologies are introduced, the software testing approach that is used should match the complexity and the nature of disruptive technologies.
