Introduction - Challenges that Affect Large Projects
Quality assurance of large projects is exceedingly difficult to achieve. In addition, QA is very problematic in terms that it costs a lot of money and requires a lot of time, and the results are not always as good as expected. In addition, testing a large system poses further challenges. So the aim of this paper is to discuss how QA can be implemented quickly and effectively.
Exactpro Company & Experience
Exactpro is a specialist firm focused on functional and non-functional testing of wholesale financial products systems. This independent company incorporated in 2009 has grown to employ 240 specialists in 5 years. Exactpro is a US company with four QA & software development centres in Russia and sales support in the UK. The company is mostly engaged in testing trading platforms for exchanges and brokers, but also has significant post trade experience.
What we are looking for in testing is for it to be comprehensive, fast, flexible enough to be adapted for different projects, we are looking for the results to be accurate, and for the testing team to make a meaningful contribution to the whole team. It is easier said than done.
How Effective QA Can Be Part of The Solution, Not Part of The Problem
QA is only a part of the whole that typically does not even start their work until later on in the project, that is why we favor their involvement as early as possible, to make the best out of their contribution. But in either case, all the functions in a project start with people. And the people fall into two categories: the ones driving the project (the “stars”) and the ones simply participating in it. And although it seems quite hard, we believe it to be possible for the “stars” to form the QA team and to give pace to the whole project. Thus, the elements of a highly effective testing team are interconnected with the personal qualities of the people in it and result in great team dynamics across the whole project team.
Here are these and some other vital elements that the Exactpro team fully measures up to:
- Highly capable proactive people with the right behaviours and deep understanding of what they do
- Efficient and effective test processes
- Right use of technology and the ability to build just the right tools with the particular task in mind
- Independent and objective measurement of progress and clear feedback
The last point is especially relevant, because that is what the ideal high-quality QA team does: they do not just give feedback based on the specifications that the business analysts used, but on the things that the business analysts missed.
All of these elements together create behaviours that are quite different from what is normally practiced. The communication between different parts of the team becomes more dynamic and more efficient.
Team Dynamics When the QA Team is Excellent
Developers try harder to get it right in the first place, get rapid/timely feedback on what is wrong and work harder to solve defects to keep the project on track.
Analysts and designers have clear problem statements to work from – with detailed supporting evidence.
Sponsors and project management have (justified) confidence in the system, fewer problems are going to be found when Live, and the project will ultimately cost less.
Features of Complex Post Trade Infrastructures
There are some external parties that contribute to the complexity of the trade infrastructure: the markets, the participants with different types of connections, settlement and payment systems, market data providers and interventions from the operations team. Inside the system there are several features that are of high complexity: 1) reference data that requires a high degree of accuracy, 2) risk management that requires a lot of resources and a high level of skills, and 3) the schedule, which will be explained below.
Every day a clearing system runs some routine processes, these are: trade upload, smart computations, settlement sessions, collateral uploads, reporting, etc. They are repeated every day, in a predefined order, and sometimes, at predefined times. Needless to say, these processes are dependent on each other. Thus, to test a settlement system properly would mean to test all these repeatedly happening processes. There are also several data inputs into the system that are kept in the system throughout all the stages of the test scenario and present an additional challenge.
Here are three test scenarios which need to be executed if we are delivering a clearing system.
- Collateral Deficit. Trades are required to invoke the initial margin call, then we check the margin computations and simulate the input from the participants that deliver collateral to cover the risk exposure by CCP, and we need to have additional computations with the regulated collateral.
- Delivery Default, where we have multiple-day scenarios which start with a trade upload and continue with settlement during the next days, and within this time we need to simulate different inputs from settlement systems, where one participant fails on these positions and another settles, and so on. We need to check for the processes that happened after the settlement period took place, which may take several days due to the nature of the process.
- Corporate action. This scenario spans several days as well. It starts with a trade upload too, and our task is to make sure that corporate actions are only applied to the positions that are eligible for them, and we also need to check that further computations take the new and original positions properly.
Six Challenges Resulting From a QA Perspective
The given list of challenges only contains 6 of the most pertinent challenges and their solutions.
- Testing complex multistep scenarios.
As we have seen above, there is a real need for testing multistep and multiday scenarios, there also could be a need to test several scenarios at the same time. Our solution includes using an automated test tool which allows running several scenarios, organizing the test steps in batches and aligning the test schedule with the system schedule. Our test tool allows us to execute all the test steps in a predefined order which facilitates aligning the testing order with the events in the clearing system.
- Limited test availability of downstream and upstream systems.
In the picture "A complex post trade infrastructure", all the external parties are connected to the clearing systems. However, in real test life (see Pic. "Limited test availability of downstream and upstream systems") there is no such system, because such connections are not available. So our task is to simulate all these interfaces. We create a test harness which controls both the simulators and the execution of the test cases.
- Most of endpoints are accessible via API or File exchange (no GUI).
To deal with this challenge, we use a test tool which defines the input in a user-readable format, after which the system transforms this input into a Request which can be understood by the system interface, and then it receives a response which can also be transferred into a human-readable format and even validated against the extracted results.
- Reference Data setup or migration.
It has already been mentioned that reference data is quite complex in the clearing systems. It requires an appropriate setup, maintenance, and there is often a need to move the reference data from one test environment to another. For this purpose, we are using data-management tools which allow to achieve this goal. In addition, the same tools can be used for data migration in production.
- Complexity of Risk Calculation Algorithms
Risk management is a key area of the clearing and settlement systems, requiring much more effort than the others. Its complexity can be explained by several factors: firstly, the mathematical concepts underlying this area are quite complex, secondly, a large volume of data is being used and should be tested appropriately. One solution is to test the model which is based on the simplified assumptions, but replicates the algorithms and the calculations done by the system, and compare the actual results of the system under test with the expected test results computed by the model. Another solution is to define the reliable set of tests from the previous version of the system and define the baseline from regression, compare it with the results that come from the system under test.
- Regression Cycle for a substantial number of test scenarios.
Normally we not only need to run a substantial number of test scenarios, but also run them in an appropriate time frame. The approach consists in creating different test cases. We need test tools which are capable of running the test cases for the system. Further, we should automate the test cases with a tool, apply the control of the test environment and make sure that the state of the environment is appropriate for the regression, and, finally, we run the automated regression. Internally, we call it “The Big Button”.
All these approaches fit into the Holistic Integrated Automated Test Solution. Put together, the generic and the specific aspects can help create a great QA team who will really facilitate the implementation of your post-trade system ahead of time, below budget, and with much efficiency.