Date:
1. Introduction
Exactpro is a Quality Assurance services company that honors quality above quantity. Hiring more and more specialists does not equal better QA. Relentlessly looking into the root cause of the issue, applying new findings across all projects, focusing on delivering a high-quality result and workable solutions is what effective QA stands for. Another part of providing efficient QA services is highlighting the potential risks: the risks arising from the given tasks, for example, the requirements that need to be implemented. To lead each project to success, Exactpro specialists address such issues in a wise way by developing a tactical approach to each assignment. “Challenge the requirements and don’t simply implement them” - this approach is underpinned by the company’s motto - “Exitus acta probat” (lat. for “The outcome justifies the deed”).
There are three main focus areas that result from this approach: 1) Deliverables as a whole, 2) The QA strategy, 3) The project needs in reaching the goals.
Providing QA to post-trade systems in an Agile world is a challenging endeavor. This white paper outlines Exactpro’s policy on tackling these challenges and the way in which the company is collaborating with agile. To the stars through hardships.
2. Implementing Agile: The Process
When introducing any popular methodology or idea into a new field, it is important to make sure that the process does not turn into a cargo-cult: replicating the circumstances of the successful outcomes and expecting an abundance of instantly effective processes is not the answer. Obviously, the real root cause of the success needs to be figured out first. To prevent this common mistake, QA providers should keep their focus on testing the overall functionality, they should focus on the entire end-to-end scenarios, as well as the deliverables. Exactpro strategically invests into the automation of a comprehensive end-to-end library.
Nowadays, too many businesses try to follow agile with such hunger and without making the proper effort to adjust it to their field. In this case, operating model adaptation is highly recommended. It is crucial to make sure that the Agile framework has a firm foundation and is not just a facade covering the chaos behind. For a successful Agile operating model implementation, a QA provider should have certain responsibilities.
QA:
- Establishes the internal processes
- Highlights the missing processes, should they find any
- Aims to ensure that the change works, rather than making sure that the story is tested and closed within the active sprint
3. Implementing Agile: The Challenges
While maintaining Agile work, QA specialists face a number of challenges that must be addressed in a timely manner:
A. The QA team’s equipment.
To support the system schedule and multi-day tests, to support continuous integration environments with the constant Big Button execution, it is crucial for a QA team to be provided with a whole orchestra of test tools.
B. End-to-end Regression library.
It is required on top of business-as-usual activities to design an end-to-end regression library covering the major functional areas based on the main production processes.
C. Daily regression/the big button framework.
Such a framework must be in place to allow for a quick regression of any drops. (these three purely QA-related challenges are touched upon in more detail later in the white paper)
D. Drop procedure.
A drop procedure formalization and improvement should cover the ability to mark each and every drop with a unique version, in addition, such drops should be accompanied by a comprehensive change log.
E. Reference Data management.
An automated procedure of reference data management should be established to avoid numerous issues caused by data asynchronization errors.
F. Requirements Knowledge Base.
The idea that an Agile process should be supported with less documentation in place may do more harm than good. The process of building the documentation should be worked out and improved.
G. Healthy communication between QA, Business Analysts and Developers.
All questions should be considered as positive input and encouraged, no question is frowned upon.
4. Case study: QA of a Complex Clearing and Settlement System
The process of providing QA for a post-trade platform (a complex clearing and settlement system) is complicated by a number of QA-related factors.
A. The number of components of complex post trade infrastructures is very high.
Such components as risks application, reporting databases and hubs, and others lead to the intricacies in designing test scenarios.
B. Upstream and downstream system dependency.
The challenge is that the full use of the surrounding applications which are not under test is hardly ever available, but the integration and dependency on them is important.
C. The participant structure is very complex.
The participants’ roles, their preferences on the trading side, their internal accounts dependencies, them being clearing and non-clearing members - all these factors add value to the parameterisation of the test library.
D. Trade/Xfer/Position/Account life cycle.
The whole life-cycle needs to be taken into account and the transitioning across different processes during the whole period of a multi-day cycle should be considered.
E. The number of Asset Classes may vary.
Clearing different asset classes requires the involvement of different system processes and increases the variety of markets and cross currencies.
F. The complexity of the Risk calculation process.
Real-time and historical data is required to be able to validate complex computation algorithms.
G. Access via a set of API endpoints.
FIX, SWIFT, Custom API should be supported by our test harness to be able to emulate the required activities from the upstreams and allow the system under test to continue its operation.
All the challenges mentioned above and their parameterizations lead to a significant number of test scenarios.
5. Exactpro: Innovation in QA
Circumstances like that call for agility and innovation. Nowadays, it is crucial to be able to execute a big variety of test scenarios within a short period of time, as well as to have a continuous integration environment with a comprehensive E2E library on board. It is more important than ever to support the implementation of changes and to identify defects at early stages of the development process. Exactpro has managed to come up with a solution for testing such complex systems effectively. Below is Exactpro’s holistic approach to addressing all these challenges:
- Simulators are developed as a part of the test harness. Understanding the upstream and downstream systems and having the format and layouts of the interim protocols to those apps allows to emulate these streams and to continue the test design, respectively. This approach provides more flexibility and independence from the availability of the surrounding systems.
- The test harnesses are configured to support APIs to be able to mimic the participant activities, market updates and other processes.
- Market data streams are defined by avoiding chaotic updates and replacing them with emulation updates. The emulation updates contain predefined datasets, so when running the checks, QA specialists can rely on this “controlled” data, as well as on the expected behaviour model.
- Test harness controllers are created, especially when the system lives very complex daily, weekly and monthly cycles.
- Test schedulers are developed for incorporating all end-to-end test scenarios into one test library run called “The Big button”.
Alyona Bulda has been with Exactpro for over 7 years, having started her career as a QA Engineer. She has worked on projects with various architecture and functional complexity, including exchanges, brokerage platforms, post-trade platforms and a ticker plant with hardware-accelerated solutions.
