Introduction
The internal workings of risk management systems are notoriously complex. They require thorough verification to ensure that CCPs and clearing firms have robust, dynamic and transparent mechanisms in place to maintain sound and stable operations. An increasingly stringent regulatory regime and quality requirements have been forming around clearing over the last decade. The more recent challenge – the gradual move towards the T+1 settlement cycle – drives organisations to further optimise and automate processes, and modernise infrastructures, while addressing new risks and maintaining the depth of coverage for “extreme but plausible” scenarios and market conditions.1
Although configured in a variety of ways throughout the post-trade space, risk management units serve a common purpose – modelling risks with the aim to mitigate them. The frameworks tasked with their quality verification have to be able to replicate both the general complexity of the industry-standard calculations and the requirements and processes specific to a client implementation. Advanced levels of automation are a must for the software testing setup, due to the sheer volume of data and the intricacy of the clearing and settlement lifecycle.
n addition to having extensive post-trade system testing experience, Exactpro continuously enhances its approaches to enable our clients to deliver higher-quality software faster, while staying compliant and responding to ongoing change.
This paper demonstrates using artificial intelligence (AI) techniques to streamline software testing for risk management systems. It highlights the advanced system exploration and test automation benefits of Exactpro’s AI Testing approach, compared to the more traditional quality validation setup.
E2E Testing and Test Automation
Exactpro’s new Testing and Test Automation strategy utilises AI-enabled end-to-end (E2E) model-based testing to address this challenge. This method is known to be effective in quality verification of complex non-deterministic systems, including transaction processing systems. In contrast to conducting select sets of pre-defined verification checks targeting isolated functionalities to merely help clients check the regulatory boxes, we achieve exhaustive test coverage of the system under test (SUT). The test library – developed using the system’s digital twin – accounts for functional and non-functional aspects of the SUT, replicating all relevant algorithms and calculations.
A well-developed digital model enables efficient automatic reconciliation of expected (model-generated) and actual (received from the system) results. This, coupled with AI-driven automation, brings down the amount of time and resources spent on test generation and results interpretation, and facilitates timely defect escalation, resulting in faster and more informed strategy and product delivery.
Why Generative AI
On the path of continuous infrastructure modernisation and increasing system complexity, an AI-enabled software testing strategy can help financial organisations achieve:
- advanced levels of system exploration and corresponding test coverage,
- increased automation and speed of software testing and product delivery, and
- enhanced data standardisation and organisation for parametric test results analyses, for regulatory reporting and faster, data-driven decision-making.
Exactpro’s AI Testing approach streamlines protocol-based software testing to provide stakeholders with objective information about the defects present in their system faster and more efficiently. With the capacity to comb through massive amounts of parameter (data-point) permutations, AI-enabled automation provides increased versatility of the test library, compared to manual or formal test generation methods.
The approach capitalises on the use of exploratory data analysis and AI techniques to intelligently automate the generation and careful selection of test ideas, as well as support human testers in the triage and interpretation of test results. These benefits are complemented by smart test execution and unified data storage capabilities of the th2-clear framework. The process – organised as a continuous loop – enables iterative enhancement of Exactpro’s proprietary Generative Risk Management System Model and the th2-clear analytical capabilities.
To demonstrate the proposed AI-enabled workflow of the approach, we are using the system configuration from Exactpro’s prior project with a Central Counterparty responsible for clearing and risk management of transactions on a leading European exchange. Compared to the previous approach, Steps 1 and 4 have gained significant AI-enabled optimisations and qualitative improvements (detailed below and illustrated in Fig. 1). The newly introduced Step 5 serves to continuously improve Exactpro’s Risk Management system model (and associated test generation capabilities) via enriching it with more intricate scenarios based on the insights extracted throughout testing. We will now review the exact improvements achieved in these steps.
Fig. 1 AI-enabled End-to-End Verification of Margin Calculation for all Cleared Instruments using a ‘Risk-Based Margining’ algorithm
Step 1 – Generation of realistic test data
Test script generation is delegated to generative AI algorithms, which allows us to generate a more versatile test set out of the variety of possible parameter and action permutations. To conserve resources and achieve faster test library execution times, the Exactpro team then uses discriminative measures – symbolic AI techniques (such as Supervised Learning) – to quantitatively optimise the generated test suite while preserving the quality and scale of the test coverage provided by the original test set. The resulting test library consists of a reasonably small subset of test scripts selected out of the pool of all possible modifications, based on their performance (i.e. the extent and uniqueness of their test coverage of all relevant data points of the SUT).
Step 4 – Aggregate results reconciliation and reporting
After test execution and respective risk and margin simulation in Steps 2 and 3, the received test results are analysed and reconciled in Step 4. Throughout the testing process, including at the result reconciliation step, each test scenario is annotated for AI training and traceability purposes. The annotation process is a key contributor to:
- quantification of system resilience – it enables us to effectively assess the test coverage via data-point coverage,
- our ability to conduct ad hoc parametric analyses,
- the optimisation of analyst resources spent on reviewing the test results, recognising anomalies and patterns, and investigating the root causes,
- continuous improvement of Exactpro’s Risk Management System Model.
Step 5 - Generative Risk Management System Model Improvement
Based on the annotated features, AI techniques are iteratively applied in combination with scripted property checking rules to continuously improve the test generation algorithms and arrive at the version of the test library that covers the system’s functionality in the most comprehensive yet resource-efficient way. The final optimised test library retains the same level of test coverage as the initially generated massive test set, while requiring significantly fewer resources and test scenarios to complete the job.
AI Testing: Generating Value in Risk Management System Testing
Both established and young infrastructures can reap the benefits of AI-enabled E2E software testing. AI-enabled software testing approach makes Exactpro well-equipped to address challenges such as:
- widespread manual testing processes,
- absence of a unified integrated test library,
- lack of historical data required for a data-driven approach,
- deficiencies/coverage gaps in the existing test library.
AI Testing serves clearing organisations by:
- providing independent, timely, and in-depth quality assessment of risk management and other financial platforms, streamlined via advanced AI-enabled automation,
- delivering the test library within reduced timeframes and with highly optimised resources spent on test library creation and required for its execution,
- developing an automated regression testing library organically produced as a result of applying the E2E testing approach and created with resource efficiency in mind. Clients often opt in to keep the resulting regression library to run ongoing checks in-house.
Fulfilling its function well beyond the supervisory requirements, AI Testing introduces optimisations that translate into:
- improved operational resilience, due to more extensive and versatile test coverage,
- more transparent and flexible test data management, for on-demand parametric analyses and supervisory purposes,
- fast-paced adaptability to compliance- or user-related change,
- reduced time to market timeframes, due to widespread automation,
- reduced operational/QA costs and resources, due to early defect resolution (if Exactpro is engaged at early stages of the delivery lifecycle).
If you are looking to explore how AI Testing can help improve test coverage and drive automation in the quality assessment of your risk management system, reach out to us via info@exactpro.com.
1 “Resilience of central counterparties (CCPs): Further guidance on the PFMI.” Committee on Payments and Market Infrastructures (CPMI), International Organization of Securities Commissions (IOSCO), 05 July 2017. URL: https://www.bis.org/cpmi/publ/d163.pdf.