Exactpro Test Automation Approaches: A Case Study in Risk Management

Introduction

This is a case study about challenges of testing Risk Management systems and our test automation and testing approach developed and implemented for our client, a Central Counterparty responsible for clearing and risk management of CCP-eligible transactions on a leading European exchange.

Challenges

1. Functional Area:

   the underlying mathematical concepts are quite complex, and large volumes of data are used;

2. Calculation of Margins:

   Margins are calculated using:

   • the MVP (Method for Portfolio Valuation) methodology for Governments Bonds;
   • the MARS (Margining System) methodology for Equity Derivatives and Equity Cash products;

3. The Aim of Initial Margin (IM):

   Initial Margin is called on a daily basis to cover theoretical costs of liquidation in order to close the open positions in the worst possible market scenario, within a maximum price variation range called “Margin Interval”. The “Margin interval” is specific for each financial instrument;

4. Intraday Margin:

   intraday margins are called by the margining system if sudden sharp price variations occur or in the case of a Member’s excessive overall risk exposure;

5. Default Fund:

   a Default Fund is managed by the system as additional protection aimed at covering risks associated with sharp price/interest rate movements. The Default Fund amounts are calculated as a result of periodic stress tests.

Test Automation Approach

Our Solution is to test by using a model based on simplified assumptions, but replicating the algorithms and calculations done by the system under test, and compare the actual results taken from the system under test with expected test results computed by our model.

A Risk Simulator was developed and integrated with our test harness created to replicate Margin Run activities, including message flows and margin values.

Smart Actions were developed to calculate margin values and perform default fund verification and reporting.

Risk Engine Log Parser and Risk Parameters Analyzer were created to analyze and verify margin and risk parameters based on input data from both Margining and Clearing systems. It provides the user with a possibility to check the actual margin values received in the systems against the calculated expected values.

End-to-End Verification of Margin and Risk Calculations for all Cleared Instruments Using a ‘Risk-Based Margining’ Algorithm

As we discuss Exactpro’s E2E testing approach developed for the Client, we will expand on the workflow used to verify margin calculations and other risk calculations for all cleared instruments on the CCP. In this example, the Clearing system is separated from the Risk Management system. Other infrastructures may have the two systems merged into one. In that case, the software testing framework would be customised to that type of configuration. The workflow of the approach (illustrated in Fig.1) can be broken down into four steps.

1. Generation of realistic test data
   To produce realistic test data for our modelling, we used available specifications and Exactpro’s proprietary general-use Risk Management System Model that is fine-tuned and expanded on, with each client implementation. System-specific information such as price and instrument variations, trade volumes, clearing accounts, positions, and exchange rates formed the test basis – the volume of parameter and action combinations subject to verification. After the test basis is compiled, test cases were generated based on pre-modelled scenarios.
2. Scheduled message injection
   For the inputs to reach the required phase of the system’s daily lifecycle (DLC), messages were organised and injected into the Clearing system by ClearTH during carefully selected events aligned with the system’s DLC or larger (weekly) operational cycles. ClearTH’s capacity to simulate the operational day/week of a risk management system via multi-day test scenarios enabled the Exactpro team to effectively verify all the required workflows and parameters.
3. Margin and risk calculation
   Upon the completion of the Clearing system cycles, the clearing positions were passed on to the Margin and Risk systems for subsequent real-time/scheduled margin and risk calculations. These include calculations such as participants’ initial margins (VaR(value-at-risk)-based and SPAN-based), the variation margin, credit stress tests and liquidity stress tests, as well as backtesting. Clearing fund (default fund) calculations were also performed at this step. The results of calculations are passed on to the E2E Model for verification (which includes reconciling the received results against pre-simulated expectations).
4. Aggregate reconciliation and results reporting
   After the relevant calculation and verification cycles were completed, the output data was passed on to the ClearTH database for analysis. ClearTH verified the parameters output by the Margining and Clearing system against the values pre-computed by the Model. The Margining system values were returned to the Clearing system for subsequent calculations and verification, while the Risk system parameters were verified separately. The goal of the reconciliation is to detect discrepancies (potential defects) between the margin values received from the three systems against the pre-modelled expected values.

Using an E2E approach enabled the Exactpro team to effectively replicate the system’s components and produce a comprehensive automated test library. It makes the quality validation practice quantifiable, putting a limited and finite timeframe on test execution, defect discovery and analysis.

Are you looking to improve the operational resilience, capacity and availability of your risk management system or a larger CCP/CSD infrastructure? Get in touch with us via info@exactpro.com to set up an introductory discussion on how our AI Testing approach can help.