Exactpro Test and Automation Approaches: A Case Study in Market Surveillance

Introduction

This case study highlights the challenges and the complexity of testing Market Surveillance Systems connected to trading platforms, market data providers, involving various data mining processes, alerting mechanisms, and having different degrees of process distribution complexity.

Exactpro have tested a number of Market Surveillance Systems across different markets and locations. Our expertise spans:

  • Building and maintenance of test/regression libraries covering system platforms E2E;  
  • Automating the testing process using proven tools and methodologies;
  • Building up a holistic approach encompassing the full cycle of Market Surveillance Systems Testing based on our experience of testing the world’s leading Trading, Exchange and Clearing platforms.

Market Surveillance Systems are typically interconnected with numerous trading platforms, market data feeds and news feeds, and the introduction or upgrade of any of those systems can seriously impact the effectiveness of the monitoring platform. 

Testing is frequently performed by members of the compliance or market control functions – themselves valuable business resources – because testing requires a comprehensive understanding of the surveillance monitoring function. 

The optimisation of the Market Surveillance System ensures that false positives and errors are minimised, meaning the monitoring team are able to be more efficient, and, hence, are able to: 

  • Mitigate the costs and effort of the compliance team in testing;
  • Ensure market regulatory requirements are met and documented.

Market Surveillance System Testing: Key Challenges

A market surveillance system is a complex system that includes a set of subsystems:

1. A gateway subsystem that obtains data from different data sources such as:

  • trading systems
  • market data providers
  • trade reporting systems
  • reference data systems
  • unstructured data
  • 3rd-party data vendors: FX rates / regulatory news announcements

2. A data enrichment subsystem:

  • internally calculated statistics
  • analytics module

3. Real-time and offline alert engines to detect abusive market behaviour:

  • rule-based engines
  • advanced engines that include machine learning
  • data mining engines

Exactpro Test and Automation Approaches: A Case Study in Market Surveillance

4. A data repository to store structured and nonstructured data:

  • Oracle
  • in-house data warehouses
  • big data technology solutions such as Apache Hive / Apache Parquet
  • data lake

5. A GUI module that provides drill down capabilities to investigate the detected alerts:

  • real-time monitoring dashboard
  • offline monitoring: Market Replay system
  • third-party dashboards

Exactpro Test and Automation Approaches: A Case Study in Market Surveillance

Exactpro Test Approach

Exactpro Test and Automation Approaches: A Case Study in Market Surveillance

Automated testing using the Sailfish test tool – real-time data

Sailfish test tool is used for functional testing. Sailfish allows us to conduct end-to-end testing for the whole message flow starting from injection of data in the upstream system via the trading interface (e.g. FIX Gateway), receiving messages via the market surveillance stream gateway, and compares the received messages with the expected ones.

The high-level approach is as follows:

Exactpro Test and Automation Approaches: A Case Study in Market Surveillance

Exactpro Test and Automation Approaches: A Case Study in Market Surveillance

Testing by comparison – offline data

If there is no way to get a response from the market surveillance system in real-time via a channel, testing by comparison is used. This type of tests comprises comparing the contents of output data from the system against the actual results.

Pre-conditions/assumptions:

  • we have a “master copy” of the output data
  • the master copy had been carefully verified in the previous release (i.e. assumption: master copy is correct)
  • changes on the upstream system side are minimal
  • no changes in the input data
Exactpro Test and Automation Approaches: A Case Study in Market Surveillance

The Data Consistency Part

The data consistency verification test is performed after each operational cycle of the system.

The consistency test is based on the comparison of messages from different end-points of the market surveillance system under test and the external (integrated) systems.

The following end-points are used in the approach:

  • data of the downstream gateways of the exchange system and the trade reporting system
  • data of the input gateway of the market surveillance system
  • data from the Data Warehouse of the market surveillance system

We use the Shsha tool to parse the tcpdump files captured from the external systems (if the system itself didn’t store the sent messages) in order to compare the output data of the external systems and the input data of the surveillance system. It allows us to perform the consistency check for the whole message flow.

Alerts Testing

One of the main purposes of Market Surveillance systems is the prevention and investigation of abusive, manipulative or illegal trading behaviour in the markets. There are a set of pre-configured patterns with known abusive behavior that help streamline regulatory compliance in this area. Market Surveillance systems also contain modules that allow users to create new patterns – or adjust the existent ones – and integrate them into the system.

The most common types of such patterns are known as:

  • Layering
  • Spoofing
  • Marking The Close
  • Wash Trading
  • Front Running

Exactpro performs the testing of the Market Surveillance System patterns to ensure that the actual results match the business requirements.

Test and Automation Approaches: a Case Study in Testing Market Surveillance Systems

The Test stages here are the same: automated testing approach using the Sailfish test tool that was described above. Test Scenarios are designed based on the requirements provided.

Alerts Testing – Production Data Replay

But trust no one! The requirements may be wrong, or may not fully meet ongoing behaviour on the market and, as a result, the pattern may behave abnormally. The most common samples of such anomalies are:

  • pattern thresholds are not calibrated
  • benchmarks may be wrong 
  • false positive alerts are triggered – here we need to look at adjusting the parameters
  • or we are missing something - false negatives

All of the above anomalies are found, especially when a new pattern is designed from scratch.

What is the Exactpro approach here? We look at this as a Data Science challenge. We use data mining techniques to ensure that our analysts can see the number of alerts triggered with the updated pattern based on the production or synthetic data, and then manage the quality of these alerts.

The Next Generation of Market Surveillance Systems

Realizing the advantages of cloud computing services providers (AWS/Azure, etc.), the financial industry firms are now actively migrating their solutions – including market surveillance systems – into the cloud. Exactpro test harness tools are cloud-ready and can help clients to make the migration as smooth as possible.

The financial industry is also actively leveraging machine learning and artificial intelligence techniques. An increasing number of companies tends to apply AI to market surveillance systems for:

  • smart market monitoring
  • prediction
  • fraud detection
  • pattern recognition from data based on machine learning models, instead of rules-based algorithms

Exactpro is currently conducting research in these areas to enhance our test harness for Market Surveillance Systems.