This case study describes Exactpro’s passive-testing-based solution for streamlining customer conformance testing automation for trading, broker and post-trade venues.
Conformance Certification
Conformance certification (also known as conformance testing) is a mandatory step in ensuring that customer systems comply with the officially declared exchange/broker certification rules. Conformance certification is conducted in order to prevent compatibility issues between the trading platform and the systems of the trading participants connecting to it.
For the purposes of this case study, we will use the term ‘trading venue’ to imply any parties conducting conformance certification. The list of these includes but is not limited to exchanges, alternative trading systems (ATSs), multilateral trading facilities (MTFs), Swaps Execution Facilities (SEFs), broker and post-trade systems.
Some of the typical challenges faced during conformance certification are:
- world time zones: time differences cause difficulties with planning and coordination of testing by teams located in various parts of the globe;
- performance: certification can be required by several customers concurrently due to business or technical events, such as the deployment of new software, regulatory or technical requirements;
- competency: specialists conducting certification must have outstanding technical and business knowledge;
- coverage: certification tests must be based on typical scenarios specific to the system and produce meaningful results.
One of the ways to cope with these challenges is creating a scalable automated technology solution for conformance testing and the analysis of test results.
Exchange platforms typically apply the following three approaches to conformance certification and its automation:
- Customer testing against an exchange simulator: provision of a specialised simulator to customers to conduct certification. The customers conduct testing using the tool and provide certification logs;
- Customer testing against a replica of the exchange production system:
- provision of access to a specialised test environment which represents a smaller replica of a production system,
- documenting the procedures,
- conducting joint certification with the customer or customer self-certification,
- using passive testing methods – such as validation of the traffic capture of network connections or data from log files – to analyse the success of the certification procedure.
- Customer testing in the production system during dress rehearsal periods (before platform’s go-live date or during planned weekends):
- provision of access to the production (or soon-to-be production) environment,
- defining specific time windows for the access,
- defining specific market/segment or set of instruments for customer testing in the production environment,
- documenting the procedure and dress-rehearsal steps in order to have smooth weekend transitions,
- validation and analysis of the traffic capture of the network connections or data from log files.
This case study is focused on the second approach. However, trading venues’ certification teams can easily reap the benefits of passive testing automation for all the conformance approaches listed above.
Passive Testing Approach Benefits
Passive testing is the process of finding defects in a system under test by investigating its behaviour without affecting its normal operation. In contrast, active testing tools directly affect the trading system and are used for interacting with it, e.g. by exchanging messages, analysing responses from the system, conducting load testing, etc.
A passive testing tool enables prompt and comprehensive analysis of high volumes of data, as well as detection of deviations of the system’s behaviour from requirements, if it meets the following conditions:
- data regarding all events in the system is collected (traffic, logs);
- system impact is minimal;
- alerts regarding deviations from normal behaviour are generated in real time;
- criteria for pattern recognition are flexible and adjustable;
- storage of and access to historic data are provided;
- there’s an ability to monitor/review a sequence of events and internal states of the system as a specific snapshot in time.
Exactpro’s Test Harness for Passive Testing
Exactpro’s proprietary test harness tailor-made for passive testing helps to streamline the onboarding processes and provides comprehensive conformance test coverage.
The framework performs the following functions step by step:
- Parses the captured network traffic and, optionally, the logs;
- Stores transaction messages in a database;
- Allows for automatic validation of the saved scenarios as ‘passed’/’failed’;
- Via a user-friendly UI, generates a test report for a particular customer and time period, with the required level of detail.
The use of the test harness (See Fig. 1) significantly facilitates customer conformance testing for trading venues.
Using the Exactpro Test Harness for Conformance Procedures and Automation
We will now review the approach to setting up automation of the customer certification process using the Exactpro proprietary test harness for passive testing.
As part of every system’s go-live or upgrade, the customer’s technical support team should plan to support a seamless migration of the participants and vendors from the legacy system to the new generation of the platform or from the previous version to the newer one equipped with an additional set of features. It is as important to regularly optimise the conformance procedure to ensure that the readiness accreditations are given to participants and vendors on time.
Exactpro's customised passive testing tool can be deployed and self-hosted by the trading venue's customer support team. The tool is well-equipped to tackle most conformance-testing-specific challenges.
The characteristics of the system under test that directly affect testing are (See Fig. 2):
- the availability of specifications for all protocols supported by the system, i.e. FIX protocol, order entry or market data binary protocols;
- platform network schema complexity, i.e. the number of components of the distributed system under test, the existence of stand-by components, load balancers, tcp/ip, multi/unicast routes, and others;
- the number and intricacy of business flows which should be considered for the onboarding process;
- daily data loads, i.e. the number and frequency of the transactions which might appear in the system daily.
The aforementioned challenges add on to the following requirements set forth for the conformance tool (Fig. 3):
- data storage – the requirement stems from daily data load projections made to allow the trading venue’s technical support team to access data for conforming customers;
- tool availability hours – some of the customers connect to the system at different hours and for different time periods (up to multiple days), and it’s beneficial for the customer support team to generate a conformance report at a convenient time for any required period;
- performance – to support the provision of near-real-time conformance results;
- configurable conformance testing report structure – to adapt the report to the representation most beneficial to the trading venue’s support team and to support any new changes that should be applied to the reporting process.
Typical challenges encountered in implementing conformance testing automation are: protocols to be parsed, the volume of data, the necessity to keep raw binary as well as processed data from both successful and unsuccessful customer tests, and the complexity of the logic of rules validation in the tool.
A financial platform’s interfaces can be extremely flexible and detailed: they may contain more functions than those offered by standard FIX protocols. Trading Interfaces can support various transactions, market data streams and different queries. Extremely complex as it is, they also contain some interdependencies. All these factors create an additional challenge when it comes to analysing the results of certification tests automatically.
The Exactpro test harness addresses all these challenges. Additional customisation can also be performed in order to fine-tune the tool to the trading venue’s specific needs.
Conformance Testing Workflow with Exactpro Test Harness for Passive Testing
Conformance testing and test tool customisation workflows consist of the following steps (see Fig. 4):
STEP 1: Creating Conformance test scenarios: Trading venue’s technical support specialists create test scenarios for сonformance testing – or have them generated automatically – based on the specifications, regulatory requirements, and participant types.
STEP 2: Customising the tool to parse the traffic of specific financial protocols, storing it and analysing the saved data. The Exactpro team customises the test harness for passive testing to make sure that all traffic sent and received via the required protocols can be parsed and stored in the database. During the parsing process, customer’s messages are checked against the specification. Certain pre-developed Conformance Automation tool’s SQL procedures called ‘rules validators’ are also implemented to be run on the saved data in order to detect logical deviations in the incoming and outgoing messages. These validators mark all captured and saved messages as ‘passed’ or ‘failed’ (See Fig. 5).
STEP 3: Verification of the Conformance test scenarios and Conformance Automation tool customisation. The Exactpro test team automates all the сonformance testing scenarios and runs them using test tools simulating the participants' positive and negative behaviour. The test traffic logs are captured and analysed in the Conformance Automation tool. Thus, the certification flow is checked along with the customisations. Business and technical support teams’ feedback on the content of the сonformance tests is also incorporated as improvements.
STEP 4: Conformance tests capture. Customer’s real-time transactional data in the platform’s financial protocols is collected, covering their conformance test executions. As a result, the Conformance Automation tool’s database contains captured data for each particular customer with the conformance test results markdown: either ‘passed’ or ‘failed’.
STEP 5: Conformance certification tests report generation. A team responsible for conformance certification and onboarding can generate a report with a set of Mandatory, Optional, Supported and Non-supported functionalities per each customer, for a particular time period. This report indicates the status of each сonformance testing scenario – ‘passed’ or ‘failed’, including the number of attempts with the required level of detail – to be shared with the trading participant. The conformance report is configurable and can be adjusted after review and suggestions by the trading venue’s technical support team.
When Steps 1-2 are done and the Conformance Automation tool is fully customised and tested, the repetition of conformance tests and the reports generation steps are performed faster and can be reapplied to any new customer to be onboarded. Conformance test reports can be easily accessed and checked by the venue’s technical support team via the UI or in the remote shared file storage (See Fig. 6). The auto-generated conformance test reports can be transferred to the locations shareable with customers, such as user portals or SSH File Transfer Protocol (SFTP) servers.
While trading venues are encouraged to perform regular conformance testing, they must repeat conformance testing of their software as soon as possible in the following circumstances:
- when their software is modified in any way that may affect or directly impact connectivity to the platform,
- when their platform undergoes an upgrade or mandatory changes in its production environment,
- when the network architecture changes,
- during extended idle periods when the customer’s software is not connected to the platform,
- upon a specific request from the owner of the platform.
Using th2 Framework Modules to Optimise the Conformance Automation Solution: Reference Architecture and Goals
Exactpro's Conformance Automation test harness is continuously improved based on our clients' feedback and suggestions. The latest design improvements involve leveraging th2 framework modules (See Fig. 7).
The characteristics improved by the use of th2 components are as follows:
- Scalability: the number of conformance testing scenarios that can be verified has increased tenfold; new protocols can be added with minimal changes to the code;
- Speed of traffic, log parsing into the DB;
- Speed of rules validation on the saved data;
- Usability and flexibility of the generated conformance testing reports.
To achieve scalability, cloud technologies and a microservices architecture have been introduced. In particular, the main th2 data lake is implemented on the Cassandra Database (DB) which can be distributed between multiple nodes. Network traffic can be stored per customer on the nodes.
The speed of data parsing to the DB is achieved by parsing only the required information from the network traffic. This is implemented via PCAP readers – th2 services that can be configured per protocol, per customer, per folder, or per test environment.
Multiple services called ‘crawlers’ are constantly pulling new messages that appear in the DB. A crawler can be assigned to a specific customer(s) or protocol(s).
The Conformance scenarios themselves can be checked by th2’s rule processors. The number of scenarios checked by a rules processor is adjustable. The scenarios are validated faster, as rule processors can only be run against the selected data on demand. The crawler component can query not only the transactional data, but also the events collected previously after rule processors executions, and the report viewer can be organised hierarchically.
So, every th2 module can be scaled and adjusted for the required performance. This makes the th2 framework highly customisable and sensitive to trading venues and market needs.
Conclusion
Conformance certification can be significantly simplified and enhanced by using a passive testing tool. A scalable tool fine-tuned to suit a particular client’s requirements and specifics provides great visibility and transparency to the conformance testing process, further data analysis and reports. Exatpro's experience of using the Conformance Automation tool highlights the benefits this approach can bring to the exchanges and post-trade systems space.
The Conformance tool the Exactpro team has developed based on our industry-proven test harness for passive testing can significantly improve the efficiency of processing conformance tests for our clients’ technical support teams. Our team’s software testing and automation expertise has been critical in creating conformance test suites and perfecting the approach over the years. Leveraging this approach and workflow significantly reduces the chances of the onboarded customers experiencing a malfunction in a real production environment.
Alyona Bulda, Head of Global Exchanges, Exactpro
About Exactpro:
Exactpro is an independent software testing services provider for financial market infrastructures. Our clients are exchanges, post-trade platforms, and banks across 20 countries. Founded in 2009, the Exactpro Group is headquartered in the UK and has operations in the US, Georgia, Sri Lanka, Lithuania, Armenia, Canada and Italy. The firm employs over 650 specialists. Exactpro is involved in a variety of transformation programmes related to large-scale cloud and DLT implementations at systemically important organisations. Our area of expertise comprises API gateways, matching engines, market data, clearing and settlement, market surveillance, helping improve scalability, latency and operational resiliency. We Build Software to Test Software: our set of test automation tools covers most financial industry use cases and is easily customisable to meet new requirements.
Learn more at exactpro.com or follow us on LinkedIn and Twitter.