Special Features of Testing Tools Applicable for Use in Trading Systems Production

Special Features of Testing Tools Applicable for Use in Trading Systems Production

Abstract — The paper examines basic requirements for tools developed for verification of correct work of electronic trading systems by applying High Volume Automated Testing (HiVAT) methods and analyzes the applicability of such tools during production operation of trading systems.

Keywords: test automation, trading systems, HiVAT.


Rapid development of technologies used in electronic trading systems is recognized as one of the most significant factors that influences changes in the structure and stability of the international financial markets [1]. Market participants and regulators of the financial sector experience a growing need in solutions supporting the quality of software and architectural components [2; 3].

From the technology standpoint, most of the trading platforms are complex distributed adaptive software-hardware systems, which handle parallel execution of a large number of transactions in real-time [4]. Modern systems provide response times within fractions of a millisecond and generate significant volumes of data [5].

Due to the fact that the share of financial transactions executed by automated computer-based systems [6] is growing, the main trading systems interactions are based on various open-source and proprietary network protocols and program access interfaces [7]. The specifics of automated workstations of users of trading systems is such that it involves, in particular, high frequency of data updates pertaining to quotes, financial transactions progress statuses, current positions and risks [8].

  • Let us examine several key aspects of quality assurance for trading systems. As any multi-thread systems, trading platforms are prone to malfunctions that are hard to reproduce [9] and appear only when the system is under high load. Often, the malfunctions are of functional type and are not due to the exhaustion of system resources, but rather are the result of conditions that conflict with each other over time (race conditions) during parallel processing of the occurrence of statistically unlikely internal integrity violations [10]. Troubleshooting such problems requires verification at the confluence of functional and load testing [11];
  • The need to have test coverage of the trading system’s functionality requires having a large number of scenarios as part of an automated tests library [12]. The number of combinations that require verification is particularly large for equities and derivatives [13]. Even if the scenarios included in such a library are run once, it leads to continuous execution of automated tests sequence. Multiple runs allow finding hidden problems related to the internal state of the system;
  • Regulating authorities, shareholders and trading participants expect that exchanges and broker systems are highly resilient to unpredicted external influences [3]. The academic literature has detailed analysis of existing methods of resilience testing based on running a large number of data randomly created through the system: fuzzing [14].

The aspects mentioned above mean that there is a need for sending a large number of test messages into the system during a continuous period of time. In order to expose defects related to message flow processing, scrupulous functional verification of outgoing messages and the system’s internal states is required.

The methods used for it are known as HiVAT – High Volume Automated Testing [15; 16; 17]. They are designed to recognize problems that are likely to be undetected when other testing approaches are used that involve manual creation of scenarios for automated testing and therefore produce statistically insufficient quantity of output data.

The authors’ experience shows that usage of HiVAT techniques for high-frequency trading systems not only serves as a possible way of expanding the test coverage, but is also a mandatory way of their testing in production-like conditions.

The usage of HiVAT methods over a long period of time has allowed formulating a set of main requirements for testing tools. These requirements are listed in the second part of this paper. The third part provides a comparison of testing tools and trading systems used in production. The last part describes possible expansion of the existing tools and directions of future research.


In terms of influence on the system under test, testing tools can be divided into two categories: passive and active [18]. Passive testing is a process of finding defects in a system under test done by investigating its behavior without affecting its normal operation. Active testing tools directly affect the trading system and are used for exchanging messages, analyzing responses from the system, and load testing.

2.1 Passive testing tools

Passive testing tools are used for automated log collection, data structuring, monitoring, system behavior analysis and user certification. Test tools allow promptly analyzing high volume of data, reacting to deviations in the system’s behavior from requirements and troubleshooting.

A testing tool allows achieving that effectively if the following conditions are met:

  • data regarding all events in the system is collected;
  • system impact is minimal;
  • alerts regarding deviations from normal behavior are generated in real-time;
  • criteria for image recognition are flexible and adjustable;
  • storage of and access to historic data are provided;
  • the ability to monitor the sequence of events and internal states of the system as of a certain moment in time.

Finding the root cause of a malfunction that occurred while using HiVAT-techniques is quite often a more complex process in comparison to standard methods of manual and automated testing. This is due largely to 2 factors: automated creation of test scenarios and high volume of heterogeneous data run through the system. The tester needs easy-to-use tools that inform him about problems and enable him to analyze the state of the system before and after failures in detail.

The nondeterministic nature of incoming data stream, which is a distinctive feature of HiVAT techniques, presupposes the need to have flexible scenarios of finding problems and to provide opportunities for the tester to customize them.

For a highly loaded system, the provision of an opportunity to collect full information is impossible without having the effect of measurement. The goal of the testing tool is to minimize its influence on the functionality and performance of the system under test.

The tool should provide access to comprehensive and aggregated data obtained as the result of executing current and previous test iterations. The below diagram (Picture 1) shows components that are necessary for the implementation of the above- mentioned requirements for passive test tools.

Spesial Features of Testing Tools - High level schema of the main components of a passive tool for testing trading systems

Pic. 1. High level schema of the main components of a passive tool for testing trading systems

2.2 Active Testing Tools

An active testing tool should be multi-functional. In other words, it should be able to connect to different tested environments using various protocols. Usage of fuzzing for distributed trading systems applies restrictions on the created messages [25] in order to let them get to the core and other internal components without blocking them at exterior gateways of protocol connection or at the initial stages of risk control modules.

Automated creation of test scenarios and their considerable volume require saving of information regarding the messages that have been sent and the internal states of the test tool as well as its settings. When complex techniques are used, the verification of data between the tool and the system under test is required.

The application of HiVAT-methods for highly loaded trading systems requires building a massive infrastructure for testing. The efficiency of using such an infrastructure can be achieved only when it is available for execution of different test tasks simultaneously. Test tools should give the possibility to execute tests manually and to schedule automated runs of test scenario suites even when regression test scenarios or tests based on random test generation are run concurrently and repeatedly.

In order to create complete functional tests, the tool should provide suitable programming opportunities for managing the process of generating automated test scenarios. The diagram below (Picture 2) displays the components that are necessary for implementing active test tool requirements mentioned above.

Spesial Features of Testing Tools - High-level diagram of the main components of a tool for active testing of trading systems

Pic. 2. High-level diagram of the main components of a tool for active testing of trading systems

2.3 General requirements for tools that test trading systems

The following characteristics are mandatory for passive and active tools when HiVAT methods are used:

  • scalability and high capacity;
  • resiliency;
  • adaptability;
  • ease of use and interactivity.

These mandatory characteristics match the requirements for production trading systems.


In this part, we will compare the requirements for tools that test trading systems with the help of HiVAT methods with the requirements for trading and surveillance systems used in production.


Table 1. Requirements for market surveillance systems
Requirement Production Use
Completeness of all system data collection The primary task of a market surveillance system is to support the analytics gathered and analyzed by departments responsible for recognition of possible market abuse [19]. A surveillance system must collect the information pertaining to all incoming orders, system responses, data from an external sources and relevant internal states of the trading platform.
Minimizing impact on the trading system Collection of a large amount of information is impossible without having a scalable monitoring infrastructure. Monitoring the market is a process of primary importance, which is mandatory in the majority of financial jurisdiction. Nevertheless, in order to provide minimal response to incoming messages in real time, the operators of trading systems try to avoid the negative effect of metrics on the main functionality of the trading platform.
Real time notification about non-standard situations System operators should be immediately notified if any technical problems or suspicious transactions occur. Such notifications are called surveillance alerts. An effective surveillance alert system is a key component of a surveillance system.
Flexibly customized criteria of pattern identification Quite often, bad-faith market participants try to hide the irregularities and market manipulations under the pretense of a legitimate financial transaction. Monitoring and market control systems have to draw conclusions based on fuzzy logic, where the parameters have to constantly be adapted to new trading situations and market participant behavior.
Data storage and access to historic data Upon the request of auditors or regulators, the operators of trading systems must provide historic data and the results of its processing.
The ability of tracing the sequence of events and the internal states of the system at a particular time Formal criteria by themselves are not enough to prove market abuse. System operators need to be able to restore the sequence of events related to a certain episode that triggered a surveillance alert. This function is called an order book replay. Customized implementation of this function enables the operators to investigate more events and make conclusions about the correctness of pattern identification mechanisms’ work.
High-level diagram with the main components of a market surveillance system created at Exactpro Systems, LLC

Pic. 3 High-level diagram with the main components of a market surveillance system created at Exactpro Systems, LLC.

The diagram on Picture 3 shows the conceptual proximity of the system and the tools for passive testing of trading platforms.


Table 2. Requirements for exchange and algorithmic order execution systems
Requirement Production Use
Versatility. Connection to multiple external systems and support of their protocols Due to financial markets fragmentation, broker systems must ensure connectivity to various exchange systems and external brokers [21]
Provision of concurrent access A high load trading system must provide concurrent access to a large number of trading participants
The opportunity to execute difficult scenarios The main function of algorithmic trading is to provide the users with the opportunity to set a strategy of sending orders to execute financial transactions depending on the market conditions, portfolio, risk parameters, information messages, etc.
Storage of information regarding all sent messages and internal conditions. The opportunity to verify this data with the data received from the external systems, including clearing and depositories The operator of a trading system proving access for his clients to financial markets must store the information about all executed financial transactions [22] and verify this data with the data of the clients and counterparties and with the data from post-trading systems.
High-level diagram of the main components of exchange and algorithmic order execution system created at Exactpro Systems, LLC

Pic. 4. High-level diagram of the main components of exchange and algorithmic order execution system created at Exactpro Systems, LLC.

The diagram on Picture 4 shows the conceptual proximity of the system and tools for active testing of trading platforms.


The comparison of requirements and concept diagrams of testing tools with the corresponding production systems allows affirming that once testing tools are mature enough, they can be used as a sub-system of a trading platform. However, the main purpose of testing is not finding accurately functioning subsystems, but the identification of issues and shortcomings in the application under test. The transformation of testing tools into production trading systems can potentially lead to most attention being given to correctly functioning segments, reduction of test coverage, and too much focus being given to positive scenarios. The main direction of further research is finding methods to overcome this tendency.

If testing tools become part of production infrastructure, the introduction of slight alterations into their code and settings is in fact an alteration to the trading platform they are part of. Thus, the proposed approach opens new opportunities to apply the system-level methods of mutation testing to the research of complex integrated trading systems [23]. The alterations that are introduced are called mutations, which are based on mutation operators [24] that imitate typical errors and undesirable effects. Mutations also allow estimating the efficiency of the existing set of tests and the quality of testing tools.

The following alterations can become operators of a trading system mutation for the verification of non-functional features:

  • Introduction of random delays into the internal components, logical algorithms and the functioning of external connections;
  • Replacement of optimized TCP/IP data flows by multiple parameterized libraries in different languages;
  • Filling the memory or disk space on the computer where the system under test is installed with a large amount of logs;
  • Loading the network with parasitic messages or error injection into the data structure.

The other direction of further research is to study fuzzing methods that are compatible with the structure and consistency of production systems [25].


Based on summarizing the authors’ experience with the verification of accurate work of high-load electronic trading systems from functional and non-functional standpoints, this article analyzes the methods of their quality assurance based on a high volume of automated testing (HiVAT).

The application of these methods in practice has allowed defining a set of main requirements for passive and active testing tools used to verify the correct work of such systems.

The conclusion is that a testing tool compliant with a certain set of requirements is in fact itself a system applicable in trading systems production.

The results obtained by the authors have proven that financing of testing tools development based on the methods and principles described above is justifiable. A market surveillance system and an algorithmic trading support system are being developed as part of Exactpro Systems, LLC projects. Both systems are of dual purpose and can be used both as a testing tool and an element of production trading infrastructure [26].

Additional opportunities for using system-level mutation testing methods for the analysis and expansion of functional and non-functional testing coverage were described above. These opportunities become available when testing tools are integrated into the trading platform.


  1. The Future of Computer Trading in Financial Markets. Final Project Report. / Foresight. The Government Office for Science, London. // [Electronic resource]. – Access mode
    http://www.bis.gov.uk/assets/foresight/docs/computer-trading/12-1086- future-of-computer-trading-in-financial-markets-report.pdf
  2. Bloomfield,R., Wetherilt,A.: Computer trading and systemic risk: a nuclear perspective. / Foresight. The Government Office for Science, London // [Electronic resource]. – Access mode
    http://www.bis.gov.uk/assets/foresight/docs/computer-trading/12-1059- dr26-computer-trading-and-systemic-risk-nuclear-perspective.pdf
  3. Commission Roundtable on Technology and Trading: Promoting Stability in Today's Markets. / U.S. Securities and Exchange, October 2, 2012 // [Electronic resource]. – Access mode
  4. Itkin, I.L.: Testing of High Load Trading Systems / HighLoad ++ Conference of high load systems developers // [Electronic resource]. – Access mode
  5. Penhaligan,P.: Equity Trading: Performance, Latency & Throughput. / ExTENT Conference // [Electronic resource]. – Access mode
  6. Avellaneda,M.: Algorithmic and High-frequency trading: an overview / New York University & Finance Concepts LLC Quant Congress USA 2011. // [Electronic resource]. – Access mode
    http://www.math.nyu.edu/faculty/avellane/QuantCongressUSA2011Alg oTradingLAST.pdf
  7. Millennium Exchange Technical Specifications / The official web-site of the London Stock Exhcange // [Electronic resource]. – Access mode
  8. Zverev,A., Bobrov,I, Pryadkina,N..: Testing of HFT GUI. / ExTENT conference // [Electronic resource]. – Access mode
  9. Yu,T.: An observable and controllable testing framework for modern systems. // ICSE '13: Proceedings of the 2013 International Conference on Software Engineering Publisher: IEEE Press, May 2013
  10. Netzer,R.H.B, Miller,B.P.: What are race conditions?: Some issues and formalizations // Letters on Programming Languages and Systems (LOPLAS) , Volume 1 Issue 1, March 1992
  11. Zverev,A., Bulda,A., Bobrov,I.: Trading Systems: Testing at the Confluence of FT&NFT. / ExTENT conference // [Electronic resource]. – Access mode
  12. Heider,W., Rabiser,R., Grünbacher,P., Lettner,D.: Using regression testing to analyze the impact of changes to variability models on products. // SPLC '12: Proceedings of the 16th International Software Product Line Conference - Volume 1, September 2012
  13. Eurodollars on CME Globex: Implied Price Functionality // [Electronic resource]. – Access mode
  14. Sutton,M., Greene,A., Amini,P.: Fuzzing: Brute Force Vulnerability Discovery. // Addison-Wesley Professional, 2007
  15. McGee,P., Kaner,C.: Experiments with high volume test automation. // SIGSOFT Software Engineering Notes, September 2004
  16. Kaner,C: An Overview of High Volume Automated Testing. // [Electronic resource]. – Access mode
  17. Teaching High Volume Automated Testing (HiVAT). //12th Workshop on Teaching Software Testing (WTST 2013), January 25-27, 2013, Melbourne, Florida, at the Harris Institute for Assured Information. // [Electronic resource]. – Access mode http://wtst.org/
  18. Cavalli,R., Montes de Oca,E., Mallouli,W., Lallali,M.: Two Complementary Tools for the Formal Testing of Distributed Systems with Time Constraints. //12th 2008 IEEE/ACM International Symposium on Distributed Simulation and Real-Time Applications.
  19. Diaz,D., Zaki,M., Theodoulidis,B., Sampaio,P.: A Systematic Framework for the Analysis and Development of Financial Market Monitoring Systems. // 2011 Annual SRII Global Conference
  20. Itkin, I., Pryadkina, N., Kruykov, A.,: Data Analysis in High Load Trading Systems / AIST-2013 Conference // [Electronic resource]. – Access mode
  21. The official web-site of Fidessa Fragmentation Index // [Electronic resource]. – Access mode
  22. OATS Reporting Technical Specifications. // [Electronic resource]. – Access mode
    http://www.finra.org/web/groups/industry/@ip/@comp/@regis/docume nts/appsupportdocs/p197473.pdf
  23. Mateo,P.R., Usaola,M.P., Alema´n,J.L.F.:Validating Second-Order Mutation at System Level. // IEEE Transactions on softvare engineering, vol. 39, No. 4, April 2013
  24. Mateo,P.R., Usaola,M.P., Offutt,J.: Mutation at System and Functional Levels. // Third International Conference on Software Testing, Verification, and Validation Workshops
  25. Wang,T., Wei,T., Gu,G., Zou,W.: Checksum-Aware Fuzzing Combined with Dynamic Taint Analysis and Symbolic Execution. // ACM Transactions on Information and System Security, Vol. 14, No. 2, Article 15, Publication date: September 2011
  26. Itkin,I., Matveeva,A., Barch,A.: Test Tools Evolution. / ExTENT conference // [Electronic resource]. – Access mode