Sailfish - Thorough Test Automation for Trading Systems
- Sends a message, captures the response and produces a report with expected vs. actual result comparison
- Allows concurrent automated runs of multiple scenarios overnight so that reports can be checked in the morning
- Uses versatile and independent scripts which are adaptable to different systems
- Has flexibility to expand or customize functionality depending on the clients' needs
- An open-source test tool (GitHub download here)
About Sailfish
Sailfish is an active real-time tool used for exchange, MTF and broker systems testing. It can also be used as an exchange simulator for testing post-trade systems. Sailfish is a web-based application which supports simultaneous connections of multiple users for both incoming and outgoing streams. It is written in Java.
Sailfish
- Tests Order Entry, Market Data and Post-Trade connections in one test scenario
- Executes test scripts in any order
- Simulates multiple user connections
- Simulates servers
- Stores all messages in a database
- Generates test reports
Can Sailfish Be Customized?
You are welcome to make the best of the modular structure of Sailfish: build on the shared framework with the plug-ins that you need, or have us make a bespoke plug-in for your system requirements. Every extension is used independently for each protocol version. You can go with the industry-standard protocols, such as FIX or SWIFT, or proprietary ones.
Sailfish is a simple keyword-driven test tool. The test libraries we developed for it include tests for various business contexts that range from Regulated Markets to MTFs, from Dark Pools to Clearing Houses and Brokerage Systems. They have been successfully implemented in a wide range of technical and middleware infrastructures.
Test Automation
Being a back-end tool, Sailfish is connected to gateways/APIs capturing trading and/or market data traffic.
Sailfish makes it possible to achieve fully autonomous scheduled test execution that does not require ongoing monitoring.
Sailfish Test Script
Sailfish test scripts are Comma Separated Values (.CSV) files, where each column corresponds with a separate message field.
Different test scripts are independent from each other and can be run in any sequence.
Each line in the spreadsheet contains an action that Sailfish performs in the system (e.g. Send, Receive, Verify, Execution Report)
The Test Execution Report
The test execution report reflects the difference between the expected outcome and actual behavior. If there are no discrepancies in the actions, the test is passed. If at least one action is failed, the whole test-script is failed. The report is transparent : you can easily identify which actions are failed and why.
Exactpro Active Real-Time Testing Tool Sailfish
Download
Characteristics of Sailfish
- Capacity & Precision
- Throughput 40 transactions per second with validations / 800 in performance mode. Time precision ~25-50 ms
- Testing Type
- Active Real-Time
- Target SUT
- Trading Platforms, Market Data Delivery and Post-Trade Systems
- SUT Interface
- Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities supported via plug-ins to other tools (e.g., Selenium)
- SUT Interaction Method
- Message injection and capture for testing of real-time low-latency bi-directional message lows; DB queries for data verification
- Protocols
- Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Reuters, Fidessa OA, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request (codecs are shared between Sailfish and Shsha)
- Test Scripts
- Human-readable CSV files; scripts generated manually by test analysts or automatically by test script generator using results of passive testing performed by another tool (e.g., Shsha)
- Test Management, Execution and Reporting
- Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, consecutive execution of multiple planned scripts, test results summary and detailed test reports. REST API supports remote control of Sailfish instances. Optional Big Button framework is supported.
- Platform requirements
- Low footprint cross-platform application, MySQL or other RDBMS.
Read More
Dynamic verification of input and output data streams for market data aggregation and quote dissemination systems (Ticker Plant)
Market data aggregation and quote dissemination systems (such as Ticker Plant) are widely used across the electronic trading industry. A Ticker Plant is responsible for distributing information about multiple execution venues over a normalized protocol. This paper presents a dynamic verification approach for such systems. Based on a set of programs developed by the authors, it allows processing large data sets, including those collected during non-functional testing of trading platforms and using them in real-live production. The paper also outlines benefits and shortcomings of the selected approach for real-time and historical transactions analysis.
Alyona Bulda, VP, Senior QA Project Manager, Exactpro, LSEG
Maria Orlova, QA Lead, Technology, FTSE, Exactpro, LSEG
Trading Day Logs Replay Limitations and Test Tools Applicability
This paper is an experience report on replaying full trading day production log files for dynamic verification of securities exchange matching engines. Three types of test automation tools developed in-house are described along with their characteristics. The paper analyzes various approaches to reproduce processes and scenarios observed in the systems during their production usage. The applicability and limitations of these approaches are also considered. The authors point out that for most complex distributed real-time trading systems it is extremely difficult to obtain an identical behavior using production logs replay via external gateways. It might be possible to achieve this by implementing additional instrumentation inside the exchange system’s core. The authors assume however that such an intrusion has limited value and should not be prioritized
over other, more appropriate, test design methods for testing such systems.
Pavel Protsenko, Account Management, Exactpro
Anna Khristenok, Head of NFT, Exactpro
Anna-Maria Lukina, QA Team Lead, Exactpro
Andrey Alexeenko, QA Team Lead, Exactpro
Tatiana Pavlyuk, NFT QA Team Lead, Exactpro
Iosif Itkin, CEO, Exactpro
Special Features of Testing Tools Applicable for Use in Trading Systems Production
The paper examines basic requirements for tools developed for verification of correct work of electronic trading systems by applying High Volume Automated Testing (HiVAT) methods and analyzes the applicability of such tools during production operation of trading systems.
Usage of Exchange Simulators and Test Exchanges as Tools for Ticker Plant Systems Testing
Ticker Plant systems are widely used in modern day trading. They allow collecting in real-time quotes from several markets, present the data in a unified format, and disseminate it electronically depending on requests and goals of external clients, traders. This paper presents a view on the possibility of using market simulators for testing such systems. A set of main functional and non-functional test scenarios required to control the quality of quote dissemination has been identified. A comparison of market simulators and real test markets has been presented.