EXTENT-2016: Non-functional testing approach

Load Injector non-functional test tool

Item Description Exactpro Test Strategy Step(s)

Load generation in trading systems

Testing Type

Active Load and Non-Functional Testing

Target SUT

Trading Platforms, Market Data Delivery and Post-Trade Systems and their combinations

SUT Interface

• Back-end (typically connected to message gateways / APIs);

• Data streams generation (mcast/ucast);

• GUI Testing Capabilities not supported

SUT Interaction Method

Inputs and outputs are generated based on the configured load shapes, parameters and templates. Captured messages can be viewed and analyzed post-factum using the DB queries (Shsha) or/and performance calculator tool (also developed by Exactpro)

Protocols

Extant plug-ins for Industry-standard (FIX and dialects, FAST, ITCH, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request

Test Scripts

Capable to stress the system with a high rate of transactions including microbursts. Used for Throughput, Bandwidth, Latency tests. Can be used for support of fault tolerance (Failover) tests

Test Management, Execution and Reporting

Simulation of multiple client connections with specified load shape for each connection or group of connections (configurable number of connections, messages templates, Load Shape for each connection or group of connections, messages distribution for each connection or group of connections). Throughput up to 80000 msg per core per second. Simulation of market data streams with required SLAs

Platform requirements

Linux on 64-bit platform

Primary Competitor

VeriFix TestPilot, HP Load Runner, IBM Rational Performance Tester, JMeter, Yandex.Tank

NFT tests

• Capacity test

  • Check all external interfaces:

    • Traiding FIX

    • Trading Native

    • PostTrade

    • DropCopy

    • ITCH Replay/Recovery

  • Check hardware resources consumption by system components:

    • CPU

    • Memory

    • Disk

    • Network

• DLC test

  • Put the system through Production like schedule:

    • All trading cycles

    • All scheduled sessions

  • Apply appropriate load during various cycles to trigger extensions, special sessions, etc

  • Perform some functional tests under load

  • Data consistency check

    • All expected messages are disseminated via ITCH during session transitions

    • All Downstream systems connected and confirmed daily checks

• Failover tests

  • Application failovers

  • Hardware outages

• Application failovers

  • Simulation of various outage types (SIGKILL, SIGSTOP, Stop via SysGuard)

  • Failovers during different system state (at startup / trading day / EOD)

• Hardware failovers

  • Network equipment failovers (Switches, Ports, NICs) - impact on Application

  • Server isolations

  • Oracle instances

• Rapid user actions tests (connect-disconnect, logon-logout)

  • The system should withstand such user behavior

  • HW resources consumption should not grow up

• Slow consumer tests

  • The system should handle such users and should have protection against them

  • HW resources consumption should not grow up

• Intensive usage of recovery channels

  • The system should be able to handle a large number of requests on recovery channels and should be able to satisfy them

• Massive actions from Market Operations (mass order cancels, mass trade cancels, mass instrument halts)

  • The system should handle Market operations' actions like mass cancel of 10k active orders or trades