Date:
FT focus on:
-
Test scenarios are based on functional requirements
-
Test tools which allow to run simplistic tests such as to send an order -> to verify a response
-
Prepare a functional library starting from simple scenarios and adding more complex ones step by step
-
Test scenarios are automated
-
A test Library is enhanced as we proceed with its testing and production clients scenarios
-
An automated library will be reused as a regression test library which is executed against each release after go live and can be taken for consequent projects
NFT focus on:
-
Testing is based on non-functional requirements
-
Load scenarios are driven by the technical characteristics in PROD.
-
The main target of the tests is to assess non-functional requirements: latency, capacity and fault tolerance
-
A lot of efforts are invested in correct latency measurements and determining conditions when the system breaks
-
Further regressions are done to compare system performance indicators with the previous releases
What do we miss when we keep FT and NFT separated?
-
Any complex system must sustain a certain level of concurrency. Some issues appear when several events happen simultaneously (Race conditions).
Example 1: Clients send a CxlReplace request and approx. at the same time the ME sends a fill back.
These 2 events meet at the Order Management server, and the state of the order can get corrupted.
-
Production order books and various system object states are more complex than in our FT scripts, and no amount of test cases can cover all possible scenarios
Example 2: A User tries to hit two different implied orders coming from the same parent order which is being restated in the middle of the process.
-
Components may behave differently under the load
Example 3: The System must support various throttling limitations and the requirements of the system behaviour. The correctness of functional system behavior can be tested only under the load.
-
Integer memory overflow
Example 4: Turnover (VWAP) fields in the statistics messages and incremental OrderID overflow.
-
Dynamic Mass events which happen during DLC
Example 5: Market Open, Market Close, Auctions, ... The state of the system at exact instance when the system goes from one session to another should be verified with the load applied against the system.
