Principles
- Low Start
-
- Don’t wait for the full comprehensive test harness to be available; use simple tools as soon as they are ready
- Start with getting tools to execute with partial validation & progressively add more validations
- Capture logs and executed files. Even semi-manual test execution provides better evidence and serves as the foundation for future scripting
- Passive & Active Testing
-
- Passive testing is a good way to collect test evidence from other tests and can be used to perform client certification
- Messages stored during passive testing can be used to generate active test cases and repeat previously executed activities
- Codecs, Drivers and Stubs
-
- Codecs are test code capable of parsing a particular protocol or file format; using these allows increasing testing depth, localizing defects and validating the components earlier
- A Driver is a test program that can initiate the injection of messages into the system
- A Stub is a test program that simulates an absent component called by the system under test
- Drivers and Stubs both rely on Codecs
- A Test Harness is a set of drivers and stubs for every interface link in the system
- An automated test library relies on the Test Harness to execute the test scenarios
- The “Big Button” Framework
-
- “Big Button” organizes large automated test libraries, so that they can be executed in an unattended mode
- It is necessary to carefully design reference/test data sets and plan operational schedule of the test environment
- Reduces effort required for regression test execution, but also allows having a better test library structure
- The Big Button approach requires more stability from the test tools and test scripts
- High Volume Automated Testing (HiVAT)
-
- Like other complex distributed real-time systems, Trading and Clearing platforms have intermittent defects that manifest themselves only if load is applied
- They can be attributed to race conditions or to rare internal integrity failures
- Such defects might not be spotted in the course of ordinary functional test execution due to their low statistical probability under low volumes
- HiVAT techniques are based on automated test case generation
Essential Ingredients
Highest calibre development staff on the team
Pragmatism and determination
Bespoke test automation tools built as an investment over the course of several QA projects, in combination with open source tools
Control of test environment back-end by QA and the ability to emulate external systems in order to set the required pre-conditions and increase test automation coverage
Experience
Type of Tests
- Transactional
-
- Processes incoming requests in real-time
- Set up controlled transaction data set
- Execute volume of independent test rapidly
- Independent test cases and is easily supportable
- Scheduling
-
- Tests aggregation and batches functionality
- Execute scenarios lasting multiple business days
- Execute many test scenarios in parallel
- Internal scheduler aligned with scheduler of the system under the test
- Reconciliation
-
- Verify system consistency by monitoring inputs and outputs
- Agnostic to test input source
- Complements active test approaches
- Participants
-
- Emulation of participants activity in electronic markets
- Reproduce complex scenarios
- Testing at the confluence of functional and non functional testing
- Verify system from participant's interfaces