This case study discusses the development of new techniques for the functional and non-functional testing of Distributed Ledger Technologies (DLT), in particular, for their application to mission-critical Financial Market Infrastructures.
Exactpro expertise encompasses:
- testing enterprise DLT platforms – we have extended our comprehensive functional and non-functional test automation capabilities to meet the new and exacting testing requirements of enterprise delivered infrastructure, whether it is delivered on-prem or in the cloud, or in multi-cloud deployments;
- building applications on DLT – building and testing workflow solutions based on the latest DLT technologies;
- network testing of notaries – applying existing tools and expertise to test the latency and throughput of a complex multi-geography, multi-node network.
There are many challenges in testing distributed ledger platforms, including the interoperability of nodes in the network and the large number of connection permutations between them, understanding the differences between varying DLT permissioning logic. For testing, this leads to hundreds of millions of tests – a problem only solved by automation.
Furthermore, the technology is new and evolving quickly, and projects tend to be driven by two main factors: taking POC software into production and ensuring that platforms meet existing regulatory standards.
Functional and Non-Functional Testing for Distributed Ledger Technology Platforms: Challenges
We have encountered the following challenges when testing DLT-based platforms:
- Complexity – DLT-based infrastructures are complex and vary in their architecture. They can be extremely heterogeneous, i.e. it can be possible to deploy different versions of an application on different security devices, different databases, different Java machines and operating systems. This creates a combinatorial explosion when one tries to test them.
- Interoperability – A number of issues arise at the confluence between a distributed ledger and its connectivity to a non-DLT platform.
- High transaction throughput – such platforms are expected to be performant. So, it is necessary to do quite sophisticated performance tests.
- DLT-based infrastructures can be highly distributed and multi-threaded; that requires security, they need to be consistent (e.g. it becomes consistent after a certain delay).
- DLT platforms evolve because there are continuous upgrades to nodes, applications, networks.
- High availability (HA) is a must. The instruments ensuring there is no double-spend need to be highly resilient.
Test Automation Framework for DLT projects
ClearTH – an innovative way to test Clearing, Settlement and Back-Office systems. It is a unique Exactpro tool able to simultaneously execute multiple End-to-End test scenarios in batches. ClearTH easily detects abnormal behavior in the system under test and effectively predicts potential issues. It offers many built-in actions to cover the majority of activities in post-trade systems.
- Verifies each stage of the DLC
- Has an integrated schedule
- Automatically runs test scripts
- Creates multiple-day test scenarios
- Performs multiple concurrent tests
- Has integrated simulators
- Supports SWIFT ISO protocol
Woodpecker – a ClearTH extension acting as a platform for daemons combining functional and non-functional test approaches.
- Highly customizable (data source, data types, load, time, interface)
- Easy to transfer to different environments due to its ability to extract data right from the system under test
- Automatically generated test scenarios cover a vast majority of diverse conditions/data combinations
- A productive intersection of functional and non-functional testing approaches
ClearTH Test Automation Framework Adapted to DLT Projects
- The same complexity of participants/processes/asset classes
- Hybrid nature: conventional functionality + new business logic involving interactions within a blockchain network
- High diversity of underlying DLT frameworks
- New form of business logic execution – smart contracts
- Ability to support messaging interaction
- Ability to emulate separate nodes
- Triggering workflow via REST API/RPC calls
- Protocol-level interactions simulation
- Ability to capture the data flow and perform ledger verification checks
Case study 1: Functional Testing for Distributed Ledger Technology Platform – Corda
ClearTH allows for:
- Nodes deployment with an ability to change the configuration
- CorDapps deployment
- Nodes Administration
- Nodes registration in the Network
- Trigger transactions generation, verification and distribution across the nodes
- JVM statistic gathering
- Log monitoring and analysis
- DB verification and monitoring (low-level ledger update validation)
Case study 2: Non-Functional Testing for DLT Platform – Corda
Non-Functional Testing of any platform is complex and requires robust configuration and may require preliminary development activities.
To run automated non-functional tests, it is required to monitor various metrics of the platform under test and its hardware usage. Having the option to start/stop/adjust load injection or other actions based on the current system state is also a must. The Exactpro automation framework allows to initiate flows across large-scale networks of nodes against a remote network. We use open-source tools to gather metrics and visualize the results.
Load & Stress Testing
Examples include – Determining the maximum throughput processing on the reference infrastructure with no chain history and one state per transaction. The PerfTest CorDapp and a specific custom Flow were used, for a known use case, minimised the variables and allowed us to report the values in a meaningful way.
In a DLT Network, TPS is a function of many dimensions (attachments, chain length, number of input/output states).
Here, we look to determine that there is no loss of service in transaction notarisation with a variety of test CorDapps. The Exactpro tool forced node failures within the network and ensured no loss in service or notarisation.
Initially, we determined the latency characteristics at set quanta of the maximum transaction processing rate with a no-chain history.
We then delivered a report on latency values with set input rates of 25, 50, 100, 150% of the maximum value, i.e. for each input rate report:
- Average latency per transaction processing (mean and 99th percentile)
- Average compute time (mean and 99th percentile)
- Latency and compute time distribution charts
Soak testing looks to evaluate the ability of a system to sustain load for prolonged periods of time. In the test, we setup a DLT environment on Microsoft Azure, with the multi-threaded RPC injector distributed across multiple global locations to initiate flows across large-scale networks of nodes against a remote network outside of the framework. Docker images were configured to run specific Corda enterprise versions, CorDapps and RPC clients for the instantiation of test flows. System monitoring was via open-source tools DataDog, Graphite (for JMX stats), and bespoke scripts for additional memory monitoring.
Here we determined the memory usage of each node with a variety of test Apps.
Two Apps were used to drive transactions through the network over an extended period of time, and a known input rate near to its maximum value. We monitored the memory usage of the node JVM with time while processing tens of millions of transactions (e.g. does it grow, is memory released, do nodes fail and/or restart, etc.).
Case Study 3: Distributed Ledger Technology in Position Update Business Flow
3 steps on the ledger side:
1st Initialisation of positions
2nd Trade creation and participant notification
3rd Positions update
Across each and every step, the test framework provides different methods to validate the endpoints, inputs and outputs.
The actual business flow steps and validation check items are transformed into the ClearTH matrix, after the execution of which the deviations from the expected behaviour can be analysed.
Case Study 4: PoC for IRS & CDS Swaps Lifecycle Module Based on CDM
ClearTH simulates a participant who submits a trade. The script sets the ledger time to the next fixing date and simulates floating rate fixings by the market data provider. Then it “time travels” to the next reset date (payment date) and generates derived events.
Model for test automation:
- A template with the swap contract info (swap type and structure) and lifecycle specifics (e.g. segregation of trades or multiple contracts by one member, event schedules, etc.).
- Spreadsheets with Input data: a Config file with different combinations of parameters values; reference and market data (e.g. floating rate fixings).
- Macros generating multiple scenarios in the form of csv matrices – scripts executed by ClearTH.
ClearTH for DAML-based dApps Working on Top of Different DLT Frameworks
- DAML – a smart contract language with the capacity to run on top of multiple blockchains.
- ClearTH triggers different life cycle events (e.g. cash flow generation, trade execution, novation, termination, etc.).
- Simulates potential actions of different parties involved in a business process (e.g. initiating transactions or accepting trades).
- The ClearTH framework performs checks of the transaction results (account balance totals, interest rates calculations, the status of the trade – proposed, accepted or applied).
Exactpro Awarded Innovation Accolade for Blockchain Project with R3
London, 12th May 2020
Exactpro, a leading software testing provider for financial market infrastructures (FMIs), is pleased to announce the receipt of QA Financial’s “Automation Technology Project of the Year: Exactpro and R3” award, in recognition of its work with R3’s Corda Enterprise Distributed Ledger Technology (‘DLT’). Exactpro delivers testing services underpinned by tools and proven methodologies as a result of a significant R&D investment. The project saw the extension of comprehensive functional and non-functional capabilities to meet the new and exacting testing requirements of the latest Corda Enterprise DLT release, itself forming the basis of a few significant new FMI-driven platforms.
"There are many challenges in testing distributed ledger platforms, including the interoperability of nodes in the network and the near infinite permutations of connections between them. It's a challenge that expands into hundreds of millions of tests, and automation is the only viable solution," added James Carlyle, CIO, Head of Production, at R3.
Exactpro’s ClearTH Wins Best Tool Demo Award
Exactpro’s ClearTH test automation framework for DLT-based Post-Trade Systems research based on R3 Corda technology wins Best Tool Demo award at the 12th IEEE International Conference on Software Testing, Verification and Validation in 2019.
The demo can be viewed at https://www.youtube.com/watch?v=eKFc9yKGfuI
Distributed ledger technology (DLT) innovation is an important trend shaping the financial services industry. It’s no surprise that under the pressure of ongoing overall digital transformation, many financial institutions are keen to adopt DLT in development of their platforms. However, when the innovation is introduced on a scale of a major capital market participant, its potential impact is likely to be shared by a larger financial services community. To facilitate the smooth transition towards the emerging technologies, the regulatory bodies require systemically important financial market infrastructures to ensure that while innovating, they also stay operationally resilient. Read more