Introduction: Non-Functional Testing and NFT Automation
Non-functional testing is validating the system against all the non-functional requirements expected from it. NFT covers a wide range of tests that can vary from the classic stress testing to testing at the confluence between functional and non-functional testing. Since the number of NFT tests in a typical project is enormous, any QA team will eventually start looking for ways to automate what they do. However idealistic, the idea of The Big Button seems favorable: the regression library is fully automated, execution time is short, QA pays minimum attention to the process and gets reliable results. Clients dream of it to deliver new features faster; the QA team strives for it to avoid routine; and QA leads - to divert the resources from manual regression to expanding the testing library. However, the complexity of designing an effective test automation suite is often highly underestimated. Some may find this idea utopian. It is because NFT as a process poses a number of challenges:
The Production-like/Stress Testing Dilemma
The Absence of Clear “Pass/Fail” Criteria
Analysis of the Results
Audit of the Automated Test Library
1. The Production-like/Stress Testing Dilemma
The process of test scenario creation is closely connected with the dilemma between production-like testing and stress testing. When a non-functional issue is raised, the business side will want to know if it is a production-like scenario to understand how severe the issue is and if it is likely to happen in real life. At the same time, it is senseless to only rely on the current production picture. QA specialists should be pessimistic and imagine all possible adverse scenarios. Thus, it is necessary to stress the system and take it outside the production-like limits to reveal the system’s break point. Stress testing also being a MIFID II requirement, it is very difficult to find a balance between keeping the tests production-like and testing with “room to grow”.
2. The Absence of Clear “Pass/Fail” Criteria
The constant attempts to keep this balance exacerbate the most significant problem for the NFT Big Button: there is no definite ‘passed/failed’ result. The same ‘passed’ test taken with an increased transaction-per-second rate can lead to a system crash and give a ‘failed’ result. Therefore, the process of interpreting the results to assign a “bug” status to an issue is challenging as well: there is often a fine line between an issue being a bug and a test being unrealistic. Another NFT peculiarity is that due to vagueness of most requirements, it is typical for occasional “failed” results to be accepted as expected system behaviour. These are the reasons why non-functional issues are perceived more like observation opportunities than things to label or categorize.
But even if a requirement, such as KPI, is clear and specific, it will vary for different systems and may be subject to change based on the production analysis. That’s why non-functional regression is not static: it is a process of continuous adjustment.
3. Processing the Results
Even after the ‘pass/fail’ criteria are defined, they still need to be confirmed with no manual assistance. Some test runs may produce up to 500 GB of network captures and application logs, and it is a challenge not only to process everything efficiently, but even to get storage space for it.
4. Audit of the Automated Test Library
Audit of an automated test library is a major non-functional testing endeavor.
It is crucial to audit the scenario before its implementation to be confident in the automated test. However, non-functional test cases are: a) often big, b) complex, c) impossible to divide into simpler actions, unlike a functional testing case. That makes auditing the overall scenario challenging. Because of the test case complexity, it is also hard to audit all the elements that the test consists of. Automated tests rely on the test data and the load profile to produce repeatable results. The best solution to audit that is to use standard generators that always give the same template and setup. Nonetheless, a new release may bring some changes, or the staff may change the test data/the system configuration. To tackle these challenges, data migration checks are introduced to protect the systems from unexpected release changes. As for firmware/hardware upgrades, it is recommended to re-run the testing after every change to get the latest baseline.
5. Conclusion: Exactpro’s Non-Functional Testing Tool Suite
The variety of NFT tests dismisses the possibility of getting things done with just one test tool: every specific test requires a separate testing approach and sometimes – a specific test tool.
Over the years, Exactpro specialists have developed load generators, consistency verification and reconciliation tools, automated execution tools and various system health checkers. Some are simple python and bash scripts which can be run manually or called automatically, and some are independent tools dedicated to a specific function, for example, failover testing or latency analysis. The possibility of The Big Button is still questionable, but Exactpro are well-equipped to lead the way to better automation ideas and solutions.
Anna Khristenok is Head of the Non-Functional Testing Department at Exactpro, LSEG.
Anna joined Exactpro as a Quality Assurance Engineer in 2012. She is currently leading Non-Functional testing for the London Stock Exchange, Turquoise and Oslo Børs. Anna graduated as a Master of Science from the Automation and Electronics Department of Kostroma State Technological University, Russia.