Research Papers

Black-Box Testing of Financial Virtual Assistants

We propose a hybrid technique of black-box testing of virtual assistants (VAs) in the financial sector. The specifics of the highly regulated industry imposes numerous limitations on the testing process: GDPR and other data protection requirements, the absence of interaction logs with real users, restricted access to internal data, etc. These limitations also decrease the applicability of a few VA testing methods that are widely described in the research literature. The approach suggested in this paper consists of semi-controlled interaction logging from the trained testers and subsequent augmenting the collected data for automated testing.

Read more

Toward reducing the operational risk of emerging technologies adoption in central counterparties through end-to-end testing

Emerging technologies, such as artificial intelligence (AI) and distributed ledger technology, are increasingly being adopted by financial institutions, promising functional efficiency and cost reduction to stakeholders and users. However, the structural and functional changes associated with the technological transformation of software platforms pose significant operational risks. While some aspects of these risks are well known and studied (such as AI trustworthiness, data privacy and consistency, platform availability and information security), others are underestimated. The extreme complexity and nondeterministic nature of existing technology infrastructures still need to be addressed, as they will soon be inherited by the platforms built with new technologies. The only way to mitigate these risks is extensive endto-end professional testing. This paper discusses the software-testing challenges of traditional central counterparties as well as the risks, biases and problems related to new technologies. It also outlines a set of requirements for an end-to-end validation and verification solution aimed at the new generation of clearing platforms.
Focusing on one of the most common use cases in the capital markets industry, this paper considers the challenges posed by the introduction of blockchain and AI into the post-trade area.

Read more

Building a Classification System for Failed Test Reports: Industrial Experience

Running complex test suites against a financial transaction system produces huge amounts of responses, both expected and unexpected. In this article, we outline our experience of using ML for reliable automatic extraction of "that" unexpected response from a big number of same type messages produced a by system under test. We describe classification approaches and data manipulations we have tried, and explain the final choices. Also we outline business constraints and final design decisions for the resultant tool. We also address the task of classifying difference patterns between expected and actual responses in attempt to provide automated pre-judgement on a reason for test failure. We outline clustering considerations and results achieved.

Read more

Raising the Quality of Bug Reports by Predicting Software Defect Indicators

The descriptive quality of bug reports is one of the essential parts of defect management. Sometimes they provide inadequate or incorrect data about software problems, which can lead to incomplete defect fixing or omission of serious defects. Therefore, it is vital to evaluate and improve the quality of bug reports. This paper proposes an approach that helps to resolve this problem by predicting various indicators. The values of these indicators allow QA engineers to evaluate the quality of defect description and correct it in a suitable way. This paper also introduces Nostradamus, a new open source tool built to implement the approach. The tool uses machine learning techniques to analyze the data stored in software defect repositories and evaluates the interdependence of defect attributes, including such a crucial element as a defect description. This paper describes the approach, the tool that is based on it and the typical use cases.

Read more

Development of Intelligent Virtual Assistant for Software Testing Team

This is a vision paper on incorporating of embodied virtual agents into everyday operations of software testing team. An important property of intelligent virtual agents is their capability to acquire information from their environment as well as from available data bases and information services. Research challenges and issues tied up with development of intelligent virtual assistant for software testing team are discussed.

Read more

Analysis of the Characteristics and Causes of Underestimated Bug Reports

The bug-fixing process requires software maintenance resources. Usually, defects are submitted, fixed and closed, but sometimes they have to be reopened because of a change of resolution. It happens when a defect was evaluated incorrectly at the beginning. This problem can increase maintenance costs and software quality in general. In this paper, we investigate the characteristics of such defects and their bug reports and call them “underestimated”. Our research is based on general statistical indicators and text descriptions of defect reports. We propose using different methods of feature selection and ranking in order to reveal the significant terms of such defects. The top of significant terms of the underestimated bug reports can help to find the root causes of such life cycles.

Read more