Date:
This article highlights some of the essential common challenges of FPGA-based solutions in terms of testing and elaborates on how Exactpro approaches testing them from the standpoint of functional and non-functional testing, and verifies the correctness of their work at the confluence of functional and nonfunctional testing.
It is a well-known fact that the FPGA technology is highly beneficial. Performance is one of the several notable key advantages of the FPGA-based products. Taking advantage of hardware parallelism, FPGAs exceed the computing power of digital signal processors (DSPs) by breaking the paradigm of sequential execution and accomplishing more per clock cycle. Another major FPGA benefit is throughput. FPGA has many high speed input/output pins to interface with memory, Ethernet, etc. It has dedicated channels for peripherals, unlike CPU, whose peripherals share a bus. Reliability is also a remarkable FPGA trait. Software programs are usually built upon several layers of abstraction, for instance, driver or OS, to help schedule tasks and share resources. They are at risk of incompatibility, resource contention, deadline violation, among other issues. On the other hand, the FPGA circuitry is a hard implementation, which is deterministic and well predictable.
The reasons of typical challenges encountered in providing quality of FPGA-based systems implementation.
The aforementioned advantages are truly remarkable, however, to achieve them, FPGA-based systems should be tested properly, and testing coverage should be adequate. This presents a difficulty because of a number of reasons:
-
Most of the sophisticated FPGA-based systems are designed in such a way that part of the algorithm is implemented on hardware and the rest of the logic is moved to software. This could be happening owing to the limitations of the selected FPGA-board, so the developers have to simplify the algo. It could also be, naturally, the overall goal that the project team works to achieve when developing a platform, for instance, being more interested in the performance characteristics of particular parts of the platform rather than the system as a whole and identifying acceptable combinations of various non-functional parameters. In certain cases, validation may be excluded from the software part, provoking a number of issues.
-
A challenge can also arise from the necessity to duplicate the code logic on both the FPGA side, and the software side.
-
Another concern is the high cost of building up the equipment for testing and live environments. And it is a real dilemma, whether it is absolutely necessary to set up another environment for QA. Should the decision be negative, the QA team will have less time and fewer fields for being flexible.
Typical challenges in ensuring the quality of the FPGA-based systems implementation.
Cost efficiency of the environment is by far the main testing issue for the FPGA-based products. Needless to say, the major contributing factor of a successful QA process is a sufficient number of test environments. This is a challenge for testing activities for an FPGA-based product due to the high cost of FPGA boards. Maintainability and operability represent another issue. The existence of an FPGA board as a part of the application's infrastructure also implies the presence of additional hardware/software components for monitoring and deployment purposes, which, in its turn, complicates the maintainability and the operability of such environments. Test case development is a challenge too: it is an extremely complicated process to develop a proper set of test cases with a convincing level of coverage, due to a hybrid nature of the application which has FPGA-based components.
Issues encountered when testing FPGA-based systems can be fascinating.
What are the typical issues that are common for hybrid solutions? There are in fact two main streams where issues exist, the first type of issues possibly stemming from the duplication of logic in the software and FPGA. Depending on the situation, the core of the problem could lie in different data type representation in the software and the FPGA, data structure implementation, such as layouts, the layers of internal algo paths and so on; it could also be the fact that the same logic path is duplicated in both segments: the software and the FPGA.
A whole other type of issues can be encountered in the processing flow. When a control part is on the software side, this could, somehow, slow down the processing of the hardware queue events. To avoid such bottlenecks, developers usually simplify the communication between the software and the FPGA. Subsequently, certain validation parts on the software are excluded, which, in some cases, leads to defects.
Some fascinating issues could be related to the hardware resources, used for the internal algorithms. For example, some FPGA-boards have their own clock timing source and can use it for processing, while, at the same time, the software relies on hardware clocks.
Ways to tackle quality assurance challenges. Test environment availability.
Unfortunately, from the QA standpoint, there does not seem to be an easy solution. If FPGA is a part of the application, test environments with enough amount of FPGA on board have to be used, otherwise there is no way of proving the quality of the product under test. This should be kept in mind while planning the budget. As a possible workaround at early stages of testing, aimed at proving the functionality, low cost analogs of the FPGA board, selected for the project, can be used. However, it is paramount to make sure that the chip of such a low-cost analog performs adequately and is capable enough to provide at least the same level of functionality.
What else can be considered an enhancement for FPGA-based solution process flows? A redesign of the implementation part may be needed: either based on the lessons learned, or by redesigning the product, keeping in mind the aforementioned challenges. One way is to exclude the logic duplication in the software and the FPGA or design the system in such a way that it is fully on FPGA. This, of course, would make the development process more complicated and challenging.
In order to reduce the impact that the presence of the FPGA in the ecosystem of the application has on maintainability and operability, first of all, it is recommended to make the firmware upgrade process as easy and transparent as possible. As for complex FPGA-based applications, a big improvement may be made by providing a set of automated deployment applications and avoiding the transition of servers into a special programming mode. Consider a possibility to keep several FPGA firmware versions on the same card, with an option to switch easily between them. This could allow reducing timeframes required for rollback of the firmware under test. Ideally, having a single version of the operating system driver will reduce the time, the complexity and the number of different support teams required for deployment and rollback activities in a test or a live environment.
