The FDA once performed an analysis of 3,140 medical device recalls conducted between 1992 and 1998. They found that 242 of these recalls – nearly 8% – were due to software failures. In addition, 79% of those failures were caused by bugs and other defects introduced in software updates after the device’s initial production and distribution.
What does this mean? It’s critical to invest in rock-solid testing not only during initial product development, but continuously over the life of the product as well. In this article, we’ll talk about some of the top factors to consider when deciding on a testing strategy for your medical software product.
Note that when we say software, we mean every kind of software that interacts with your device or product: it may be the product itself (for example, a blood test app) or a component or accessory of the product – or it may be internal software you use to develop and manufacture your product, or your internal quality system software. Whatever your product is, if it needs FDA or CE approval then it needs rigorous testing.
Planning
Good testing begins at the planning stage. One major factor in planning is to perform hazard and risk analysis in order to understand the Level of Concern of every part of your software. What does that mean? The FDA describes Levels of Concern as based on how the operation of the software affects the patient or operator of the software. How severe is the effect of the software? Is the effect direct or indirect?
For example, one product we worked on at matrix Medika involved a pacemaker, while another device had an integrated temperature sensor. Both of these elements were important. However, a pacemaker failure was likely to affect the patient much more acutely than a temperature sensor failure – even to the point of fatality. As a result, we assigned the highest Level of Concern to testing pacemaker-related modules, while temperature-sensor modules were tested at a lower (though still high) priority.
Another aspect of planning is understanding how your software design can be broken up into discrete units for testing. Unit testing is a major element of a solid testing plan, and goes hand-in-hand with testing automation, which we’ll discuss further on. Unit design is also closely related to hazard analysis: you’ll want critical components with high Levels of Concern to be separate from lower LoC components, so that you can test them separately. Units should be as targeted and specific as possible, to allow you to test components individually.
Your testing strategy will also be affected by your software development methodology. Are you using Agile or a less flexible approach? Are you planning to release updates twice a month or twice a year? Understanding your requirements will help you design the testing plan that works best for your methodology.
Implementation
Now let’s move on to the implementation stage. At this point, you’ll need to decide how much of your testing will be manual and how much will be handled by automated testing suites. There’s no one-size-fits-all answer here, but it’s important to note that the FDA strongly encourages automated testing, especially for software with update releases. This is because automated testing can more reliably identify problems in complex software and software updates than manual testing, which is more prone to human error. In addition, it can be far faster to run automated tests, which means they can be run much more frequently – a vital benefit in the face of constantly changing cybersecurity threats. Overall, automated testing is a time- and cost-effective way to increase confidence in your continued compliance with regulatory standards for software quality.
Another decision to make is who will be responsible for testing. Will you have a dedicated QA team, or will developers be responsible for testing their own code? Alternatively, will you choose to take advantage of third-party service providers or off-the-shelf testing solutions? Often, working together with specialists in testing automation, such as Matrix Top-Q, is an effective way to ensure that you’re using the latest industry-standard best practices in a way that is most effective for your company.
Whatever you decide, you will want to make sure that the resources you assign to testing a given unit are proportional to its Level of Concern. Any code whose failure poses a risk to patient safety or health needs to be tested as thoroughly as possible. Code whose failure mode is less severe may be given a lower priority. Again, this is one of the reasons that division into testing units is so important.
What makes medical software development unique is the need to adhere to regulatory requirements both for testing and for documenting your testing. The IEC 62304 standard defines these requirements in both Europe and the United States, and adhering to it is critical for any product seeking FDA or CE approval. Today there are tools that work together with automated testing tools to generate the necessary documentation – another major advantage for automated testing. Working with development experts like matrix DevOps can help you decide which automated workflows can add the most value for your company or project.
Conclusion
Testing is a major element in medical software development today, and performing effective, rigorous testing in compliance with international requirements is one of the main challenges that any medical developer needs to address. At the same time, development companies do not need to deal with this challenge alone. Working with specialist providers in automated testing such as matrix Top-Q and matrix DevOps, or with experts in end-to-end medical software development, like matrix Medika , is a way to ensure that your product is professionally developed and tested to the highest industry standards – and has a smooth path to FDA/CE approval.