With the design and prototyping of exciting novel technologies dominating the spotlight in product development, medical device verification and validation (V&V) activities sometimes don’t get enough attention during project planning.
V&V is essential to successful medical device development, regulatory submission, and commercialization. However, if not correctly accounted for, it can quickly derail project budgets and schedules. Below are some common hidden costs themes in medical device verification and validation along with some of the mitigation measures we employ.
Supporting Activities
Hidden Costs: Execution of V&V testing was included as an item in the Gantt but the plethora of supporting activities needed to prepare for, and document, V&V were not anticipated.
Mitigations: During planning, it is important to understand and detail out all the activities that lead up to, and enable, the execution of V&V. Forgetting to breakdown all the activities involved in V&V exposes the project to significant budget and timeline risk. In fact, the preparation for V&V often far exceeds the effort to conduct the testing itself.
Activities to account for in V&V plans include:
- writing test protocols, test forms, and reports
- confirming design input traceability
- procuring parts for, building, and performing QC testing on the test units
- developing test software to enable the testing
- designing and validating test fixtures
- executing dry runs
- communicating and coordinating logistics with test houses
- analysing data
If usability validation is required, also account for:
- recruiting participants
- building or coordinating the study environment
- travel to test sites
Scope
Hidden Costs: The process of executing V&V felt bloated. It was much more arduous and lengthy than initially planned due to the number and rigor of tests.
Mitigations: To limit the scope of V&V and reduce its burden, it is important to understand that not all requirements need to be tested. In fact, V&V can be fulfilled by means of inspection, demonstration, analysis, or testing. By identifying the appropriate V&V method, sample sizes and level of documentation can be reduced, saving time and decreasing the quantity of units needing to be built. More on this can be found in this blog on Statistical Techniques for Streamlining Medical Device Design Verification.
3rd Party Testing
Hidden Costs: 3rd party testing (EMC, IEC 60601, biocompatibility, shelf-life, sterilization, etc.) took longer and cost more than anticipated due to larger than expected sample sizes, challenges with the tester’s operation of the system, and misinterpretation of the system or applicability of standards.
Mitigations: Early understanding is key for 3rd party testing. Talk to candidate test houses during V&V planning to understand how many units/samples are needed, how long the tests take, and in what order they should be conducted. It is not uncommon for test houses to booking weeks or months out. Confirm when they have availability and how much the tests cost.
In some situations, the test house can provide guidance on which tests should be performed and which are unnecessary. Understand which tests are destructive to inform the number of units to build. Determine what parts of the system are critical for testing and look for opportunities where parts can be omitted or replaced with placeholders to reduce cost.
For some testing (e.g. EMC), plan on having a member of the design engineering team travel to the test house to provide in-person system and troubleshooting support. The engineer should have an expert understanding of the standards being tested to ensure the test house is applying the standards correctly to the device, and interpreting the data and results accurately. Inconsistent application of testing or misinterpreted results can get flagged by regulatory bodies during submission resulting in significant delays in commercialization.
Failures
Hidden Costs: The device failed parts of V&V. As a result, elements of the design needed to change, more units needed to be built, and re-testing needed to be performed causing significant schedule and budget impacts.
Mitigations: Although some failures in formal V&V are unpredictable, it is possible to significantly de-risk V&V and avoid having to repeat costly efforts. Informal dry runs or confidence testing of high-risk elements of V&V are recommended to quickly bring design deficiencies to light. Dry runs also enable incremental improvements to test protocols that would otherwise be difficult to change in the formal run.
Focus on tests that would have significant project impacts if they were to fail in formal V&V. Informal testing does not require the same level of rigor in documentation and execution as formal V&V. It can be done at a fraction of the cost and time, allowing for repeat attempts and design improvements. When appropriate, minimize design and change controls during informal testing to reduce the burden on any required design changes. Avoid initiating formal V&V without a high confidence of success.
Lastly, have timeline and budget contingencies in place to account for unexpected failures. Medical device development is iterative and findings during V&V are not uncommon. Establish a common understanding among stakeholders that findings could result in more work.
Conclusion
Successful completion of V&V is a huge win for any program, but the activity can produce unexpected schedule slips and missed budget targets. To minimize the risk of incurring hidden costs, remember to account for all the supporting activities that enable the V&V execution, identify the most appropriate V&V method for each requirement, reach out to 3rd party test houses early, and de-risk formal V&V by first performing informal dry runs.
Image: 135034385 © Wrightstudio | Dreamstime.com
Matthieu Lemay is a Systems Engineer in Product Development at StarFish Medical. Matt holds an architecture degree from the University of Waterloo and a BASc in Biomedical Mechanical Engineering from the University of Ottawa. He draws on his design background when working on early-stage medical devices.
