Feature Analysis of Estimated Causes of Failures in Medical Device Software and Proposal of Effective Measures
•
2 likes•1,309 views
Background/Objectives <research> •383 medical device software failures from 1983 to 1997 are analyzed in “FAILURE MODES IN MEDICAL DEVICE SOFTWARE:AN ANALYSIS OF 15 YEARS OF RECALL DATA” (DOLORES R. WALLACE1 and D. RICHARD KUHN, 2001) •Medical devices are getting more and more relying on software. <objectives> •Analyze 86 medical device software failures in FY 2010 and identify the change of the trend of failure causes from the former analysis •Confirm whether the International standard issued in 2006, IEC 62304 (Medical device software - Software life cycle processes), is effective to new failure causes or not.
Feature Analysis of Estimated Causes of Failures in Medical Device Software and Proposal of Effective Measures
- 1. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Feature Analysis of Estimated Causes of Failures in Medical Device Software and Proposal of Effective Measures Seiko Shirasaka The Graduate School of System Design and Management, KEIO University Yoshio Sakai Engineering Promotion Center, NIHON KOHDEN CORPORATION Yasuharu Nishi Department of Systems Engineering, The University of Electro-Communications
- 2. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Background/Objectives Research Background • 383 medical device software failures from 1983 to 1997 are analyzed in “FAILURE MODES IN MEDICAL DEVICE SOFTWARE:AN ANALYSIS OF 15 YEARS OF RECALL DATA” (DOLORES R. WALLACE1 and D. RICHARD KUHN, 2001) • Medical devices are getting more and more relying on software. Objectives • Analyze 86 medical device software failures in FY 2010 and identify the change of the trend of failure causes from the former analysis • Confirm whether the International standard issued in 2006, IEC 62304 (Medical device software - Software life cycle processes), is effective to new failure causes or not. 2
- 3. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Recall Data • 383 medical device software failures from 1983 to 1997 • The paper written by DOLORES R. WALLACE1 and D. RICHARD KUHN, 2001 3 • Analyze 86 medical device software failures in 2010 From • Pharmaceuticals and Medical Devices Agency (Japan) Recall Information • FDA（USA） Medical Device Safety • Health Canada（Canada） Medical Device Recall Listings • BfArM（Germany） News -Field Corrective Actions • MHRA(England) Field Safety Notices for medical devices • Swiss medic(Swiss） Archive "Field Actions“
- 4. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Fault class distribution • FAILURE MODES IN MEDICAL DEVICE SOFTWARE:AN ANALYSIS OF 15 YEARS OF RECALL DATA (DOLORES R. WALLACE1 and D. RICHARD KUHN) Fig.3. Fault class distribution 4 Fault class distribution From 1983 to 1997 Fault class distribution 2010 383 86
- 5. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 MEDICAL DEVICE SOFTWAR Fault class Changes 5 Fault class distribution From 1983 to 1997 2010 calculation 24% 7% change impact 6% 0% CM 1% 3% data 5% 2% fault tolerance 1% 21% Initialization 2% 0% interfaces 2% 14% logic 43% 26% omission 3% 0% other 3% 20% quality assurance 3% 0% requirements 4% 0% timing 3% 7%
- 6. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Feature of Fault class Change • “Calculation” and “logic” decreased, “fault tolerance” and “other” increased. • Reason of “Calculation” and “logic” decrease – Diversity of medical device system components cause the diversity of fault causes and the rate of “Calculation” and “Logic” decrease relatively. – Causes of faults are getting complicated. It is difficult to categorize to one fault class. • Reason of fault tolerance increase – Increase functions to detect hardware random failure by software or to control risks by software • Reason of “other” increase – Increase recall caused by IT/Network, Usability and Database • Increase the functions to realize user interface by software, to utilize data through IT network and to manage data using database. 6
- 7. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 FY 2010 Analysis Result :Medical Device Category 7 • There are many medical device categories. • Medical device categories which has more than one recall are listed.
- 8. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Feature of Fault class Change • Calculation and logic decreased, “fault tolerance” and “other” increased. • Reason of “Calculation” and “logic” decrease – Diversity of medical device system components cause the diversity of fault causes and the rate of “Calculation” and “Logic” decrease relatively. – Causes of faults are getting complicated. It is difficult to categorize to one fault class. • Reason of fault tolerance increase – Increase functions to detect hardware random failure by software or to control risks by software • Reason of “other” increase – Increase recall caused by IT/Network, Usability and Database • Increase the functions to realize user interface by software, to utilize data through IT network and to manage data using database. 8
- 9. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Detail Causes in “Other” 9
- 10. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Detail Causes in “Other” 10 Central Monitoring Station stops monitoring functionality unintentionally when the following condition occurs simultaneously. 1) Multi terminal monitors are connected to central monitoring station through network and one or more terminal monitors are set to display patient data which comes from other terminal monitor. 2) The terminal monitor which provides patient data stops
- 11. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Detail Causes in “Other” 11 If the buttons of a certain medical device is pushed for more than 5 second, its mode is automatically changed to monitor mode. And in this mode, a user can not perform its functions.
- 12. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Detail Causes in “Other” 12 If a picture data is stored as the same file name at “picture archiving and communication system”, the picture data of a patient overwrites the picture data of other patient.
- 13. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Analysis Results：Feature of Probable Fault causes (2010) 13 Systematic Software Failures / Faults rate
- 14. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Trend of fault and probable cause • Most software faults are Systematic Software Failures. (About 84%) • Most common probable cause is lack of Validation which is related with environment and usage situation of medical device. Its number is 53. • On the other hand, there are some hardware random failures, hardware systemic failures and timing caused failures. Their numbers are 12, 8 and 3. 14
- 15. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Medical device software related international standards which are newly relieved or revised after 2003 15 Risk Management (2003) Software Processes (2006) Usability (2007) IT/Network (2010)
- 16. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Medical device software related international standards which are newly relieved or revised after 2003 16 Risk Management (2003) Software Processes (2006) Usability (2007) IT/Network (2010)
- 17. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effective counter measure for recurrence prevention Probable Cause Feature of Fault and Probable Cause No. Effective Counter Measure Internal Hardware Hardware Random Failure 12 <-> 1. Statistical Bug Analysis, Risk Control by Software Hardware Systemic Failure 3 <-> 2. Hardware Design, Design Process Management Internal Software Systemic Failure/Faults Consequence Analysis required Failure 18 <-> 3. FMEA Configuration Management Mistakes 3 <-> 4. Software Configuration Management, Change Management, Regression Test Software Systemic Failure 72 <-> 5. Software Process Management Interface/Timing Interface caused failure 16 <-> 6. Input Simulation Test Timing Caused Failure 8 <-> 7. Timing Caused tests Requirement Analysis Usability Lack of Usability Analysis 6 <-> 8. Usability Analysis and countermeasure, FMEA Environment and usage condition 53 <-> 9.Requreiment Analysis. Scenario Test under user environment, Validation External Software IT Network , Outer-Communication 14 <-> 10. IT/Network related test, Integration Test COTS Software 4 <-> 11. SOUP/OTS Risk Analysis and Evaluation Severity Probability of serious harm 13 <-> 12. FTA, 17 Activities requested by IEC62304
- 18. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Medical device software related international standards which are newly relieved or revised after 2003 18 Risk Management (2003) Software Processes (2006) Usability (2007) IT/Network (2010)
- 19. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effective counter measure for recurrence prevention Probable Cause Feature of Fault and Probable Cause No. Effective Counter Measure Internal Hardware Hardware Random Failure 12 <-> 1. Statistical Bug Analysis, Risk Control by Software Hardware Systemic Failure 3 <-> 2. Hardware Design, Design Process Management Internal Software Systemic Failure/Faults Consequence Analysis required Failure 18 <-> 3. FMEA Configuration Management Mistakes 3 <-> 4. Software Configuration Management, Change Management, Regression Test Software Systemic Failure 72 <-> 5. Software Process Management Interface/Timing Interface caused failure 16 <-> 6. Input Simulation Test Timing Caused Failure 8 <-> 7. Timing Caused tests Requirement Analysis Usability Lack of Usability Analysis 6 <-> 8. Usability Analysis and countermeasure, FMEA Environment and usage condition 53 <-> 9.Requreiment Analysis. Scenario Test under user environment, Validation External Software IT Network , Outer-Communication 14 <-> 10. IT/Network related test, Integration Test COTS Software 4 <-> 11. SOUP/OTS Risk Analysis and Evaluation Severity Probability of serious harm 13 <-> 12. FTA, 19 Activities requested by ISO14971
- 20. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Medical device software related international standards which are newly relieved or revised after 2003 20 Risk Management (2003) Software Processes (2006) Usability (2007) IT/Network (2010)
- 21. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effective counter measure for recurrence prevention Probable Cause Feature of Fault and Probable Cause No. Effective Counter Measure Internal Hardware Hardware Random Failure 12 <-> 1. Statistical Bug Analysis, Risk Control by Software Hardware Systemic Failure 3 <-> 2. Hardware Design, Design Process Management Internal Software Systemic Failure/Faults Consequence Analysis required Failure 18 <-> 3. FMEA Configuration Management Mistakes 3 <-> 4. Software Configuration Management, Change Management, Regression Test Software Systemic Failure 72 <-> 5. Software Process Management Interface/Timing Interface caused failure 16 <-> 6. Input Simulation Test Timing Caused Failure 8 <-> 7. Timing Caused tests Requirement Analysis Usability Lack of Usability Analysis 6 <-> 8. Usability Analysis and countermeasure, FMEA Environment and usage condition 53 <-> 9.Requreiment Analysis. Scenario Test under user environment, Validation External Software IT Network , Outer-Communication 14 <-> 10. IT/Network related test, Integration Test COTS Software 4 <-> 11. SOUP/OTS Risk Analysis and Evaluation Severity Probability of serious harm 13 <-> 12. FTA, 21 Activities requested by IEC62366
- 22. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Medical device software related international standards which are newly relieved or revised after 2003 22 Risk Management (2003) Software Processes (2006) Usability (2007) IT/Network (2010)
- 23. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effective counter measure for recurrence prevention Probable Cause Feature of Fault and Probable Cause No. Effective Counter Measure Internal Hardware Hardware Random Failure 12 <-> 1. Statistical Bug Analysis, Risk Control by Software Hardware Systemic Failure 3 <-> 2. Hardware Design, Design Process Management Internal Software Systemic Failure/Faults Consequence Analysis required Failure 18 <-> 3. FMEA Configuration Management Mistakes 3 <-> 4. Software Configuration Management, Change Management, Regression Test Software Systemic Failure 72 <-> 5. Software Process Management Interface/Timing Interface caused failure 16 <-> 6. Input Simulation Test Timing Caused Failure 8 <-> 7. Timing Caused tests Requirement Analysis Usability Lack of Usability Analysis 6 <-> 8. Usability Analysis and countermeasure, FMEA Environment and usage condition 53 <-> 9.Requreiment Analysis. Scenario Test under user environment, Validation External Software IT Network , Outer-Communication 14 <-> 10. IT/Network related test, Integration Test COTS Software 4 <-> 11. SOUP/OTS Risk Analysis and Evaluation Severity Probability of serious harm 13 <-> 12. FTA, 23 Activities requested by IEC80001
- 24. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Medical device software related international standards which are newly relieved or revised after 2003 24 Risk Management (2003) Software Processes (2006) Usability (2007) IT/Network (2010)
- 25. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effective counter measure for recurrence prevention Probable Cause Feature of Fault and Probable Cause No. Effective Counter Measure Internal Hardware Hardware Random Failure 12 <-> 1. Statistical Bug Analysis, Risk Control by Software Hardware Systemic Failure 3 <-> 2. Hardware Design, Design Process Management Internal Software Systemic Failure/Faults Consequence Analysis required Failure 18 <-> 3. FMEA Configuration Management Mistakes 3 <-> 4. Software Configuration Management, Change Management, Regression Test Software Systemic Failure 72 <-> 5. Software Process Management Interface/Timing Interface caused failure 16 <-> 6. Input Simulation Test Timing Caused Failure 8 <-> 7. Timing Caused tests Requirement Analysis Usability Lack of Usability Analysis 6 <-> 8. Usability Analysis and countermeasure, FMEA Environment and usage condition 53 <-> 9.Requreiment Analysis. Scenario Test under user environment, Validation External Software IT Network , Outer-Communication 14 <-> 10. IT/Network related test, Integration Test COTS Software 4 <-> 11. SOUP/OTS Risk Analysis and Evaluation Severity Probability of serious harm 13 <-> 12. FTA, 25 Activities requested by ISO13485
- 26. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Medical device software related international standards which are newly relieved or revised after 2003 26 Risk Management (2003) Software Processes (2006) Usability (2007) IT/Network (2010)
- 27. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effective counter measure for recurrence prevention Probable Cause Feature of Fault and Probable Cause No. Effective Counter Measure Internal Hardware Hardware Random Failure 12 <-> 1. Statistical Bug Analysis, Risk Control by Software Hardware Systemic Failure 3 <-> 2. Hardware Design, Design Process Management Internal Software Systemic Failure/Faults Consequence Analysis required Failure 18 <-> 3. FMEA Configuration Management Mistakes 3 <-> 4. Software Configuration Management, Change Management, Regression Test Software Systemic Failure 72 <-> 5. Software Process Management Interface/Timing Interface caused failure 16 <-> 6. Input Simulation Test Timing Caused Failure 8 <-> 7. Timing Caused tests Requirement Analysis Usability Lack of Usability Analysis 6 <-> 8. Usability Analysis and countermeasure, FMEA Environment and usage condition 53 <-> 9.Requreiment Analysis. Scenario Test under user environment, Validation External Software IT Network , Outer-Communication 14 <-> 10. IT/Network related test, Integration Test COTS Software 4 <-> 11. SOUP/OTS Risk Analysis and Evaluation Severity Probability of serious harm 13 <-> 12. FTA, 27 Activities requested by IEC62304
- 28. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Overview of software development processes and activities (IEC62304) 28
- 29. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effectiveness Analysis of IEC 62304 Process and Activities-1 29 Software development process 1, (3), 4, 5, 6, 7, 8, 9, 10, 11, 12 REMARKS 5.1 Software development planning 1, 4, 5, 8, 11 Software Development Plan is useful to clarify how much Verification and Validation should be conducted.5.1.1 Software development plan 5 5.1.2 Keep software development plan updated 5 5.1.3 Software development plan reference to system design and development --- 5.1.4 Software development standards, methods and tools planning --- 5.1.5 Software integration and integration testing planning 5 5.1.6 Software verification planning 5 5.1.7 Software risk management planning 5 5.1.8 Documentation planning 1, 5, 8, 11 5.1.9 Software configuration management planning --- 5.1.10 Supporting items to be controlled 4, 5 5.1.11 Software configuration item control before verification --- 5.2 Software requirements analysis 4, 5 Clarification of Intended Use is effective to clarify the function of software. 5.2.1 Define and document software requirements from system requirements 9 5.2.2 Software requirements content 9 5.2.3 Include risk control measures in software requirements 9 5.2.4 Re-evaluate medical device risk analysis (3), 8, 9, 10, 11, 12 5.2.5 Update system requirements --- 5.2.6 Verify software requirements ---
- 30. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effectiveness Analysis of IEC 62304 Process and Activities-2 5.3 Software architectural design 10, 11, 12 Complexity of software architecture causes software anomaly. 5.3.1 Transform software requirements into an architecture 12 5.3.2 Develop an architecture for the interfaces of software items 10, 11, 12 5.3.3 Specify functional and performance requirements of SOUP item 11 5.3.4 Specify system hardware and software required by SOUP item 11 5.3.5 Identify segregation necessary for risk control 12 5.3.6 Verify software architecture 12 5.4 Software detailed design --- 5.4.1 Refine software architecture into software units --- 5.4.2 Develop detailed design for each software unit --- 5.4.3 Develop detailed design for interfaces --- 5.4.4 Verify detailed design --- 5.5 Software unit implementation and verification --- 5.5.1 Implement each software unit --- 5.5.2 Establish software unit verification process --- 5.5.3 Software unit acceptance criteria --- 5.5.4 Additional software unit acceptance criteria --- 5.5.5 Software unit verification --- 30
- 31. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effectiveness Analysis of IEC 62304 Process and Activities-3 5.6 Software integration and integration testing 4, 5, (6), (7), 9, 10, 11 Software interface causes some failures. Software integration tests are effective to cover these causes.. 5.6.1 Integrate software units --- 5.6.2 Verify software integration 9 5.6.3 Test integrated software 5, (6), (7), 10, 11 5.6.4 Integration testing content 5, (6), (7), 10, 11 5.6.5 Verify integration test procedures (6), (7), 10, 11 5.6.6 Conduct regression tests 4 5.6.7 Integration test record contents --- 5.6.8 Use software problem resolution process 5 5.7 Software system testing 4, 5, (6), (7), 9, 10 Software system testing is effective to cover software systematic failures. 5.7.1 Establish tests for software requirements (6), (7), 9, 10 5.7.2 Use software problem resolution process 5 5.7.3 Retest after changes 4 5.7.4 Verify software system testing 5 5.7.5 Software system test record contents --- 5.8 Software release 4, 5 Software release process is effective to cover software systematic failures. 5.8.1 Ensure software verification is complete 5 5.8.2 Document known residual anomalies 4 5.8.3 Evaluate known residual anomalies 4 5.8.4 Document released versions 5 5.8.5 Document how released software was created 5 5.8.6 Ensure activities and tasks are complete 5 5.8.7 Archive software 4 5.8.8 Assure repeatability of software release 5 31
- 32. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Effectiveness Analysis of IEC 62304 Process and Activities-4 Software maintenance process 3, 4, 5, 8, 9 Software maintenance process is effective to monitor anomaly occurred in the field.6.1 Establish software maintenance plan 5 6.2 Problem and modification analysis 3, 4, 5, 8, 9 6.2.1 Document and evaluate feedback 3, 8, 9 6.2.1.1 Monitor feedback 3, 8, 9 6.2.1.2 Document and evaluate feedback 3, 8, 9 6.2.1.3 Evaluate problem report's affects on safety 3, 8, 9 6.2.2 Use software problem resolution process 5 6.2.3 Analyze change requests 4 6.2.4 Change request approval 4 6.2.5 Communicate to users and regulators 5 6.3 Modification implementation 4, 5 6.3.1 Use established process to implement modification 4, 5 6.3.2 Re-release modified software system 4, 5 32
- 33. [email protected] ISSRE 2011 Nov 29 - Dec 2, 2011 Summary • As the results of the comparison between old and new fault class distribution, definite difference is identified. Three differences are identified. The percentage of “Logic” and “Calculation” are decreased and that of “Fault tolerance” and “other” are increased. • The reasons are : – software in medical device is getting more complicated ( that causes systematic failures) – software technology is introduced into medical device ( that causes IT network failures , usability failures and database failures) • Recent standards can be effective counter measures for recent failures. 33