Clinical simulation an opportunity to evaluate digital health technologies

Clinical simulation is an opportunity to evaluate digital health technologies, but regulatory frameworks are needed – new study and report.

A new white paper highlights a recent study led by Imperial College London, which outlines a proposed framework to evaluate the use of clinical simulation in evidence generation for digital health technologies (DHTs).  

The increasing use of digital health technology in healthcare provision has brought many new efficiencies and advances to patient care. However, DHTs often lack the evidence base (e.g. safety, outcomes, value for money) that is commonplace for other clinical interventions such as medications. It is often difficult to evaluate DHTs using traditional methods, such as randomised controlled trials, which have historically been the gold standard used to generate evidence for interventions in healthcare.

Software as a Medical Device (SaMD) is one example of a DHT, and is software that can carry out medical functions, such as predicting diagnoses or identifying patients at risk of disease. SaMD is increasingly used by clinicians but is challenging to evaluate in a controlled environment. Clinical simulation, where end-users are placed in simulated real-life scenarios to test a digital health technology, offers an opportunity to provide high-quality and more cost-effective evidence to support their use.  

Dr Saira Ghafur, Digital Health Lead, Institute of Global Health Innovation, said: "We hope that our research can further the evidence base for applying clinical simulation to generate evidence for digital health technologies."

This is the context for a new white paper and study by Imperial College London’s Institute of Global Health Innovation (IGHI), which proposes the Simulation for Regulation of SaMD (SIROS) framework following a Delphi study with international experts.  

We hope that our research can further the evidence base for applying clinical simulation to generate evidence for digital health technologies Dr Saira Ghafur Institute of Global Health Innovation

The white paper highlights the challenges of regulating digital health technologies and outlines how clinical simulation provides an opportunity to generate an evidence base for their use. The paper specifically focuses on Software as a Medical Device used by clinicians providing care, as opposed to patient-facing tools. The white paper shares the proposed Simulation for Regulation of SaMD (SIROS) framework, developed through an online Delphi process, to assess clinical simulation methods from a regulatory perspective.   

Evidence generation for digital health technologies 

Digital health technologies have revolutionised care delivery, but their fast, iterative development process ​makes them challenging to evaluate and regulate using existing approval processes. The use of artificial intelligence and machine learning algorithms within these technologies adds to their complexity​, due to post-approval changes and continuous upgrades.  

The authors encourage clinical simulation to be increasingly used as an ​evidence-generation method for digital health technologies and for regulatory processes to be advanced to address this. A new approach to regulatory approval to adapt to the advent of digital health technologies is required to safeguard patients and clinicians using these tools and ensure that safe, high-quality​ care is not compromised.  

Regulatory support for clinical simulation  

Imperial College London conducted an online Delphi study with over 30 international experts in policy, healthcare and pharmaceutical sectors. The study sought to gain consensus on the standards for using clinical simulation as a research method to evaluate software as a medical device from a regulatory perspective.  

Professor the Lord Ara Darzi, Co-Director, Institute of Global Health Innovation, said: "Regulators should adopt, use and iterate the SIROS framework to evaluate Software as a Medical Device technologies” 

Regulators should adopt, use and iterate the SIROS framework to evaluate Software as a Medical Device technologies Professor the Lord Ara Darzi Institute of Global Health Innovation

Guiding the participants through an initial scoping process and two Delphi rounds, the research team developed the Simulation for Regulation of Software as a Medical Device (SIROS) framework, comprising 43 criteria across seven domains. Each ​criterion​ ​was agreed to be relevant for evaluating the use of clinical simulation to provide evidence for software as a medical device. The seven domains range from factors such as the software’s context of use to clinical simulation delivery and fidelity ​and artificial intelligence aspects of the technology.  

The SIROS framework aims to support regulators evaluating clinical simulation to test software as a medical device. It aims to improve the regulatory approval process for all stakeholders, from small to large-scale pharmaceutical suppliers, healthcare providers, regulators, and ultimately ensure that the best quality care is provided to patients.  

Key recommendations 

Based on this white paper and study, the Imperial College London team makes a series of recommendations to advance the application of clinical simulation to provide evidence for digital health technologies and support regulators with guidance. These include: 

• Apply the SIROS framework in practice to evaluate clinical simulation methods to generate evidence for software as a medical device  
• Iterate the SIROS framework and expand its use across other digital health technologies  
• Work cross-sectorally to develop new regulatory frameworks and processes for evaluating digital health technologies

Lord Darzi said: “Our new white paper shares our recent research furthering the complex regulatory approach to digital health technologies. We hope that our report can support innovators and regulators in working together to develop new ways of developing, evaluating and approving digital health technologies.” 

Download the white paper 'Evidence generation for digital health technologies: Clinical simulation and its regulation'

Read the research paper ‘Clinical Simulation in the Regulation of Software as a Medical Device: An eDelphi Study’, published in JMIR Formative Research