Key measure of hospital quality does not accurately indicate avoidable deaths

Standardised mortality ratios (SMRs) for hospitals do not give an accurate measure of how many deaths could have been avoided, according to a study.

Writing in the BMJ, researchers at the London School of Hygiene & Tropical Medicine and Imperial College London say the widely used hospital-wide standardised mortality ratios, such as HSMR (Hospital Standardised Mortality Ratio) and SHMI (Summary Hospital level Mortality Indicator), should not be used as a benchmark for hospitals’ quality of care.

Hospital-wide SMRs compare the number of deaths in a hospital with the expected number, based on a statistical probability derived from routine administrative data. These measures have been widely used in many countries for more than 20 years and are often used to identify ‘problem’ hospitals.

The study was commissioned by Professor Sir Bruce Keogh, Medical Director of NHS England, following his review of 14 trusts in 2013 which questioned whether SMRs for hospitals provide an accurate indication of the number of avoidable deaths.

Led by Dr Helen Hogan and Professor Nick Black at the London School of Hygiene & Tropical Medicine and Professor the Lord Ara Darzi at Imperial, the study is the largest nationally representative case record review of hospital deaths in England, based on 3,400 deaths in 34 randomly-selected hospital trusts. Experienced doctors reviewed patients’ case notes and the proportion of deaths they judged likely to have been avoidable was compared with the trust’s HSMR and SHMI.

The researchers found that the proportion of avoidable hospital deaths was 3.6 per cent. There was no significant association between hospital-wide SMRs and the proportion of avoidable deaths in a trust.

Lead author Dr Helen Hogan, Senior Lecturer in Public Health at the London School of Hygiene & Tropical Medicine, said: “Our findings suggest that the commonly used hospital-wide SMRs are not a useful reflection of the proportion of avoidable deaths in a trust.

“Dreadful though each avoidable hospital death is, they are too infrequent to be the basis of a robust indicator to detect significant differences between trusts. There are credible alternatives for assessing the quality of hospitals which give a fairer and more accurate picture.”

Study senior author, Nick Black, Professor of Health Services Research at the London School of Hygiene & Tropical Medicine, said: “Hospital-wide SMRs can be distracting and potentially misleading and should not be used as a basis to praise or condemn a hospital or trust. We need to support and help hospitals to raise standards not criticise and punish them.

“Given the complexity of hospitals, with many different clinical departments and activities, it is more helpful for the public, patients, staff and politicians to use a variety of specific measures of quality, such as adherence to good practice guidelines; outcomes for specific diseases or procedures; patient surveys of their experiences; infection rates; and staff surveys.”

Fellow study author Ara Darzi, Director of Imperial’s Patient Safety and Translational Research Centre and Chair of the Institute of Global Health Innovation at Imperial, said: “Quality must remain the organising principle of our National Healthcare System. We know that the measurement of patient safety within a hospital or clinical environment can be a useful benchmark for assessing quality in healthcare and our study highlights the importance of clinician-led case-note reviews as a robust and invaluable methodology for determining such quality.”

The authors recommend that staff continue to review individual deaths to identify local problems with care, but reviewing should be standardised so that all acute hospital trusts adopt a rigorous approach and take action to prevent future deaths. They suggest SMRs should only be used to assess the quality of care for those conditions with high case fatality for which good quality clinical data exist.