EBMT NEWSLETTER | January 2018 | Volume 60 - Issue 1

Important dates

Benchmarking of clinical outcome

The US Agency for Healthcare Research and Quality (AHRQ) defines benchmarking as
Benchmarking is the process of comparing a practice’s performance with an external standard.[1]

Benchmarking in medicine is not a new concept but it is a highly complex issue with very varying degrees of acceptance among healthcare professionals. The prime concern is that the comparison is fair i.e. that ‘apples’ are really being compared with ‘apples’ and not ‘oranges’ and the inherent complexity of transplantation increases the depth of this challenge.

Benchmarking should be of high interest to all care providers but particularly in the interest of patient care. Additionally, regulators and payment bodies are increasingly interested to see more data on performance by centres and some have looked into developing their own performance measurement schemes which do not always match the professionals’ needs nor assuage their concerns about accuracy.

In March 2015, the 6th edition of the FACT-JACIE Standards introduced the requirement for centres to compare 1-year survival outcome with national or international data. When published, one inspector described the significance of this requirement as “giving JACIE ‘teeth’ for the first time”.

In practical terms, this requirement was easier for US-based centres to comply with given the legal requirement for US centres to report to the Stem Cell Therapeutics Outcomes Database (SCTOD)[2]. Conversely few schemes of this kind exist anywhere else in the world aside from the UK, Italy and France among others which presented a significant challenge to centres as to how they could compare their results to an wider sample.

Not long after the 6th edition was published, the EBMT was planning Project 2020 to redesign the EBMT Registry. The EBMT registry is an obvious source of international data and therefore among the different work packages supporting the project is one dedicated to Clinical Quality Benchmarking. The objective is to provide a neutral platform to allow centres to compare their performance in the interest of improving care for their patients. Centres that appear to be underperforming will be approached to help find out why while centres that stand out as leaders will be asked to share their approaches in order to disseminate best practice.

Since work started in the first half of 2017, the benchmarking project has made important progress. Project implementation is being done by the team from the Dep. of Medical Statistics & BioInformatics Leiden University Medical Centre (The Netherlands) with Hein Putter, Theodor Balan (programming), Erik van Zwet and Ronald Brand (advisor) developing and implementing an EBMT benchmarking approach.

The main work so far has focussed on constructing case-mix-corrected funnel plots to compare centers to the national or European average performance with respect to 1-year and 5-year overall survival. Funnel plots are a very well-known tool for performance evaluation. The output will be a performance ranking of centers.

As part of the early development work, the LUMC methods have been applied to data from British and Italian centers for comparison with the existing BSBMT and GITMO mechanisms. It is important to note that the goal is not to supplant existing performance evaluations but to supplement them and to facilitate comparisons across Europe. This validation phase should conclude in March 2018.

Later in 2018 an expert group will be formed to discuss model assumptions, interpretation and limitations of the results, and consider the impact of the introduction of a benchmarking system and a process will be opened in which EBMT will decide how the benchmarking information should be delivered to the centres and how to prevent misinterpretation of results.

 [1] https://www.ahrq.gov/professionals/prevention-chronic-care/improve/system/pfhandbook/mod7.html. Accessed 12/01/2017
[2] https://www.cibmtr.org/About/WhatWeDo/SCTOD/Pages/index.aspx
Bookmark and Share