Rankings of public services - can we trust them?
Posted on 9 February 2007
To find out, researchers in the University of York’s Centre for Health Economics (CHE) investigated composite indicators, the most widely used tools for compiling performance league tables.
We found that even small changes to the way...composite ratings are constructed can have a substantial impact on the result, so performance league tables should be viewed with caution
Rowena Jacobs
Composite indicators - an aggregate of a number of underlying performance indicators - have instant appeal because they provide a single score or rating that is easy to understand and summarises a wealth of performance data. But the researchers discovered that that even small changes in how composite ratings are constructed can result in a substantial impact on the result.
Composite scores such as ‘star ratings’ of hospitals or local authorities take on great importance as they can be used to reward or penalise an organisation. League table position can lead to changes in leadership for poor performers, or greater freedom from Government control for the best.
Under the Public Services Programme funded by the Economic and Social Research Council (ESRC), researchers in CHE investigated how far rankings based on composite indicators reflect real differences in performance - rather than differences due to random variation or measurement error.
Rowena Jacobs, who led the research, said: "We used an innovative approach to assess the extent of uncertainty in performance indicators making up a composite, by using simulation methods. We tested different composite scores for their sensitivity to random variation, uncertainty and alternative ways of aggregating indicators into a composite.
"We found that even small changes to the way these composite ratings are constructed can have a substantial impact on the result, so performance league tables should be viewed with caution. These findings are important for how we design performance measurement systems in future."
The researchers have launched a Policy Discussion Briefing on the Centre for Health Economics website which provides details of their research findings.
Notes to editors:
- This research was conducted by Rowena Jacobs (rj3@york.ac.uk), Maria Goddard (rj3@york.ac.uk) and Peter C. Smith (pcs1@york.ac.uk) at the Centre for Health Economics.
- The Policy Discussion Briefing is available at: www.york.ac.uk/inst/che/publications/hpolicypubs.htm
- More information on the Centre for Health Economics is available at: www.york.ac.uk/inst/che/
- More information on the ESRC Public Services Programme is available at: www.publicservices.ac.uk/