Structured expert elicitation course
We are in the process of developing a course on structured expert elicitation.
If you are keen to hear more, please complete this course enquiry form to indicate your interest and express preferences for the course content.
Your details will be used to email you about the course details once these become available.
Structured expert elicitation resources (STEER)
The materials on this website were developed to help streamline the process of conducting structured expert elicitation (SEE). They include:
- An overview of SEE for healthcare decision making - Structured expert elicitation for healthcare decision making An overview (PDF
, 1,048kb) - A practical guide to SEE for healthcare decision making - Structured expert elicitation for healthcare decision making A practical guide (PDF , 717kb)
- Easily-adaptable R code for developing bespoke web applications for conducting SEE exercises
- Excel template for developing bespoke SEE exercises:
- Chips and Bins (also referred to roulette or histogram method) - STEER Chips and Bins elicitation tool (MS Excel
, 260kb) - Tertiles - STEER Tertiles elicitation tool (MS Excel , 112kb)
- Quartiles (also referred to as bisection) - (STEER Quartiles elicitation tool (MS Excel , 112kb)
- Chips and Bins (also referred to roulette or histogram method) - STEER Chips and Bins elicitation tool (MS Excel
- Adaptable PowerPoint slides to use for training experts to complete SEE exercises. STEER_Template for elicitation training (MS PowerPoint
, 5,997kb)
To use, please cite: Horscroft J, Lee D, Jankovic D, Soares M, Bojke L. Structured Expert Elicitation – Example Training Deck. STEER. 2022 - Open access resources from previous examples (SEE protocols, training materials, tools for conducting SEE)
- Examples of published SEEs applied in healthcare decision modelling
Why is SEE valuable in healthcare decision making?
Global trends
Global trends are leading to higher uncertainty at the point of decision making
Recognition
Recognised as a preferred method where empirical evidence is lacking
Minimises bias
Minimises known biases associated with expert judgements
Use Longitudinally
Can be used longitudinally to predict clinical outcomes
Bounds uncertainty
Provides bounds to uncertainty for key or economic parameters
How were these materials developed?
The materials have been developed as a collaboration between the Centre for Health Economics (CHE), which co-authored the protocol for structured expert elicitation (SEE) funded by the Medical Research Council (MRC), and Lumanity, which regularly conducts SEE to support health technology assessment (HTA) and market access strategic planning.
The materials are based on the original MRC protocol and published SEE exercises in healthcare decision modelling.
How is SEE viewed by decision-makers?
"It is recommended that researchers continue to focus on identifying appropriate data sources for informing parameter estimates and, as elicitation methods continue to evolve, consider expert elicitation as a potential source of data for filling in gaps in the available information."
CADTH Guidelines for the Economic Evaluation of Health Technologies: Canada, 2017
"In the absence of empirical evidence from RCTs, non-randomised studies, or registries, or when considered appropriate by the committee taking into account all other available evidence, expert elicitation can be used to provide evidence ... Structured methods are preferred as they attempt to minimise biases and provide some indication of the uncertainty. Structured approaches should adhere to existing protocols (such as the Medical Research Council protocol)."
NICE Health Technology Evaluations: The Manual, 2022
Long-term survival in CKD (NICE TA775)
Cooke’s classical method to explore long-term survival in patients with CKD receiving placebo
• Calibration
• Online survey
• Aggregation
Objectives:
to explore uncertainty around long-term survival given the immaturity of trial date at the time of NICE submission
Quantities:
survival rates in patients with CKD receiving placebos at 10 and 20 years
Experts:
Six leading disease experts
Method:
Online survey with calibration questions
Value:
confirmed that the survival function selected for the cost-effectiveness model has clinical validity
Anti-microbial resistance
York method to explore outcomes of specific infections following treatment with antimicrobials
• Online Survey
• Aggregation
Objectives:
to explore uncertainty surrounding short-term outcomes of specific infections in absence of empirical data
Quantities:
30-day survival, hospital length of stay and type of ward following treatment of specific types on infections
Experts:
seven UK hospital consultants, microbiologists and pharmacists specialising infectious diseases
Method:
Online surveys with mathematical aggregation
Value:
provided probability distributions for 30-day survival and length of stay, and point estimates for the type of ward