The issue
The University of York has long been committed to the idea that scientific research is at its most valuable when it is ‘open’, meaning that its aims, methods, tools, execution and results are all made as freely accessible as possible. Academics in the university’s medical school were therefore quick to receive a green light when they proposed addressing a growing issue of transparency arising from recent advances in machine learning.
“Researchers are increasingly using machine learning methods to analyse observational data such as electronic healthcare records,” explains Dr Lewis Paton, Lecturer in Data Science at Hull York Medical School (HYMS) and the Department of Health Sciences. “This could have enormous potential benefits, but when you read the papers based on these analyses, it’s not always clear how the academics arrived at their results. There isn’t enough information given about the algorithms and parameters underpinning the machine learning. That means you can’t replicate the science, which makes it harder to have confidence in the findings.”
Part of Dr Paton’s role within HYMS involves teaching medical students how to go about critiquing the latest academic research and deciding how – if at all – to incorporate it into their practice. “When the healthcare practitioners of the future read about a potentially powerful new tool,” he explains, “they need to be able to decide whether to apply it to their particular population of patients. And if the research behind the tool involves machine learning – which it increasingly will – they can only do that safely if the academics have conducted their research and reported their results in a properly transparent way. We need to develop protocols for communicating this sort of detail – just as we have successfully done for other research methods, such as, say, randomised control trials.”
The research
Together with Professor Paul Tiffin in HYMS and the Department of Health Sciences, Dr Paton began by applying for a Research England ‘Enhancing Research Culture’ grant to set up what he called the SPORE project (Supporting Practices in Observational research Reporting and Execution). He then embarked on two streams of work.
First he worked with a professional web developer to create a website called openscienceready.org, which he populated with guidance for researchers wanting to make use of machine learning. Organised into steps that can be taken either before, during or after conducting any analysis, the site includes details on subjects such as pre-registration, code-sharing, and guidelines on the transparent reporting of results.
Next, Dr Paton organised two online workshops, at which invited speakers provided attendees with expert advice on code-sharing, intellectual property considerations, and software that makes it easier to conduct more reproducible forms of analysis.
The outcome
“We set up SPORE so that we could join others within the university who are looking to change this aspect of research culture,” says Dr Paton. “We wanted to help raise awareness of the issue within HYMS and the university, and change the working practices of individual researchers. I think we have achieved both.
“The website has allowed us to reach many hundreds of people locally and internationally, and the two workshops attracted over 75 people – mostly early career researchers from HYMS and Health Sciences, but also colleagues from Education and Psychology.”
Dr Paton has since built on this work in a number of ways: building both the principles and the practical resources of SPORE into the Faculty of Science’s MSc programme (MSc Data Science); joining the editorial board of a journal and incorporating SPORE’s principles into its publishing requirements; and making his case at the ‘Research on Research’ conference where his presentation was singled out for its quality and likelihood to have an impact. He is also planning a training event on the subject for PhD students.
“Changing research culture doesn’t happen overnight, but we know it’s achievable,” says Dr Paton, “It’s not that long ago that people started calling for academic papers to be submitted to open-access publications, but now it’s just expected. In fact I can’t remember the last time I published a paper that wasn’t open-access. That’s where we need to get to with the way researchers make use of machine learning – we need to make it part of the culture. As new technologies like this come up, we have to do everything we can to harness their power, but that doesn’t just mean using them, it means using them the right way. In the end, it’s about constantly striving to do science better.”