New statistical method brings together studies at an early stage
During the coronavirus pandemic, scientists were in a hurry to find drugs that would help fight the disease. To combine the research that was being carried out around the world, PhD candidate Judith ter Schure developed a new statistical method: ALL-IN meta-analysis. This helps determine sooner whether a vaccine works, for example. PhD defence on 7 April.
At the start of the pandemic, 15 research groups worldwide began looking into whether a tuberculosis vaccine could also be used against coronavirus. The vaccine is known not only to help against tuberculosis but also to offer protection against other infectious diseases.
For their studies, the scientists gave the tuberculosis vaccine to some of their patients and a placebo to others. Then they had to wait until the subjects developed coronavirus. If each individual study group had to wait until it had enough data to draw a conclusion, this would take a long time. Ter Schure’s method makes it possible to combine the interim data from all these studies and thus reach faster conclusions about whether the vaccine works. You can also immediately put the results of various research groups into the context of other studies, something that is often only done afterwards in classical statistics.
Just like at a casino, a series of random results can fool researchers.
Roulette
Science is always a gamble. And just like at a casino, a series of random results can fool researchers, causing them to make the wrong decision. Ter Schure’s method helps prevent a chance wrong conclusion. ‘There’s a weird statistical phenomenon in which if you ask the same question often, there’s a good chance you’ll get the wrong answer at some point.’ She compares it to playing roulette. Most people don’t win much at this game, but there is always one lucky person who wins a lot of money. You shouldn’t then conclude that that one person has shown there is always money to be won.
Poker
To prevent a wrong conclusion from being reached too often by pure chance, statistical methods require a certain minimum result. Take, for example, the number of coronavirus infections in subjects who received a placebo as opposed to the number of coronavirus infections in subjects who received the vaccine. The method indicates when a certain threshold has been reached to draw a conclusion. After a certain amount of time, coincidence is too unlikely. The statistical method thus resembles how professional poker players work. Ter Schure: ‘They can earn their salary in a casino. To recognise a professional, you observe their results over many poker tournaments. They have a long-term strategy. That’s what science needs too.’
‘Professional poker players have a long-term strategy. That’s what science needs too.’
Software
A software package and tutorials online mean that others can use ALL-IN meta-analysis as well. They can aggregate the results of studies and see how far away the finish line still is. This can make science more dynamic and efficient and revolves around collaboration. For example, each research group working on the same question can upload data and see on a dashboard the effect on the results so far. The method can now be used for clinical trials similar to the tuberculosis vaccine study and many other simple analyses. Ter Schure and her colleagues at the Centre for Mathematics & Computer Science, where she did her doctoral research, also want to develop software for more-complex studies.
Cultural shift
The results of the research into using the tuberculosis vaccine for coronavirus will be published soon. Ter Schure is pleased that the researchers prioritised the research line rather than their own research. She thinks it would be nice if this happened more often in the future, but that would be quite a cultural shift in academia. ‘At least we now have an example that it can be done. Who knows, ALL-IN might help with science where everyone is less concerned with their own paper and more with the big picture.’
Text: Dagmar Aarts
Photo: Unsplash