Machine Model Exponentially Shortens Expert Review Period for COVID-19 Treatments and Vaccines

Replication, which means that the results of the study can be produced a second time with a new test population, is a key signal that study conclusions are valid. With the new AI tool, researchers can bypass the human-scoring method, allowing the research community and policymakers to make faster decisions about how to prioritize time and funding on the studies that are most likely to succeed.

In comparison to the average time of about 314 days required for the human process, the machine model can scale up to review a larger number of papers in minutes. The machine model is as accurate as scientific experts who review and rate submitted research studies based on how likely they are to be replicable. Used on its own, the model has comparable accuracy to the DARPA SCORE method. In fact, the researchers believe that the machine model’s prediction of the likelihood of replicability may actually be more accurate than the traditional human-scoring prediction, as it considers more of the narrative of the study, while expert reviewers tend to focus on the strength of the relational statistics in a paper.

When paired together, the combination of human-machine approach predicts which findings will be replicable with even greater accuracy than either method on its own, the researchers found. Because the algorithm examines the words of thousands of papers, it recognizes word-choice patterns that might be hidden from human consciousness. It has a much bigger schema to draw upon for its predictions, which makes it an extraordinary partner for human reviewers. The researchers’ model can be used immediately to analyze COVID-related research papers and quickly determine which show the most promise.

“In the midst of a public health crisis, it is essential that we focus our efforts on the most promising research. This is important not only to save lives, but also to quickly tamp down the misinformation that results from poorly conducted research,” said Northwestern’s Brian Uzzi, who led the study. “This tool is particularly useful in this crisis situation where we can’t act fast enough. It can give us an accurate estimate of what’s going to work and not work very quickly. We’re behind the ball, and this can help us catch up.”


Related Links:
Northwestern University