Researchers have used a combination of automated text analysis and the ‘robot scientist’ Eve to semi-automate the process of reproducing research results. The problem of lack of reproducibility is one of the biggest crises facing modern science.
The researchers, led by the University of Cambridge, analysed more than 12,000 research papers on breast cancer cell biology. After narrowing the set down to 74 papers of high scientific interest, less than one third – 22 papers – were found to be reproducible. In two cases, Eve was able to make serendipitous discoveries.
A successful experiment is one where another scientist, in a different laboratory under similar conditions, can achieve the same result. But more than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce some of their own experiments: this is the reproducibility crisis.
“Good science relies on results being reproducible: otherwise, the results are essentially meaningless,” said Professor Ross King from Cambridge’s Department of Chemical Engineering and Biotechnology, who led the research. “This is particularly critical in biomedicine: if I’m a patient and I read about a promising new potential treatment, but the results aren’t reproducible, how am I supposed to know what to believe? The result could be people losing trust in science.”
Several years ago, King developed the robot scientist Eve, a computer/robotic system that uses techniques from artificial intelligence (AI) to carry out scientific experiments.
“One of the big advantages of using machines to do science is they’re more precise and record details more exactly than a human can,” said King. “This makes them well-suited to the job of attempting to reproduce scientific results.”
As part of a project funded by DARPA, King and his colleagues from the UK, US and Sweden designed an experiment that uses a combination of AI and robotics to help address the reproducibility crisis, by getting computers to read scientific papers and understand them, and getting Eve to attempt to reproduce the experiments.
For the current paper, the team focused on cancer research. “The cancer literature is enormous, but no one ever does the same thing twice, making reproducibility a huge issue,” said King. “Given the vast sums of money spent on cancer research, and the sheer number of people affected by cancer worldwide, it’s an area where we urgently need to improve reproducibility.”
From an initial set of more than 12,000 published scientific papers, the researchers used automated text mining techniques to extract statements related to a change in gene expression in response to drug treatment in breast cancer. From this set, 74 papers were selected.
Two different human teams used Eve and two breast cancer cell lines and attempted to reproduce the 74 results. Statistically significant evidence for repeatability was found for 43 papers, meaning that the results were replicable under identical conditions; and significant evidence for reproducibility or robustness was found in 22 papers, meaning the results were replicable by different scientists under similar conditions. In two cases, the automation made serendipitous discoveries.
While only 22 out of 74 papers were found to be reproducible in this experiment, the researchers say that this does not mean that the remaining papers are not scientifically reproducible or robust. “There are lots of reasons why a particular result may not be reproducible in another lab,” said King. “Cell lines can sometimes change their behaviour in different labs under different conditions, for instance. The most important difference we found was that it matters who does the experiment, because every person is different.”
King says that this work shows that automated and semi-automated techniques could be an important tool to help address the reproducibility crisis, and that reproducibility should become a standard part of the scientific process.
“It’s quite shocking how big of an issue reproducibility is in science, and it’s going to need a complete overhaul in the way that a lot of science is done,” said King. “We think that machines have a key role to play in helping to fix it.”
The research was also funded by the Engineering and Physical Sciences Research Council (EPSRC), part of UK Research and Innovation (UKRI).
Gary Price (gprice@gmail.com) is a librarian, writer, consultant, and frequent conference speaker based in the Washington D.C. metro area.
He earned his MLIS degree from Wayne State University in Detroit.
Price has won several awards including the SLA Innovations in Technology Award and Alumnus of the Year from the Wayne St. University Library and Information Science Program. From 2006-2009 he was Director of Online Information Services at Ask.com. Gary is also the co-founder of infoDJ an innovation research consultancy supporting corporate product and business model teams with just-in-time fact and insight finding.
From a Plan S / cOAlition S Announcement: cOAlition S releases today the end-user portal of the Journal Comparison Service (JCS). This secure, free, online service aims to shed light on publishing ...
From the University of Victoria, Canada: Canada’s genocide is getting harder to deny. So too is the growing recognition that human rights and the perspectives of Indigenous Peoples offer solutions ...
From the American Library Association: With funding from the Library of Congress Teaching with Primary Sources program, the American Library Association (ALA) has published “Programming with Primary Sources: Women’s Suffrage,”a resource ...
From a PEN America News Release: James Patterson, the best-selling author, has donated $2 million to PEN America to build communications capacity and help the premier writers’ organization speak louder ...
From a Introductory Blog Post by Dylan Ruediger and Danielle Miriam Cooper: Today, Ithaka S+R releases findings from one of the largest in-depth studies of teaching practices across social science ...
From Stateline/Pew: There are more than 600 oral history recordings housed where Lina Ortega is an associate curator for the Western History Collections at the University of Oklahoma Libraries. Ortega speaks ...
From the University of Louisville: The current and former ownership of one of America’s most respected newspapers has made it possible for a treasure trove of an estimated 3 million ...
Elsevier Appoints Emily Singley to the Newly Created role of Vice President, North American Library Relations Exploring the Potential Role of Citizen Science in the Warning Value Chain for High ...
From the UC Berkeley Library The UC Berkeley Library is full of stories — and not just the ones that line its shelves. The facts you’ll find here are at ...
From The Chicago Tribune: Last week, city and Chicago Public Library officials declared Chicago a sanctuary city for those stories, by establishing “Book Sanctuaries” across the city’s 77 distinct community ...
From Penn State: Penn State University Libraries has expanded its robust scholarly research potential of one of America’s foremost 20th-century writers through the acquisition of the Toby and Betty Bruce ...
From Politico: Far from the trenches in states like Florida and Texas, organizations in deep-blue New York are stepping into the fray by directly lending 25,000 books to non-residents since spring, ...
