Center for Open Science (COS) Receives Investment by DARPA (Defense Advanced Research Projects Agency) to Improve Assessment Of Research Credibility

From COS:

The Center for Open Science (COS) has been selected to participate in DARPA’s new program Systematizing Confidence in Open Research and Evidence (SCORE) with a 3-year cooperative agreement potentially totaling more than $7.6 million.  This program represents an investment by DARPA to assess and improve the credibility of social and behavioral science research.

DARPA identifies the purpose of SCORE is “to develop and deploy automated tools to assign ‘confidence scores’ to different SBS research results and claims. Confidence scores are quantitative measures that should enable a DoD consumer of SBS research to understand the degree to which a particular claim or result is likely to be reproducible or replicable.”  If successful, consumers of scientific evidence–researchers, funders, policymakers, etc.–would have readily available information about the uncertainty associated with that evidence.

[Clip]

To achieve this aspirational objective, the SCORE program will produce four primary artifacts:

1. A massive database of approximately 30,000 claims from published papers in the social-behavioral sciences. The database will be enhanced with evidence about those claims that is automatically and manually extracted from the paper itself and merged into the database from other sources, such as how often the work has been cited and whether the data are openly accessible or the research was preregistered.
2. Experts will review and score about 3,000 of those claims in surveys, panels, or prediction markets for their likelihood of being reproducible findings.
3. Teams will use the database of information about the claims to generate algorithms (artificial intelligence) to score the same claims as the experts.
4. Hundreds of researchers will conduct replications on a sample of the claims to test the experts’ and algorithms’ ability to predict reproducibility.

The COS team, in collaboration with partners from the University of Pennsylvania and Syracuse University, is responsible for creating the database.  Other teams will recruit experts and create the algorithms to evaluate the claims. And, COS will coordinate a massive collaboration of researchers from every area of the social and behavioral sciences to conduct replication and reproducibility studies.

“DARPA’s investment signals the onset of the next phase of the reformation that is underway in the social and behavioral sciences,” said Brian Nosek, Executive Director of COS and Principal Investigator of the COS team. “For the last eight years, the research community has been scrutinizing the reproducibility of its findings and the quality of its research practices.  Now, that learning is being translated into opportunities to improve research practices to accelerate the pace of discovery.”

[Clip]

Coordination of such a large-scale collaboration will be aided by the Open Science Framework (https://osf.io/) an open-source collaborative management tool that facilitates rigorous, reproducible research that is maintained by COS.  “OSF is essential for us to be able to manage such a complex program with so many contributors, projects, and associated data,” noted Beatrix Arendt, Program Manager at COS.  “We are committed to transparency of process and outcomes so that we are accountable to the research community to do the best job that we can, and so that all of our work can be scrutinized and reproduced for future research that will build on this work.”

[Clip]

Researchers interested in potentially joining this program to conduct replication or reproduction studies are encouraged to review the Call for Collaborators.

Direct to Complete Announcement

Filed under: Data Files, Journal Articles, Management and Leadership, News, Open Access, Reports