Transform to Open Science (TOPS) | Science Mission Directorate

“From 2022 to 2027, TOPS will accelerate the engagement of the scientific community in open science practices through events and activities aimed at:

Lowering barriers to entry for historically excluded communities
Better understanding how people use NASA data and code to take advantage of our big data collections
Increasing opportunities for collaboration while promoting scientific innovation, transparency, and reproducibility….”

Study Shows Ensuring Reproducibility in Research Is Needed – IEEE Spectrum

“About 60 percent of IEEE conferences, magazines, and journals have no practices in place to ensure reproducibility of the research they publish. That’s according to a study by an ad hoc committee formed by the IEEE Computer Society to investigate the matter and suggest remedies.

Reproducibility—the ability to repeat a line of research and obtain consistent results—can help confirm the validity of scientific discoveries, IEEE Fellow Manish Parashar points out. He is chair of the society’s Committee on Open Science and Reproducibility….

The goal of the ad hoc committee’s study was to ensure that research results IEEE publishes are reproducible and that readers can look at the results and “be confident that they understand the processes used to create those results and they can reproduce them in their labs,” Parashar says….

Here are three key recommendations from the report:

Researchers should include specific, detailed information about the products they used in their experiment. When naming the software program, for example, authors should include the version and all necessary computer codes that were written. In addition, journals should make submitting the information easier by adding a step in the submission process. The survey found that 22 percent of the society’s journals, magazines, and conferences already have infrastructure in place for submitting such information.
All researchers should include a clear, specific, and complete description of how the reported results were reached. That includes input data, computational steps, and the conditions under which experiments and analysis were performed.
Journals and magazines, as well as scientific societies requesting submissions for their conferences, should develop and disclose policies about achieving reproducibility. Guidelines should include such information as how the papers will be evaluated for reproducibility and criteria code and data must meet….”

OSF Preprints | A survey of funders’ and institutions’ needs for understanding researchers’ open research practices

Abstract:  A growing number of research-performing organisations (institutions) and funding agencies have policies that support open research practices — sharing of research data, code and software. However, funders and institutions lack sufficient tools, time or resources to monitor compliance with these policies.

  To better understand funder and institution needs related to understanding open research practices of researchers, we targeted funders and institutions with a survey in 2020 and received 122 completed responses. Our survey assessed and scored, (from 0-100), the importance of and satisfaction with 17 factors associated with understanding open research practices. This includes things such as knowing if a research paper includes links to research data in a repository; knowing if a research grant made code available in a public repository; knowing if research data were made available in a reusable form; and knowing reasons why research data are not publicly available. Half of respondents had tried to evaluate researchers’ open research practices in the past and 78% plan to do this in the future. The most common method used to find out if researchers are practicing open research was personal contact with researchers and the most common reason for doing it was to increase their knowledge of researchers’ sharing practices (e.g. determine current state of sharing; track changes in practices over time; compare different departments/disciplines). The results indicate that nearly all of the 17 factors we asked about in the survey were underserved. The mean importance of all factors to respondents was 71.7, approaching the 75 threshold of “very important”. The average satisfaction of all factors was 41.3, indicating a negative level of satisfaction with ability to complete these tasks. The results imply an opportunity for better solutions to meet these needs. The growth of policies and requirements for making research data and code available does not appear to be matched with solutions for determining if these policies have been complied with. We conclude that publishers can better support some of the needs of funders and institutions by introducing simple solutions such as: – Mandatory data availability statements (DAS) in research articles – Not permitting generic “data available on request” statements – Enabling and encouraging the use of data repositories and other methods that make data available in a more reusable way – Providing visible links to research data on publications – Making information available on data and code sharing practices in publications available to institutions and funding agencies – Extending policies that require transparency in sharing of research data, to sharing of code

Toward Reusable Science with Readable Code and Reproducibility

Abstract:  An essential part of research and scientific communication is researchers’ ability to reproduce the results of others. While there have been increasing standards for authors to make data and code available, many of these files are hard to re-execute in practice, leading to a lack of research reproducibility. This poses a major problem for students and researchers in the same field who cannot leverage the previously published findings for study or further inquiry. To address this, we propose an open-source platform named RE3 that helps improve the reproducibility and readability of research projects involving R code. Our platform incorporates assessing code readability with a machine learning model trained on a code readability survey and an automatic containerization service that executes code files and warns users of reproducibility errors. This process helps ensure the reproducibility and readability of projects and therefore fast-track their verification and reuse.

 

Open science practices for eating disorders research

Abstract:  This editorial seeks to encourage the increased applicationof three open science practices in eating disordersresearch: Preregistration, Registered Reports, and the shar-ing of materials, data, and code. For each of these prac-tices, we introduce updated International Journal of Eating Disorders author and reviewer guidance. Updates include the introduction of open science badges; specific instruc-tions about how to improve transparency; and the intro-duction of Registered Reports of systematic or meta-analytical reviews. The editorial also seeks to encourage the study of open science practices. Open science prac-tices pose considerable time and other resource burdens.Therefore, research is needed to help determine the valueof these added burdens and to identify efficient strategies for implementing open science practices.

Closing the knowledge-action gap in conservation with open science

Abstract:  The knowledge-action gap in conservation science and practice occurs when research outputs do not result in actions to protect or restore biodiversity. Among the diverse and complex reasons for this gap, three barriers are fundamental: knowledge is often unavailable to practitioners, challenging to interpret, and/or difficult to use. Problems of availability, interpretability, and useability are solvable with open science practices. We consider the benefits and challenges of three open science practices for use by conservation scientists and practitioners. First, open access publishing makes the scientific literature available to all. Second, open materials (methods, data, code, and software) increase the transparency and (re)use potential of research findings. Third, open education resources allow conservation professionals (scientists and practitioners) to acquire the skills needed to make use of research outputs. The long-term adoption of open science practices would help researchers and practitioners achieve conservation goals more quickly and efficiently, in addition to reducing inequities in information sharing. However, short-term costs for individual researchers (insufficient institutional incentives to engage in open science and knowledge mobilization) remain a challenge to overcome. Finally, we caution against a passive approach to sharing that simply involves making information available. We advocate for a proactive stance towards transparency, communication, collaboration, and capacity building that involves seeking out and engaging with potential users to maximize the environmental and societal impact of conservation science.

 

Social Science Reproduction Platforms

“The Social Science Reproduction Platform (SSRP) is an openly licensed platform that facilitates the sourcing, cataloging, and review of attempts to verify and improve the computational reproducibility of social science research. Computational reproducibility is the ability to reproduce the results, tables, and other figures found in research articles using the data, code, and materials made available by the authors. The SSRP is meant to be used in combination with the Guide for Accelerating Computational Reproducibility (ACRe Guide), a protocol that includes detailed steps and criteria for assessing and improving reproducibility.

Assessments of reproducibility often gravitate towards binary judgments that declare entire papers as “reproducible” or “not reproducible”. The SSRP allows for a more nuanced approach to reproducibility, where reproducers analyze individual claims and their associated display items, and take concrete steps to improve their reproducibility. SSRP reproductions are transparent and reproducible in themselves since they are based on the ACRe Guide’s standardized reproduction protocol and publicly document their analyses to allow for collaboration, discussion, and reuse. Sign up for a free account now to get started in improving computational reproducibility—one claim at a time!

SSRP was developed as part of the Accelerating Computational Reproducibility in Economics (ACRE) project led by the Berkeley Initiative for Transparency in the Social Sciences (BITSS in collaboration with the AEA Data Editor)….”

Social Science Reproduction Platforms

“The Social Science Reproduction Platform (SSRP) is an openly licensed platform that facilitates the sourcing, cataloging, and review of attempts to verify and improve the computational reproducibility of social science research. Computational reproducibility is the ability to reproduce the results, tables, and other figures found in research articles using the data, code, and materials made available by the authors. The SSRP is meant to be used in combination with the Guide for Accelerating Computational Reproducibility (ACRe Guide), a protocol that includes detailed steps and criteria for assessing and improving reproducibility.

Assessments of reproducibility often gravitate towards binary judgments that declare entire papers as “reproducible” or “not reproducible”. The SSRP allows for a more nuanced approach to reproducibility, where reproducers analyze individual claims and their associated display items, and take concrete steps to improve their reproducibility. SSRP reproductions are transparent and reproducible in themselves since they are based on the ACRe Guide’s standardized reproduction protocol and publicly document their analyses to allow for collaboration, discussion, and reuse. Sign up for a free account now to get started in improving computational reproducibility—one claim at a time!

SSRP was developed as part of the Accelerating Computational Reproducibility in Economics (ACRE) project led by the Berkeley Initiative for Transparency in the Social Sciences (BITSS in collaboration with the AEA Data Editor)….”

CODECHECK: an Open Science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility

Abstract:  The traditional scientific paper falls short of effectively communicating computational research.  To help improve this situation, we propose a system by which the computational workflows underlying research articles are checked. The CODECHECK system uses open infrastructure and tools and can be integrated into review and publication processes in multiple ways. We describe these integrations along multiple dimensions (importance, who, openness, when). In collaboration with academic publishers and conferences, we demonstrate CODECHECK with 25 reproductions of diverse scientific publications. These CODECHECKs show that asking for reproducible workflows during a collaborative review can effectively improve executability. While CODECHECK has clear limitations, it may represent a building block in Open Science and publishing ecosystems for improving the reproducibility, appreciation, and, potentially, the quality of non-textual research artefacts. The CODECHECK website can be accessed here: https://codecheck.org.uk/.

 

News and Stories – Research Guides at New York University

“The Alfred P. Sloan Foundation has awarded New York University a grant of $520,503 to enable libraries and other institutions to reliably archive digital scholarship, with a focus on research code, for long-term accessibility. Vicky Rampin, NYU’s Research Data Management and Reproducibility Librarian, designed the project with her co-principal investigator, Martin Klein, Research Scientist at the Los Alamos National Laboratory (LANL).

The project follows Investigating and Archiving the Scholarly Git Experience (IASGE), an extensive NYU Libraries study also funded by the Sloan Foundation and led by Rampin, examining the landscape of current research software archiving efforts and the behavior of academics using Git and Git Hosting Platforms for scholarly reasons. The findings of both facets of IASGE underscore the vulnerability of scholarship on these platforms, from lack of holistic archival practices for research code to gaps in the research software management landscape that make long-term access more difficult. As Rampin and Klein wrote in their most recent proposal: “These factors leave us with little hope for long-term access to and availability of our scholarly artifacts on the Web.” …”

NYU Wins Major Grant From Alfred P. Sloan Foundation To Expand Capabilities For Archiving Digital Scholarship

“The Alfred P. Sloan Foundation has awarded New York University a grant of $520,503 to enable libraries and other institutions to reliably archive digital scholarship, with a focus on research code, for long-term accessibility. Vicky Rampin, NYU’s Research Data Management and Reproducibility Librarian, designed the project with her co-principal investigator, Martin Klein, Research Scientist at the Los Alamos National Laboratory (LANL).

The project follows Investigating and Archiving the Scholarly Git Experience (IASGE), an extensive NYU Libraries study also funded by the Sloan Foundation and led by Rampin, examining the landscape of current research software archiving efforts and the behavior of academics using Git and Git Hosting Platforms for scholarly reasons. The findings of both facets of IASGE underscore the vulnerability of scholarship on these platforms, from lack of holistic archival practices for research code to gaps in the research software management landscape that make long-term access more difficult. As Rampin and Klein wrote in their most recent proposal: “These factors leave us with little hope for long-term access to and availability of our scholarly artifacts on the Web.” …”

Ouvrir la Science – On the road to opening up research codes

In November 2018, the Committee for Open Science set up a ‘free and open source software group’, or the GPLO for short.

The creation of this group was based on a simple observation – that software is at the core of research and that open source practices are one of the founding elements of open science. The GPLO’s mission is to help the committee support the development of free and open software in scientific communities as such software is considered to be a pillar of open science.