SocArXiv Papers | Dynamics of Cumulative Advantage and Threats to Equity in Open Science – A Scoping Review

Open Science holds the promise to make scientific endeavours more inclusive, participatory, understandable, accessible, and re-usable for large audiences. However, making processes open will not per se drive wide re-use or participation unless also accompanied by the capacity (in terms of knowledge, skills, financial resources, technological readiness and motivation) to do so. These capacities vary considerably across regions, institutions and demographics. Those advantaged by such factors will remain potentially privileged, putting Open Science’s agenda of inclusivity at risk of propagating conditions of “cumulative advantage”. With this paper, we systematically scope existing research addressing the question: “What evidence and discourse exists in the literature about the ways in which dynamics and structures of inequality could persist or be exacerbated in the transition to Open Science, across disciplines, regions and demographics?” Aiming to synthesise findings, identify gaps in the literature, and inform future research and policy, our results identify threats to equity associated with all aspects of Open Science, including Open Access, Open/FAIR Data, Open Methods, Open Evaluation, Citizen Science, as well as its interfaces with society, industry and policy. Key threats include: stratifications of publishing due to the exclusionary nature of the author-pays model of Open Access; potential widening of the digital divide due to the infrastructure-dependent, highly situated nature of open data practices; risks of diminishing qualitative methodologies as “reproducibility” becomes synonymous with quality; new risks of bias and exclusion in means of transparent evaluation; and crucial asymmetries in the Open Science relationships with industry and the public, which privileges the former and fails to fully include the latter.

Beware performative reproducibility

I worry that, by adopting the trappings of reproducibility, poor-quality work can look as if it has engaged in best practices. The problem is that sloppy work is driven by a scientific culture that overemphasizes exciting findings. When funders and journals reward showy claims at the expense of rigorous methods and reproducible results, reforms to change practice could become self-defeating. Helpful new practices, rules and policies are transformed into meaningless formalities on the way to continuing to grab headlines at any cost.

Public access to protocols of contemporary cancer randomized clinical trials | Trials | Full Text

Abstract:  Access to randomized clinical trial (RCT) protocols is necessary for the interpretation and reproducibility of the study results, but protocol availability has been lacking. We determined the prevalence of protocol availability for all published cancer RCTs in January 2020. We found that only 36.1% (48/133) of RCTs had an accessible protocol and only 11.3% of RCTs (15/133) had a publicly accessible protocol that was not behind a paywall. Only 18.0% (24/133) of RCTs were published in conjunction with the protocol on the journal website. In conclusion, few cancer RCTs have an accessible research protocol. Journals should require publication of RCT protocols along with manuscripts to improve research transparency.

 

eLife authors relay their experiences with Executable Research Articles | Labs | eLife

“We are grateful to these authors for taking their time to share their feedback with us, and for helping us showcase how Executable Research Articles can help improve the transparency, reproducibility and discoverability of research content across a variety of research subjects. Executable Research Articles are an open-source technology available to all, and we encourage any authors or publishers interested in the format to [get in touch] for more information….”

Repeat It or Take It Back

“Outside of eLife and , to an extent , PLoS , no one of scale and weight  in the commercial publishing sector has really climbed aboard the Open Science movement with a recognition of the sort of data and communication control that Open Science will require . 

So what is that requirement ? In two words – Replicability and Retraction . …”

Supporting open and reproducible science | Stanford News

“Open science is a broad goal that includes making data, data analysis, scientific processes and published results easier to access, understand and reproduce. It’s an appealing concept but, in practice, open science is difficult and, often, the costs seem to exceed the benefits. Recognizing both the shortfalls and the promise of open science, Stanford University’s Center for Open and REproducible Science (CORES) – which is part of Stanford Data Science – hopes to make the practice of open science easier, more accessible and more rewarding.

 

Since its launch in September 2020, CORES has been hard at work on the center’s first major efforts. These include developing a guide for open science practices at Stanford – called the “Open by Design” handbook – and producing workshops and a lecture series to help people learn about and contribute to open science across the university….”

FAIR Principles for Research Software (FAIR4RS Principles) | RDA

“Research software is a fundamental and vital part of research worldwide, yet there remain significant challenges to software productivity, quality, reproducibility, and sustainability. Improving the practice of scholarship is a common goal of the open science, open source software and FAIR (Findable, Accessible, Interoperable and Reusable) communities, but improving the sharing of research software has not yet been a strong focus of the latter.

To improve the FAIRness of research software, the FAIR for Research Software (FAIR4RS) Working Group has sought to understand how to apply the FAIR Guiding Principles for scientific data management and stewardship to research software, bringing together existing and new community efforts. Many of the FAIR Guiding Principles can be directly applied to research software by treating software and data as similar digital research objects. However, specific characteristics of software — such as its executability, composite nature, and continuous evolution and versioning — make it necessary to revise and extend the principles.

This document presents the first version of the FAIR Principles for Research Software (FAIR4RS Principles). It is an outcome of the FAIR for Research Software Working Group (FAIR4RS WG).

The FAIR for Research Software Working Group is jointly convened as an RDA Working Group, FORCE11 Working Group, and Research Software Alliance (ReSA) Task Force.”

Glossary Organizing document – instructions for contributors (original doc) – Google Docs

“We invite all interested to: write definitions, comment on existing definitions, add alternative definitions where applicable, and suggest relevant references. If you feel that key terms are missing, please add it – you can let us know, or ask contact us with suggestions in the FORRT slack or email sam.parsons@psy.ox.ac.uk (please CC flavio.azevedo@uni-jena.de during the period Feb 12 to March 1st). The full list of terms will form part of a larger glossary to be hosted on https://FORRT.org, once all terms have been added, the lead writing team (Parsons, Azevedo, & Elsherif) will develop an abridged version to submit as a manuscript. We outline the kinds of contributions and their correspondence to authorship in more detail in the next section. Don’t forget to add your name and details to the contributions spreadsheet….”

Association for Interdisciplinary Meta-Research and Open Science (AIMOS)

“AIMOS (the Association for Interdisciplinary Meta-Research and Open Science) seeks to advance the interdisciplinary field of meta-research by bringing together and supporting researchers in that field.

Science aims to produce robust knowledge and the concept of reproducible experiments is central to this. However the past decade has seen a ‘reproducibility crisis’ in science.

Across a number of scientific fields, such as psychology and preclinical medicine, large-scale replication projects have failed to produce evidence supporting the findings of many original studies. Meta-research will address this challenge head on….”

Enquêter sur les pratiques numériques au sein des communautés scientifiques: regards sur les outils et les données au prisme de la science ouverte – Sciencesconf.org

From Google’s English:  “THE CALL FOR PARTICIPATION AWAITS PROPOSALS ON THE FOLLOWING SUBJECTS:

The epistemological and ethical issues of observation, analysis, representation and communication on the practices of scientific communities
Methodological issues depending on whether the approaches are quantitative, qualitative or mixed
The challenges of visualizing results
The issues of reproducibility and accessibility of results (data, source code) according to legal constraints
The challenges of recounting collective practices in the context of individual observation
The challenges of receiving these surveys in their evaluative, prescriptive or even performative nature
Prescription issues in the development of applications, or any other classification solution for software or other digital services intended for scientific communities…”

Day One Project: Re-envisioning Reporting of Scientific Methods

“The information contained in the methods section of the overwhelming majority of research publications is insufficient to definitively evaluate research practices, let alone reproduce the work. Publication—and subsequent reuse—of detailed scientific methodologies can save researchers time and money, and can accelerate the pace of research overall. However, there is no existing mechanism for collective action to improve reporting of scientific methods. The Biden-Harris Administration should direct research-funding agencies to support development of new standards for reporting scientific methods. These standards would (1) address ongoing challenges in scientific reproducibility, and (2) benefit our nation’s scientific enterprise by improving research quality, reliability, and efficiency. …

Common standards are already proving invaluable for the recognition and reuse of open data. The same principles could be applied to open methods….

Compliance could be achieved through a combination of “push” incentives from publishers and “pull” incentives from funders. As is already happening for open-data standards, federal agencies can require researchers to adhere to open-methods standards in order to receive federal funding, and scientific journals can require researchers to adhere to open-methods standards in order to be eligible for publication….”  

Easyreporting simplifies the implementation of Reproducible Research layers in R software

Abstract:  During last years “irreproducibility” became a general problem in omics data analysis due to the use of sophisticated and poorly described computational procedures. For avoiding misleading results, it is necessary to inspect and reproduce the entire data analysis as a unified product. Reproducible Research (RR) provides general guidelines for public access to the analytic data and related analysis code combined with natural language documentation, allowing third-parties to reproduce the findings. We developed easyreporting, a novel R/Bioconductor package, to facilitate the implementation of an RR layer inside reports/tools. We describe the main functionalities and illustrate the organization of an analysis report using a typical case study concerning the analysis of RNA-seq data. Then, we show how to use easyreporting in other projects to trace R functions automatically. This latter feature helps developers to implement procedures that automatically keep track of the analysis steps. Easyreporting can be useful in supporting the reproducibility of any data analysis project and shows great advantages for the implementation of R packages and GUIs. It turns out to be very helpful in bioinformatics, where the complexity of the analyses makes it extremely difficult to trace all the steps and parameters used in the study.

 

Utrecht University Recognition and Rewards Vision

“By embracing Open Science as one of its five core principles1, Utrecht University aims to accelerate and improve science and scholarship and its societal impact. Open science calls for a full commitment to openness, based on a comprehensive vision regarding the relationship with society. This ongoing transition to Open Science requires us to reconsider the way in which we recognize and reward members of the academic community. It should value teamwork over individualism and calls for an open academic culture that promotes accountability, reproducibility, integrity and transparency, and where sharing (open access, FAIR data and software) and public engagement are normal daily practice. In this transition we closely align ourselves with the national VSNU program as well as developments on the international level….”

Dockstore: enhancing a community platform for sharing reproducible and accessible computational protocols | Nucleic Acids Research | Oxford Academic

Abstract:  Dockstore (https://dockstore.org/) is an open source platform for publishing, sharing, and finding bioinformatics tools and workflows. The platform has facilitated large-scale biomedical research collaborations by using cloud technologies to increase the Findability, Accessibility, Interoperability and Reusability (FAIR) of computational resources, thereby promoting the reproducibility of complex bioinformatics analyses. Dockstore supports a variety of source repositories, analysis frameworks, and language technologies to provide a seamless publishing platform for authors to create a centralized catalogue of scientific software. The ready-to-use packaging of hundreds of tools and workflows, combined with the implementation of interoperability standards, enables users to launch analyses across multiple environments. Dockstore is widely used, more than twenty-five high-profile organizations share analysis collections through the platform in a variety of workflow languages, including the Broad Institute’s GATK best practice and COVID-19 workflows (WDL), nf-core workflows (Nextflow), the Intergalactic Workflow Commission tools (Galaxy), and workflows from Seven Bridges (CWL) to highlight just a few. Here we describe the improvements made over the last four years, including the expansion of system integrations supporting authors, the addition of collaboration features and analysis platform integrations supporting users, and other enhancements that improve the overall scientific reproducibility of Dockstore content.

 

 

REPEAT (Reproducible Evidence: Practices to Enhance and Achieve Transparency)

“Replication is a cornerstone of the scientific method. Historically, public confidence in the validity of healthcare database research has been low. Drug regulators, patients, clinicians, and payers have been hesitant to trust evidence from databases due to high profile controversies with overturned and conflicting results. This has resulted in underuse of a potentially valuable source of real-world evidence.?…

Division of Phamacoepidemiology & Pharmacoeconomics [DoPE]

Brigham & Women’s Hospital and Harvard Medical School.”