Data Management Plans: Implications for Automated Analyses

Abstract:  Data management plans (DMPs) are an essential part of planning data-driven research projects and ensuring long-term access and use of research data and digital objects; however, as text-based documents, DMPs must be analyzed manually for conformance to funder requirements. This study presents a comparison of DMPs evaluations for 21 funded projects using 1) an automated means of analysis to identify elements that align with best practices in support of open research initiatives and 2) a manually-applied scorecard measuring these same elements. The automated analysis revealed that terms related to availability (90% of DMPs), metadata (86% of DMPs), and sharing (81% of DMPs) were reliably supplied. Manual analysis revealed 86% (n = 18) of funded DMPs were adequate, with strong discussions of data management personnel (average score: 2 out of 2), data sharing (average score 1.83 out of 2), and limitations to data sharing (average score: 1.65 out of 2). This study reveals that the automated approach to DMP assessment yields less granular yet similar results to manual assessments of the DMPs that are more efficiently produced. Additional observations and recommendations are also presented to make data management planning exercises and automated analysis even more useful going forward.

 

Confused by open-access policies? These tools can help

“Funding-agency policies mandating that scientific papers and data are made publicly available have helped to drive the adoption of preprints, open-access publishing and data repositories. But agencies often struggle to measure how closely grant recipients comply with the funding policies. Awardees, and the institutes that employ them, can struggle to ensure they are following the rules. Now, digital tools are cropping up to help both sides of the funding equation stick to the regulations.

The Bill & Melinda Gates Foundation in Seattle, Washington, has invested US$1.8 million to support the development of OA.Report, a tool that helps funders to track awardee compliance with foundation open-access policies. Developed by OA.Works in London, OA.Report uses text-mining techniques to match articles with the funder that supported the work, by sifting through academic papers and open-access metadata. The software also tracks article-processing charges, as well as the subsequent reports that summarize the outcomes of grants….”

Guest Post – “We are ready to move forward”: A Professional Society’s Route to Open Access – The Scholarly Kitchen

“Some publishers have been quoted as saying it is too soon to tell if this mandate will impact their journals. My colleagues and I at the ANS have known for some time that our journals would be impacted by the wider movement toward open research publishing. In many ways, the OSTP’s latest public access guidance is a big win for federally funded researchers and the entire nuclear community. ANS has recently published numerous OA supplements alongside some national US labs and the benefit to authors and researchers is far-reaching. The Nelson Memo only reaffirms that we as publishers must continue to be proactive in finding sustainable solutions that work for authors, the publishers of those journals and for society. We are ready to move forward.

But it is not an all-or-nothing approach. The ANS has long taken a progressive stance to ensure that we stay at the fore of the evolution of scholarly publishing, whilst ensuring that we continue to meet the needs of our members and our wider research community….”

 

Research Publications and Copyright Policy | Library | The University of Sheffield

“This guidance is designed to ensure that University of Sheffield staff and PGRs can comply with the Research Publications and Copyright policy. The policy enables authors to control copyright, as set out in the University’s new IP policy, to their own journal articles and conference proceedings papers, apply a CC BY licence to them and make them available via the institutional repository, White Rose Research Online (WRRO) without embargo.

This will help you to comply with external funding requirements for open access as well as supporting our commitment to enabling and promoting research excellence across our community as set out in the University’s Statement of Open Research…”

‘The attitude of publishers is a barrier to open access’ | UKSG

“Transitioning to open research is incredibly important for the University of Liverpool for two reasons: the external environment we are now operating in, and our own philosophy and approach to research.

But there are barriers, particularly the research culture and the attitude of publishers….

In my experience, the biggest barrier is culture: researchers are used to operating in a particular way. Changing practice and mindset takes time and must be conducted sensitively.

Open research benefits all researchers, so having their support on this journey is vitally important.

Some researchers are concerned that publishing their work open access has implications for their intellectual property (IP) rights. In fact, this is a perceived problem, since the same IP protections apply to all work, whether published behind a paywall or published open access.

Despite the recognition that citation metrics are not a suitable proxy for research assessment, some researchers continue to seek the kudos of publishing in a so-called prestige journal with a high-impact factor, such as ‘Nature’.  They see this as a key career goal and worry their progression will falter without this achievement….

So, while I acknowledge there has been significant progress towards open access globally, and in particular compliance with UKRI’s open access policy, the attitude of publishers which are driven by profit margins continues to be an unacceptable barrier….”

Preparing for the 2023 Data Management and Sharing Policy | Data Science at NIH

“Sharing scientific data has been a priority for NIH for decades — data sharing accelerates biomedical research discovery and innovation, enhances research rigor and reproducibility, provides accessibility to high-value datasets, and promotes data reuse for future research studies. Ultimately, sharing data speeds up the process of turning research results into knowledge, products, and procedures to improve human health.

The NIH Data Management and Sharing (DMS) Policy was released October 29, 2020 and will take effect on January 25, 2023. Specifically, this date applies to:

Competing grant applications submitted to NIH on or after January 25, 2023
Proposals for contracts that are submitted to NIH on or after January 25, 2023
NIH Intramural Research Projects conducted on or after January 25, 2023
Other funding agreements (e.g., Other Transactions) that are executed on or after January 25, 2023, unless otherwise stipulated by NIH.

Under the NIH DMS Policy, NIH expects researchers to maximize the appropriate sharing of scientific data, and consider legal, ethical, or technical issues that may limit the extent of data sharing and preservation. All applicants planning to generate scientific data must prepare a DMS Plan that describes how the data will be managed and shared….

Individuals receiving NIH funding to generate scientific data must comply with the DMS Policy. This compliance level differs from the 2003 DMS Policy: Previously, only awards totaling $500,000 per year or more had to comply with the policy. Be sure to check out the complete list of NIH activity codes subject to the DMS Policy, as well as the specific funding opportunity announcement to determine if the DMS Policy applies to an application….”

Publishing in transformative journals to make your research openly available | Jisc

“Transformative journals are a key route to open access. Jisc is negotiating with publishers on behalf of the sector to ensure that their journals are funder compliant. This guide will show researchers how they can publish their research open access in a transformative journal in order to meet funding requirements.  …”

RI Webinar: Conforming to the REF: An international view – 1579136

“Summary

The REF is a UK-specific measure for research institutions to assess the quality of their research output and is pertinent to libraries, research offices, university planning departments and institutions. But this webcast will aim to look at this within a global context and explore other frameworks that are in place around the world.

Key takeaways

Learn how the outputs of scholarly research are evaluated globally

Hear from experts about best practices in the assessment of researchers and scholarly research

Key insights into why representation of researchers in the design of research assessment practices across the world is crucial…”

Subject analysis of routes to compliance for UKRI funded authors

As part of our work to support the implementation of the new UKRI Open Access policy, we have rapidly scaled up our negotiations and are now working with over 300 publishers (and have confirmed compliance of 100 more) across a broad range of subject areas using a range of open access publishing routes. The aim is to make it easy for authors to increase the reach and impact of their research, and to comply with UKRI, and other research funders’, open access (OA) policies.

We carry out detailed analysis of previous publishing patterns of UKRI funded authors and assign each journal a route to compliance category (Table 1). For journals that are non-compliant or ineligible for UKRI funds we then evaluate the gaps between the publisher’s existing publication routes and the UKRI OA policy requirements, and we work with these publishers to secure UKRI compliant agreements. All our negotiations with publishers are sector-led and designed to meet both sector and funder requirements.

Opinion: The Promise and Plight of Open Data | TS Digest | The Scientist

“At the same time, open data allow anyone to reproduce a study’s analyses and validate its findings. Occasionally, readers identify errors in the data or analyses that slipped through the peer-review process. These errors can be handled through published corrections or retractions, depending on their severity. One would expect open data to result in more errors being identified and fixed in published papers. 

But are journals with open-data policies more likely than their traditional counterparts to correct published research with erroneous results? To answer this, we collected information on data policies and article retractions for 199 journals that publish research in the fields of ecology and evolution, and compared retraction rates before and after open-data policies were implemented. 

Surprisingly, we found no detectable link between data-sharing policies and annual rates of article retractions. We also found that the publication of corrections was not affected by requirements to share data, and that these results persisted after accounting for differences in publication rates among journals and time lags between policy implementation and study publication due to the peer-review process. While our analysis was restricted to studies in ecology and evolution, colleagues in psychology and medicine have suggested to us that they expect similar patterns in their fields of study. 

Do these results mean that open-data policies are ineffective? No. There is no doubt that open data promote transparency, but our results suggest that a greater potential for error detection does not necessarily translate into greater error correction. We propose three additional practices, some of which could actually improve open-data practices, to help science self-correct. …”

Open access policy update: December 2022 | STM Publishing News

“UKRI has published updated information to support funded research organisations and researchers to meet its new open access policy.

Peer reviewed research articles that acknowledge UK Research and Innovation (UKRI) funding have been required to comply with UKRI’s open access policy since 1 April 2022.

From 1 January 2024, monographs, book chapters and edited collections that acknowledge UKRI funding will also need to be published open access….”

Five principles for monitoring and evaluation: The case of UKRI’s Open Access policy | Research Consulting

by Andrea Chiarelli

Between August and November 2022, almost 80 individuals from across the research and publishing landscape contributed to a study we delivered on behalf of UK Research and Innovation (UKRI), to support the development of a monitoring and evaluation (M&E) framework for their Open Access (OA) policy.

The framework will help UKRI and the sector assess open access progress, levels of compliance with the policy and its effectiveness. It will also seek to establish insights into open access publication trends across the UK and, where possible, their impact on academic practices and society.

We are in the process of finalising project outputs for public dissemination alongside our associates Bianca Kramer and Cameron Neylon, but we are now in a position to share some high-level findings and next steps. This blog covers five key principles we identified from our discussions with the research and publishing communities, as well as considering the implications for UKRI’s future M&E efforts.

 

Open access Policy Update: December 2022 – UKRI

UKRI has published updated information to support funded research organisations and researchers to meet its new open access policy.

Peer reviewed research articles that acknowledge UK Research and Innovation (UKRI) funding have been required to comply with UKRI’s open access policy since 1 April 2022.

From 1 January 2024, monographs, book chapters and edited collections that acknowledge UKRI funding will also need to be published open access.

IEEE Commits its Entire Hybrid Journal Portfolio to Transformative Journal Status Aligned with Plan S | STM Publishing News

“IEEE, the world’s largest technical professional organization dedicated to advancing technology for humanity, announced today that it has committed its full portfolio of more than 160 hybrid journals, which publish both open access and subscription-based content, to become Transformative Journals under Plan S. 

This commitment means that any authors receiving research grants from Coalition S, a group of research funders, are compliant with Plan S requirements when publishing their research articles in any IEEE fully open access or hybrid journals. In addition to the existing direct open access agreements with hundreds of institutions, all of IEEE’s hybrid journals now qualify as ‘Transformative Journals’ under Plan S….”

Publications | Free Full-Text | Adoption of Transparency and Openness Promotion (TOP) Guidelines across Journals

Abstract:  Journal policies continuously evolve to enable knowledge sharing and support reproducible science. However, that change happens within a certain framework. Eight modular standards with three levels of increasing stringency make Transparency and Openness Promotion (TOP) guidelines which can be used to evaluate to what extent and with which stringency journals promote open science. Guidelines define standards for data citation, transparency of data, material, code and design and analysis, replication, plan and study pre-registration, and two effective interventions: “Registered reports” and “Open science badges”, and levels of adoption summed up across standards define journal’s TOP Factor. In this paper, we analysed the status of adoption of TOP guidelines across two thousand journals reported in the TOP Factor metrics. We show that the majority of the journals’ policies align with at least one of the TOP’s standards, most likely “Data citation” (70%) followed by “Data transparency” (19%). Two-thirds of adoptions of TOP standard are of the stringency Level 1 (less stringent), whereas only 9% is of the stringency Level 3. Adoption of TOP standards differs across science disciplines and multidisciplinary journals (N = 1505) and journals from social sciences (N = 1077) show the greatest number of adoptions. Improvement of the measures that journals take to implement open science practices could be done: (1) discipline-specific, (2) journals that have not yet adopted TOP guidelines could do so, (3) the stringency of adoptions could be increased.