Towards a new reward system for open science

The transition to an open science system affects the entire research process. The reward systems also need to be adjusted in order to support and mirror the open research landscape, but what will this work look like, and what will change? We met Gustav Nilsonne, chair of the European working group dealing with the issue and a participant in the SUHF working group on merit reviews.

Rethinking Research Assessment for the Greater Good: Findings from the RPT Project – Scholarly Communications Lab | ScholCommLab

“The review, promotion, and tenure (RPT) process is central to academic life and workplace advancement. It influences where faculty direct their attention, research, and publications. By unveiling the RPT process, we can inform actions that lead towards a greater opening of research. 

Between 2017 and 2022, we conducted a multi-year research project involving the collection and analysis of more than 850 RPT guidelines and 338 surveys with scholars from 129 research institutions across Canada and the US. Starting with a literature review of academic promotion and tenure processes, we launched six studies applying mixed methods approaches such as surveys and matrix coding.

So how do today’s universities and colleges incentivize open access research? Read on for 6 key takeaways from our studies….”

Implementation of promotion standards to discourage publishing in questionable journals: the role of the library – ScienceDirect

Abstract:  To discourage faculty members from publishing in questionable journals, tenure and promotion standards in which the librarians play an active role can been developed. These standards have been effective in terms of identifying publications in questionable outlets. However, we need to explore how these systems are perceived by the main actors in research, which are the researchers. This study explores the perception of the researchers at a university in Ghana who have been evaluated by a system implemented to discourage publishing in questionable publication outlets. We collected data using an online, largely qualitative questionnaire distributed to all faculty members that had applied for promotion since the implementation of the verification process. The results show that the majority of the faculty members are satisfied or very satisfied with the new tenure and promotion standards. There are differences across faculties, and this seems to be tied to concerns about the choice of publication outlets. Furthermore, the dissatisfied faculty members are concerned with the role of the library in the verification process whereas the satisfied trust the judgement of the librarians. We discuss implications of the results as well as future development of the standards.

 

Do German university medical centres promote robust and transparent research? A cross-sectional study of institutional policies | Health Research Policy and Systems | Full Text

Abstract:  Background

In light of replication and translational failures, biomedical research practices have recently come under scrutiny. Experts have pointed out that the current incentive structures at research institutions do not sufficiently incentivise researchers to invest in robustness and transparency and instead incentivise them to optimize their fitness in the struggle for publications and grants. This cross-sectional study aimed to describe whether and how relevant policies of university medical centres in Germany support the robust and transparent conduct of research and how prevalent traditional metrics are.

Methods

For 38 German university medical centres, we searched for institutional policies for academic degrees and academic appointments as well as websites for their core facilities and research in general between December 2020 and February 2021. We screened the documents for mentions of indicators of robust and transparent research (study registration; reporting of results; sharing of research data, code and protocols; open access; and measures to increase robustness) and for mentions of more traditional metrics of career progression (number of publications; number and value of awarded grants; impact factors; and authorship order).

Results

While open access was mentioned in 16% of PhD regulations, other indicators of robust and transparent research were mentioned in less than 10% of institutional policies for academic degrees and academic appointments. These indicators were more frequently mentioned on the core facility and general research websites. Institutional policies for academic degrees and academic appointments had frequent mentions of traditional metrics.

Conclusions

References to robust and transparent research practices are, with a few exceptions, generally uncommon in institutional policies at German university medical centres, while traditional criteria for academic promotion and tenure still prevail.

Time to recognize authorship of open data

“The open data revolution won’t happen unless the research system values the sharing of data as much as authorship on papers….

Such a practice is neither new nor confined to a specific field. But the result tends to be the same: that authors of openly shared data sets are at risk of not being given credit in a way that counts towards promotion or tenure, whereas those who are named as authors on the publication are more likely to reap benefits that advance their careers.

Such a situation is understandable as long as authorship on a publication is the main way of getting credit for a scientific contribution. But if open data were formally recognized in the same way as research articles in evaluation, hiring and promotion processes, research groups would lose at least one incentive for keeping their data sets closed….”

What senior academics can do to support reproducible and open research: a short, three-step guide | BMC Research Notes | Full Text

Abstract:  Increasingly, policies are being introduced to reward and recognise open research practices, while the adoption of such practices into research routines is being facilitated by many grassroots initiatives. However, despite this widespread endorsement and support, as well as various efforts led by early career researchers, open research is yet to be widely adopted. For open research to become the norm, initiatives should engage academics from all career stages, particularly senior academics (namely senior lecturers, readers, professors) given their routine involvement in determining the quality of research. Senior academics, however, face unique challenges in implementing policy changes and supporting grassroots initiatives. Given that—like all researchers—senior academics are motivated by self-interest, this paper lays out three feasible steps that senior academics can take to improve the quality and productivity of their research, that also serve to engender open research. These steps include changing (a) hiring criteria, (b) how scholarly outputs are credited, and (c) how we fund and publish in line with open research principles. The guidance we provide is accompanied by material for further reading.

 

 

Pontika et al. (2022) Indicators of research quality, quantity, openness and responsibility in institutional promotion, review and tenure policies across seven countries | MetaArXiv Preprints

Pontika, N., Klebel, T., Correia, A., Metzler, H., Knoth, P., & Ross-Hellauer, T. (2022, March 3). Indicators of research quality, quantity, openness and responsibility in institutional promotion, review and tenure policies across seven countries. https://doi.org/10.31222/osf.io/b9qaw

Abstract: The need to reform research assessment processes related to career advancement at research institutions has become increasingly recognised in recent years, especially to better foster open and responsible research practices. Current assessment criteria are believed to focus too heavily on inappropriate criteria related to productivity and quantity as opposed to quality, collaborative open research practices, and the socio-economic impact of research. Evidence of the extent of these issues is urgently needed to inform actions for reform, however. We analyse current practices as revealed by documentation on institutional review, promotion and tenure processes in seven countries (Austria, Brazil, Germany, India, Portugal, United Kingdom and United States of America). Through systematic coding and analysis of 143 RPT policy documents from 107 institutions for the prevalence of 17 criteria (including those related to qualitative or quantitative assessment of research, service to the institution or profession, and open and responsible research practices), we compare assessment practices across a range of international institutions to significantly broaden this evidence-base. Although prevalence of indicators varies considerably between countries, overall we find that currently open and responsible research practices are minimally rewarded and problematic practices of quantification continue to dominate.

The Value of Data and Other Non-traditional Scholarly Outputs in Academic Review, Promotion, and Tenure in Canada and the United States | The Open Handbook of Linguistic Data Management | Books Gateway | MIT Press

“This leaves us to ask: What role do activities and outputs beyond those that appear in traditional publication channels such as journals, books, and academic conferences play in review, promotion, and tenure processes? To investigate this topic, we focus on three related sub-questions:

1. What activities and outputs are mentioned in documents related to RPT?

2. How do the activities and outputs mentioned in documents related to RPT vary across institution types and disciplines?

Finally, and more specifically, given the topic of this handbook, we examine the following third sub-question:

3. To what extent and in which ways are data-related outputs mentioned in RPT documents? ”

Open science failed to penetrate academic hiring practices: A cross-sectional study – Journal of Clinical Epidemiology

Abstract:  Objective

To evaluate the current hiring practices of academic institutions around the world, with regard to the mention of advertisements for Open Science (OS) in research based, faculty and postdoctoral positions.

Study Design

Cross-sectional study, using 189 global institutions from the Center for Science and Technology (CSTS) Leiden ranking of world universities of 2017, including the U15 Group (Canadian Research-Intensive Universities), and five self-selected supplementary institutions.

Main outcome measure

The main outcome measure for our study is the level of OS in job advertisements, assessed using the Modified Open Science Modular Scheme (MOMS).

Results

After assessing 305 job advertisements for academic positions in 91 institutions, only 2 (0.6%) had any explicit mention of OS in their job advertisements on the MOMS. The sample assessed the level of open science for 39.0% Associate/Assistant professor positions, 30.8% Researcher/Postdoctoral fellow positions, and 18.7% of Tenured positions. The remaining 11.5% were for positions such as lectureship, research associate, chair, dean, director and other.

Conclusions

This study emphasizes the need for increased recognition of OS as a characteristic in research-active job advertisements. As evident in the alarmingly low percentage of job advertisements that mentioned OS (0.6%), the movement towards enhanced OS profiles across academic institutions is highly encouraged. This can be achieved through increased recognition of OS in research job advertisements and demonstrating the means in which institutions promote OS such as, encouraging preprints, publishing in open access journals, and the importance of data sharing.

ON-MERRIT D6.1 Investigating Institutional Structures of Reward & Recognition in Open Science & RRI | Zenodo

This document reports on the research conducted under Task 6.1 “Investigating institutional structures or reward and recognition in Open Science & RRI”. Our work assesses the extent to which Open Science (OS) and Responsible Research and Innovation (RRI) are embedded in promotion processes at research performing institutions and analyses the disparity between what is valued by institutions and what is valued by researchers in the context of promotion processes.

The deliverable presents two original research studies contributing to a better understanding of current reward structures, incentives and practices as they are applied across geographical boundaries:

The first study provides a systematic analysis of institutional Promotion, Review and Tenure policies (PRT) to determine the extent to which they, at this point in time, embed OS and RRI indicators. This study builds on Task 3.1 in which an initial international dataset of PRT policies was collected and annotated.
The second study is based on an international survey of active researchers. It aims to assess their attitudes towards OS and RRI as well as their experience with the application of assessment indicators in PRT processes at their institutions. Additionally, it aims to identify promising incentives that would encourage researchers to practice OS and RRI.

Our findings hence show that researchers are ready for change. Yet as we look ahead to what those changes might be, we must be careful not merely to propagate the “tyranny of metrics” responsible for many of the ills within the current system. Simply uncritically introducing further indicators accounting for OS/RRI practices may do more harm than good. We hence close with considerations of the need to change not just indicators, but rather norms, and with provisional recommendations for policy-makers, institutions and researchers (to be developed in later ON-MERRIT tasks)

Innovation, entrepreneurship, promotion, and tenure

“Academic promotion and tenure (P&T) processes that typically prioritize faculty grants and publications can fail to fully assess and value entrepreneurial, innovative endeavors (1) that can produce the kind of societal impacts that universities are increasingly being called on to provide and that many faculty and students increasingly prioritize (2, 3). A more inclusive assessment of scholarship and creative activity to better recognize and reward innovation and entrepreneurship (I&E) will require “broadening the bar” (4) to reflect evolving forms of faculty impact without diluting or increasing the requirements for advancement. Expanding what we value as scholarship can also help augment who we value as scholars and thus support a more innovative and diverse professoriate. We highlight work by the Promotion and Tenure–Innovation and Entrepreneurship (PTIE) coalition to promote policies and practices to recognize the impact of faculty I&E. We posit that this strategy can be broadly applicable (beyond I&E) to recognize the many and evolving dimensions along which faculty create societal impacts….

I&E—along with diversity, equity, and inclusion (DEI); interdisciplinary team science; open science; community engagement; and others—represent examples of the many evolving forms of scholarship for the 21stcentury faculty member. That said, these types of scholarship can be overlooked or undervalued in the process by which universities review, reward, and advance the academic workforce (8, 11, 12). As these evolutions are incorporated into the fabric of higher education, the faculty evaluation process thus needs to be updated to reflect this changing landscape….”

Coalition Members | Promotion and Tenure – Innovation and Entrepreneurship (PTIE) Summit

“Coalition members are universities committed to being a part of the conversation with on this topic. Membership as a coalition member does not constitute endorsing specific solutions or promotion & tenure (P&T) policies. Any opinions, findings, and conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of the organizations listed below. By joining the non-binding PTIE Coalition, the representative(s) from the institution are committing to the following:

 

Stay engaged with us as we develop our program for the 2020 PTIE summit – providing suggestions and insights and serving as a sounding board for ideas.
Provide a representative from your institution who will attend the Virtual National Summit on September 16-18, 2020. 
Consider adopting the recommendations from the 2020 PTIE summit for expanding P&T guidelines on your own campus.
Allow the PTIE organizing committee to list your institution’s name and/or logo on a webpage as an institution (along with our other coalition institutions) committed to advancing I&E on their campus for their faculty and students. The webpage will be housed on our www.ptie.org website and will list that your participation in this coalition consists of these four bullet points listed here….”

How should Dora be enforced? – Research Professional News

“One lesson is that the declaration’s authors did not consider redundancy as a possible outcome of research assessment, focusing instead on hiring, promotion and funding decisions. However, in my view, redundancy processes should not be delegated to crude metrics and should be informed by the principles of Dora. 

That said, it is not Dora’s job as an organisation to intervene in the gritty particulars of industrial disputes. Nor can we arbitrate in every dispute about research assessment practices within signatory organisations. …

Recently, we have re-emphasised that university signatories must make it clear to their academic staff what signing Dora means. Organisations should demonstrate their commitment to Dora’s principles to their communities, not seek accreditation from us. In doing so, they empower their staff to challenge departures from the spirit of the declaration. Grant conditions introduced by signatory funders such as the Wellcome Trust and Research England buttress this approach. 

Dora’s approach to community engagement taps into the demand for research assessment reform while acknowledging the lack of consensus on how best to go about it. The necessary reforms are complex, intersecting with the culture change needed to make the academy more open and inclusive. They also have to overcome barriers thrown up by academics comfortable with the status quo and the increasing marketisation of higher education. In such a complex landscape, Dora has no wish to be prescriptive. Rather, we need to help institutions find their own way, which will sometimes mean allowing room for course corrections….”