Viewpoint: As part of global shift, Utrecht University is changing how it evaluates its researchers | Science|Business

Many scientists are transitioning to a new way of working, known as open science, which will require new ways of evaluating researchers’ work. At Utrecht University we are adapting the reward system so it will incentivise this shift. The change that has received the most public attention, ditching the publishing metric known as the journal impact factor, is important, but it’s just one step in a much larger transformation. Through open science, researchers and research administrators seek to improve the quality, reproducibility and social impact of research. Open science includes open access publishing, so citizens and peers can access the fruits of publicly-funded research without paying for the privilege, and moving to a system of FAIR data, making information easy for researchers to find, access, and reuse. Open science also includes software sharing.

We moeten af van telzucht in de wetenschap – ScienceGuide

From Google’s English:  “On July 19, ScienceGuide published an open letter from 171 academics who are concerned about the new Recognition and Valuation of scientists. In fact, the signatories warn that the new ‘Recognize and Appreciate’ leads to more arbitrariness and loss of quality. This will jeopardize the international top position of Dutch science, argue the writers, which will adversely affect young academics in particular.  …

It is noticeable that these young scientists, whom the letter speaks of, do not seem to be involved in drafting the message. It is also striking that signatories to the open letter themselves are mainly at the top of the academic career ladder; 142 of the 171 signatories are even professors. As Young Science in Transition, PhD candidates Network Netherlands, PostDocNL, a large number of members of De Jonge Akademies and many other young researchers, we do not agree with the message they are proclaiming. In fact, we worry about these kinds of noises when it comes to our current and future careers. Young academics are eagerly waiting for a new system of Recognition and Appreciation. …”

Nieuwe Erkennen en waarderen schaadt Nederlandse wetenschap – ScienceGuide

From Google’s English:  “A group of 171 scientists, including 142 professors, warns in this open letter that the new Recognition and Valuation will harm Dutch science. The medical, exact and life sciences in particular are in danger of losing their international top position as a result of the new Recognition and Appreciation, because it is no longer clear how scientists are judged.

An article was recently published in Nature about the new policy of Utrecht University whereby the impact factors of scientific journals are no longer included in the evaluation of scientists. Measurable performance figures have been abandoned in favor of an ‘open science’ system and elevating the team above the individual.  

Here 171 academics warn that this new ‘Recognition and appreciation’ will lead to more arbitrariness and less quality and that this policy will have major consequences for the international recognition and appreciation of Dutch scientists. This will have negative consequences in particular for young researchers, who will no longer be able to compete internationally.  …”

Why the new Recognition & Rewards actually boosts excellent science

“During the last few weeks, several opinion pieces have appeared questioning the new Recognition and Rewards (R&R) and Open Science in Dutch academia. On July 13, the TU/e Cursor published interviews with professors who question the usefulness of a new vision on R&R (1). A day later, on July 14, the chairman of the board of NWO compared science to top sport, with an emphasis on sacrifice and top performance (2), a line of thinking that fits the traditional way of R&R in academia. On July 19, an opinion piece was published by 171 university (head) teachers and professors (3), this time in ScienceGuide questioning again the new vision of R&R. These articles, all published within a week, show that as the new R&R gains traction within universities, established scholars are questioning its usefulness and effectiveness. Like others before us (4), we would like to respond. …”

Why Do We Need to Change Research Evaluation Systems? — Observatory | Institute for the Future of Education

“Can we break out of this vicious cycle? Are there alternatives? Yes, there are. For some years now, various movements worldwide have sought to change the system for evaluating research. In 2012, the “San Francisco Declaration” proposed eliminating metrics based on the impact factor. There was also the Charte de la désexcellence  (“Letter of Dis-Excellence”) mentioned above. In 2015, a group of academicians signed the Leiden Manifesto, which warned of the “widespread misuse of indicators in evaluating scientific performance.” Since 2013, the group Science in Transition has sought to reform the science evaluation system. Finally, since 2016, the Collectiu InDocentia, created at the University of Valencia (Spain), has also been doing its part. …”

From principles to practices: Open Science at Europe’s universities: 2020-2021 EUA Open Science Survey results

“KEY RESULTS: • Open Science principles: over half (59%) of the surveyed institutions rated Open Science’s strategic importance as very high or high. Open Access to research publications was considered to be highly important for 90% of institutions, but only 60% considered its implementation level to be high. However, the gap between importance and implementation is much wider in data-related areas (RDM, FAIR and data sharing): high importance at between 55-70% of the institutions surveyed, with high levels of implementation at 15-25%. • Open Science policies: 54% of institutions have an Open Science policy and 37% are developing one. Only 9% of surveyed institutions lack an Open Science policy or are not planning to draft one. • Monitoring Open Access to research publications: 80% of institutions monitored the number of publications in their repository and 70% monitored articles published by their researchers in Open Access journals. In addition, almost 60% reported monitoring the cost of publications by their researchers in Open Access journals. • Infrastructure for Open Access to research publications: 90% of the institutions surveyed have their own repository, participate in a shared repository or both. For journal hosting or publishing platforms this figure reaches 66%, and levels out at 57% for monograph hosting/publishing. In addition, 66% of those surveyed reported that their institution has participated in or supported non-commercial Open Access publishing. Data-related skills: over 50% of the surveyed institutions reported that research data skills were only partially available. Moreover, all of the institutions that indicated the absence or partial availability of data skills, considered that more of these skills are needed at institutional level. • Emerging areas of Open Science: Approximately 50% of the respondents know of citizen science and open education activities at their institutions. • Open Science in academic assessment: In 34% of institutions, none of the Open Science elements examined by the survey were included in academic assessments. Amongst the institutions that included Open Science activities in their academic assessments, 77% took into consideration article deposition in a repository. …”

DORA receives $1.2M grant from Arcadia to accelerate research assessment reform | DORA

“Research assessment reform is part of the open research movement in academia that asks the question: Who and what is research for? The San Francisco Declaration on Research Assessment (DORA), an initiative that operates under the sponsorship of the American Society for Cell Biology, has been awarded a 3-year, $1.2M grant from Arcadia – a charitable fund of Lisbet Rausing and Peter Baldwin. The generous funding will support Tools to Advance Research Assessment (TARA), a project to facilitate the development of new policies and practices for academic career assessment. Project TARA is a collaboration with Sarah de Rijcke, Professor in Science and Evaluation Studies and director of the Centre for Science and Technology Studies (CWTS) at Leiden University, and Ruth Schmidt, Associate Professor at the Institute of Design at the Illinois Institute of Technology.

The grant for Project TARA will help DORA to identify, understand, and make visible the criteria and standards universities use to make hiring, promotion, and tenure decisions. This information will be used to create resources and practical guidance on the reform of research assessment for academic and scholarly institutions. The grant provides DORA with crucial support to create the following outputs:

An interactive online dashboard that tracks criteria and standards academic institutions use for hiring, review, promotion, and tenure.
A survey of U.S. academic institutions to gain a broad understanding of institutional attitudes and approaches to research assessment reform.
A toolkit of resources informed by the academic community to support academic institutions working to improve policy and practice….”

ERC plans for 2022 announced | ERC: European Research Council

“On the occasion of the adoption of this work programme, the ERC is also announcing its formal endorsement of the San Francisco Declaration on Research Assessment (DORA), in line with its long-standing adherence to the highest standards of research assessment. The ERC is convinced that the broad implementation of research assessment procedures that integrate the DORA principles is the key to an equitable transition to Open Science.”

Analyzing Effects of Bias in Research | YIP Institute

“Such is the issue of availability today, that the ability for someone to access research is based almost entirely on their affiliations to any well-funded research program. Michael Eisen, a professor of genetics, genomics, and development at the University of California, Berkeley, has been a staunch proponent of open access and has long warned about the postponement of innovation and discovery due to slow accessibility. As Michael Eisen elaborates in his UC Berkeley Interview from his “point of view, science is getting a raw deal out of this arrangement, because it is providing all the money, but not getting access for everybody on Earth”.  Not only is science getting the short end of the deal but so are taxpayers who are responsible for more than 40% of all academic research and development.

The largest issue that Professor Eisen points out is how institutions choose to hire or tenure solely based on publication in prestigious academic journals. Effectively, what Universities are doing is handing the process of professorial evaluation to publishers like Elsevier and paying for it in the thousands of dollars they spend on journal subscriptions. 

The cost of these subscriptions help form yet another divide between educational institutions in developed and developing countries. As Peter Suber wrote in his book Open Access, “In 2008, Harvard subscribed to 98,900 serials and Yale to 73,900. The best-funded research library in India, at the Indian Institute of Science, subscribed to 10,600. Several sub-Saharan African university libraries subscribed to zero.” The cost climbing cost of these journals has pushed even endowment rich universities like Harvard to take on cancellation efforts….”

Ouvrir la Science – Deuxième Plan national pour la science ouverte

From Google’s English:  “The National Open Science Plan announced in 2018 by the Minister of Higher Education, Research and Innovation, Frédérique Vidal, has enabled France to adopt a coherent and dynamic policy in the field of open science, coordinated by the Committee for Open Science, which brings together the ministry, research and higher education institutions and the scientific community. After three years of implementation, the progress made is notable. The rate of French scientific publications in open access rose from 41% to 56%. The National Open Science Fund was created, it launched two calls for projects in favor of open scientific publication and it supported structuring international initiatives. The National Research Agency and other funding agencies now require open access to publications and the drafting of data management plans for the projects they fund. The function of ministerial research data administrator has been created and a network is being deployed in the establishments. About twenty universities and research organizations have adopted an open science policy. Several guides and recommendations for putting open science into practice have been published. About twenty universities and research organizations have adopted an open science policy. Several guides and recommendations for putting open science into practice have been published. About twenty universities and research organizations have adopted an open science policy. Several guides and recommendations for putting open science into practice have been published.

The steps already taken and the evolution of the international context invite us to extend, renew and strengthen our commitments by adopting a second National Plan for Open Science, the effects of which will be deployed until 2024. With this new plan, France is continuing the ambitious trajectory initiated by the law for a digital republic of 2016 and confirmed by the research programming law of 2020, which includes open science in the missions of researchers and teacher-researchers.

This second National Plan extends its scope to source codes resulting from research, it structures actions in favor of the opening or sharing of data through the creation of the Research Data Gouv platform, it multiplies the levers of transformation in order to generalize open science practices and it presents disciplinary and thematic variations. It is firmly in line with a European ambition and proposes, within the framework of the French Presidency of the European Union, to act to take effective account of open science practices in individual and collective research evaluations. It is about initiating a process of sustainable transformation in order to make open science a common and shared practice…”

ERAC Guideline Paper ‘Research evaluation in a context of Open Science and gender equality’

The European Research Area and Innovation Committee (ERAC) “triangle task force” has recently published the guideline paper “Research evaluation in a context of Open Science and gender equality”. The ERAC “triangle task force” combines combines forces of the ERAC group on open science, gender and human resources. This report provides stakeholders involved in research evaluation reforms with a set of guidelines that aim at fostering both Open Science and gender equality. Both topics are key dimensions in the implementation of a new European Research Area and provide policy and decision makers, funders as well as researchers with a unique opportunity to substantially renegotiate, through evaluation, the social roles and responsibilities of publicly funded research, as well as to rethink the science system as a whole.

From principles to practices: Open Science at Europe’s universities: 2020-2021 EUA Open Science Survey results

“KEY RESULTS: • Open Science principles: over half (59%) of the surveyed institutions rated Open Science’s strategic importance as very high or high. Open Access to research publications was considered to be highly important for 90% of institutions, but only 60% considered its implementation level to be high. However, the gap between importance and implementation is much wider in data-related areas (RDM, FAIR and data sharing): high importance at between 55-70% of the institutions surveyed, with high levels of implementation at 15-25%. • Open Science policies: 54% of institutions have an Open Science policy and 37% are developing one. Only 9% of surveyed institutions lack an Open Science policy or are not planning to draft one. • Monitoring Open Access to research publications: 80% of institutions monitored the number of publications in their repository and 70% monitored articles published by their researchers in Open Access journals. In addition, almost 60% reported monitoring the cost of publications by their researchers in Open Access journals. • Infrastructure for Open Access to research publications: 90% of the institutions surveyed have their own repository, participate in a shared repository or both. For journal hosting or publishing platforms this figure reaches 66%, and levels out at 57% for monograph hosting/publishing. In addition, 66% of those surveyed reported that their institution has participated in or supported non-commercial Open Access publishing. • Data-related skills: over 50% of the surveyed institutions reported that research data skills were only partially available. Moreover, all of the institutions that indicated the absence or partial availability of data skills, considered that more of these skills are needed at institutional level. • Emerging areas of Open Science: Approximately 50% of the respondents know of citizen science and open education activities at their institutions. • Open Science in academic assessment: In 34% of institutions, none of the Open Science elements examined by the survey were included in academic assessments. Amongst the institutions that included Open Science activities in their academic assessments, 77% took into consideration article deposition in a repository….”

Impact factor abandoned by Dutch university in hiring and promotion decisions

“A Dutch university says it is formally abandoning the impact factor — a standard measure of scientific success — in all hiring and promotion decisions. By early 2022, every department at Utrecht University in the Netherlands will judge its scholars by other standards, including their commitment to teamwork and their efforts to promote open science, says Paul Boselie, a governance researcher and the project leader for the university’s new Recognition and Rewards scheme. “Impact factors don’t really reflect the quality of an individual researcher or academic,” he says. “We have a strong belief that something has to change, and abandoning the impact factor is one of those changes.” …”

Incorporating Preprints into Academic Assessment | DORA

“Join DORA and ASAPbio on Tuesday Day, June 29, for a joint webinar on preprints and academic assessment….

Speakers will discuss important topics surrounding the incorporation of preprints into academic assessment: the value of considering preprints in academic assessment, how preprints can be included in existing assessment processes, and what challenges may arise along the way. Participants will have the opportunity to engage in the dialogue and ask questions of the speakers in the last section of the webinar.

 

This webinar is free to attend and open to everyone interested in improving research assessment. In particular, this webinar will aim to equip early career researchers, faculty, and academic leadership with the knowledge to advocate for the use of preprints at their institutions.”