DORA at 10: Looking back at the history and forward to the future of research assessment | DORA

“DORA will be 10 years old in May 2023 and we are planning to mark the occasion! We’ll be holding a weeklong celebration for DORA’s 10th Anniversary and we’re inviting you to join in by organizing an event on research assessment for your local community. We want to have conversations about what DORA has done and what we still need to do all over the globe! DORA’s 10th Anniversary Celebration will be comprised of two parts:

DORA’s 10th Anniversary Celebration will be comprised of two parts:

Two plenary online sessions to discuss the state of the field, our past decade of work, and our future plans.
A global program of local or regional events that will allow communities to share insights and challenges in reforming, innovating, and researching responsible research assessment policies and practices….”

‘The attitude of publishers is a barrier to open access’ | UKSG

“Transitioning to open research is incredibly important for the University of Liverpool for two reasons: the external environment we are now operating in, and our own philosophy and approach to research.

But there are barriers, particularly the research culture and the attitude of publishers….

In my experience, the biggest barrier is culture: researchers are used to operating in a particular way. Changing practice and mindset takes time and must be conducted sensitively.

Open research benefits all researchers, so having their support on this journey is vitally important.

Some researchers are concerned that publishing their work open access has implications for their intellectual property (IP) rights. In fact, this is a perceived problem, since the same IP protections apply to all work, whether published behind a paywall or published open access.

Despite the recognition that citation metrics are not a suitable proxy for research assessment, some researchers continue to seek the kudos of publishing in a so-called prestige journal with a high-impact factor, such as ‘Nature’.  They see this as a key career goal and worry their progression will falter without this achievement….

So, while I acknowledge there has been significant progress towards open access globally, and in particular compliance with UKRI’s open access policy, the attitude of publishers which are driven by profit margins continues to be an unacceptable barrier….”

Responsible Research Assessment I: Implementing DORA for hiring and promotion in psychology | PsychArchives

Abstract:  The use of journal impact factors and other metric indicators of research productivity, such as the h-index, has been heavily criticized for being invalid for the assessment of individual researchers and for fueling a detrimental “publish or perish” culture. Multiple initiatives call for developing alternatives to existing metrics that better reflect quality (instead of quantity) in research assessment. This report, written by a task force established by the German Psychological Society, proposes how responsible research assessment could be done in the field of psychology. We present four principles of responsible research assessment in hiring and promotion and suggest a two-step assessment procedure that combines the objectivity and efficiency of indicators with a qualitative, discursive assessment of shortlisted candidates. The main aspects of our proposal are (a) to broaden the range of relevant research contributions to include published data sets and research software, along with research papers, and (b) to place greater emphasis on quality and rigor in research evaluation.

 

The Commission signs the Agreement on Reforming Research Assessment and endorses the San Francisco Declaration on Research Assessment

“Today, the Commission has signed the Agreement on Reforming Research Assessment. The Agreement sets a common direction for changes in assessment practices for research, researchers and research organisations, with the goal to maximise the quality and impact of research. It covers the principles, commitments and timeframe for reforms and lays out the principles for the Coalition for Advancing Research Assessment (CoARA). The Coalition is a group of organisations willing to work together to implement the reform. The Coalition’s establishment is one of the main expected outcomes of the European Research Area (ERASearch for available translations of the preceding linkEN•••) Policy Agenda for 2022-2024Search for available translations of the preceding linkEN•••, which includes an action to advance the reform of the assessment system for research, researchers and institutions.

 

Together with the Agreement’s signature, today also marked the Commission’s endorsement of the San Francisco Declaration on Research Assessment(DORA), which sets recommendations to improve the evaluation of researchers and the outputs of scholarly research.The Commission signed the Agreement and endorsed DORA in its capacity of a research funding organisation….”

 

DORA’s new policy on engagement and outreach for organizational signatories | DORA

DORA is pleased to announce today the publication of our Engagement and outreach policy for organizational signatories to the San Francisco Declaration on Research Assessment.

This policy aims to give a more specific answer to the question ‘How should DORA be enforced?’, which has arisen on a number of occasions when reports have been received of signatory organizations apparently not acting in compliance with the provisions of the Declaration. On such occasions, DORA has re-iterated that it is not an accrediting organization but seeks through constructive dialogue to resolve any misunderstandings.

[…]

 

On International Open Access Week, IBEC is launching its virtual Open Science space – Institute for Bioengineering of Catalonia

“Taking advantage of the renewal of its website, IBEC has created a new virtual space dedicated to Open Science. This space is a public demonstration of IBEC’s commitment to Open Science, in accordance with its own values ??and mission, which has been realized with various initiatives and positions that the new virtual space gathers and makes visible. 

Due to its own conviction and due to the growth of the practical requirements of Open Science in the European, Spanish and Catalan research environments, the IBEC is articulating in recent years its alignment with this movement by including its principles in its own strategic plans, the approval in September 2021 of the research data management policy, the creation of an Open Science pillar with the Strategic Initiatives and Communication departments, and the incorporation of a new Knowledge Manager profile as support staff for researchers. 

These measures have made possible to carry out an internal training plan in aspects of Open Science; the improvement in the support and promotion of the publication in open access; the revision of the own research evaluation processes following the principles established in the DORA declaration; initiate the internal improvement process of research data management and facilitate its open publication, adopting the CSUC Research Data Repository (CORA), as an institutional repository; organize the didactic materials generated in the collaboration programs with the educational world, so that they are Open Educational Resources; development of citizen science projects and days for patients, or the reformulation of the Commission for Research Integrity which explicitly adds among its attributions monitoring the deployment of the Open Science strategy at IBEC; etc. …”

Community Call: Introducing the 2022 Project TARA tools to support responsible research assessment | DORA

“Join DORA for a community call to introduce two new responsible research evaluation tools and provide feedback on future tool development. The toolkit is part of Project TARA, which aims to identify, understand, and make visible the criteria and standards universities use to make hiring, promotion, and tenure decisions. This interactive call will explore these new tools, which were each created to help community members who are seeking:

Strategies on how to debias committees and deliberative processes: It is increasingly recognized that more diverse decision-making panels make better decisions. Learn how to debias your committees and decision-making processes with this one-page brief.
Ideas on how to incorporate a wider range of contributions in their evaluation policies and practices: Capturing scholarly “impact” often relies on familiar suspects like h-index, JIF, and citations, despite evidence that these indicators are narrow, often misleading, and generally insufficient to capture the full richness of scholarly work. Learn how to consider a wider breadth of contributions in assessing the value of academic activities with this one-page brief….”

Research assessment reform in action: Updates from research funders in Canada and New Zealand | DORA

“Research funding organizations are often thought of as leaders in the movement toward more responsible research evaluation practices. Often, the perception of “excellence” in research culture is filtered through the lens of who and what type of work receives funding. However, when a narrow set of indicators is used to determine who receives funding, the result can be a subsequent narrowing of academia’s perceptions of research excellence (e.g., journal impact factor (JIF), h-index). This places funders in the unique position of being able to help “set the tone” for research culture through their own efforts to reduce reliance on flawed proxy measures of quality and implement a more holistic approach to the evaluation of researchers for funding opportunities. Whether funders are seeking inspiration from their peers or insight on iterative policy development, the ability to come together to discuss reform activity is critical for achieving widespread culture change. At DORA’s June Funder Community of Practice (CoP) meetings, we heard how DORA is being implemented by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the New Zealand Ministry of Business, Innovation and Employment (MBIE)….”

Journal prestige is still important in how scholars judge one another

“Aside from an individual’s personal interactions with another academic, the perceived quality of the journal where a researcher publishes is the most influential factor when forming an opinion on their academic standing, with almost half (49 percent) of 9,609 respondents saying it is important and 12 percent saying it is most important.

Asked about citation metrics, 24 percent say a scholar’s h-index and other similar measures are important, and 5 percent say they are the most crucial factor….

Last month more than 350 organizations from more than 40 countries signed a new compact, building on the 2015 Leiden Manifesto, which would see research evaluated mainly on qualitative measures and the journal-based metrics abandoned. That agreement came nearly 10 years after the signing of the San Francisco Declaration on Research Assessment, which sought to phase out the use of journal-based metrics when making funding, appointment and promotion decisions, and which has now been signed by almost 20,000 individuals and 2,600 institutions worldwide….”

Gardner et al. (2022) Implementing the Declaration on Research Assessment: a publisher case study

Gardner, Victoria, Mark Robinson, and Elisabetta O’Connell. 2022. “Implementing the Declaration on Research Assessment: A Publisher Case Study”. Insights 35: 7. DOI: http://doi.org/10.1629/uksg.573

Abstract

There has been much debate around the role of metrics in scholarly communication, with particular focus on the misapplication of journal metrics, such as the impact factor in the assessment of research and researchers. Various initiatives have advocated for a change in this culture, including the Declaration on Research Assessment (DORA), which invites stakeholders throughout the scholarly communication ecosystem to sign up and show their support for practices designed to address the misuse of metrics. This case study provides an overview of the process undertaken by a large academic publisher (Taylor & Francis Group) in signing up to DORA and implementing some of its key practices in the hope that it will provide some guidance to others considering becoming a signatory. Our experience suggests that research, consultation and flexibility are crucial components of the process. Additionally, approaching signing with a project mindset versus a ‘sign and forget’ mentality can help organizations to understand the practical implications of signing, to anticipate and mitigate potential obstacles and to support cultural change.

 

Announcing a workshop kit for the SPACE rubric | DORA

“DORA is excited to announce the launch of the SPACE workshop kit, a resource designed for those seeking to share the SPACE rubric with their communities. This kit is intended to equip individuals with the materials needed to run their own version of the SPACE workshop for participants from their academic association, institution, department, and more. It contains English and Spanish versions of a slide deck, pre-workshop materials, and instructions for breakout room facilitators. The kit also includes a workbook for workshop participants to capture concrete existing and future actions for research assessment reform….”

The SCOPE Framework: A five-stage process for evaluating research responsibly

“The International Network of Research Management Societies (INORMS) Research Evaluation Group (REG) brings together representatives from a range of global member research management societies to work towards better, fairer, and more meaningful research evaluation. The SCOPE Framework was developed by the REG as a practical way of implementing responsible research evaluation principles to design robust evaluations. We hope this guide will provide a useful steer to research evaluators around the world who are keen to engage with best practice and provide the best service to their organisations….”