Annual report: a recap of the San Francisco Declaration on Research Assessment (DORA) activities in 2020 | DORA

“Over the past year, it has become increasingly clear that research assessment reform is a systems challenge that requires collective action. Point interventions simply do not solve these types of complex challenges that involve multiple stakeholders. Because of this, we dedicated our efforts in 2020 on building a community of practice and finding new ways to support organizations seeking to improve the decision-making that impacts research careers.

Current events also influenced our approach this year and evolved our thinking about research assessment reform. The Covid-19 pandemic led to the abrupt global disruption of academic research, along with many other industries. For academics with limited access to research laboratories and other on-campus resources, work stalled. Without appropriate action, this disruption will have a profound effect on the advancement and promotion of the academic workforce, and it will likely disproportionately affect women and underrepresented and minoritized researchers. So in April DORA called on institutions to redefine their expectations and clearly communicate how evaluation procedures will be modified. In May, DORA organized a webinar with Rescuing Biomedical Research to better understand specific faculty concerns as a result of the pandemic….

In the Fall of 2020, DORA initiated a new community project with Schmidt to develop a means for institutions to gauge their ability to support academic assessment interventions and set them up for success. Our goal for the project was to support the development of new practices by helping institutions analyze the outcomes of their efforts. More than 70 individuals in 26 countries and 6 continents responded to our informal survey in August, and about 35 people joined us for 3 working sessions in September. From these activities, we heard it was important to look beyond individual interventions to improve assessment, because the success of these interventions depends on institutional conditions and capabilities. We were also reminded that institutional capabilities impact interventions, so it is important not only to gauge success but also to support interventions. These and other insights led us to create SPACE to Evolve Academic Assessment: a rubric for analyzing institutional conditions and progress indicators. The first draft of the rubric was developed in the last quarter of 2020. The final version was released in 2021 after an initial pilot phase with seven members of the academic community, including a college dean, policy advisor, research administrator, faculty member, and graduate student….

Another addition to the website was a repository of case studies documenting key elements of institutional change to improve academic career assessment, such as motivations, processes, timelines, new policies, and the types of people involved. The repository, Reimagining academic assessment: stories of innovation and change, was produced in partnership with the European University Association and SPARC Europe. At the time of launch, the repository included 10 structured case studies coming from 7 universities and 3 national consortia. Nine of the 10 cases are from Europe and one is from China. The case studies have shown us the importance of coalition-building to gain bottom-up support for change. We also learned that limited awareness and capacity for incentivizing and rewarding a broader range of academic activities were challenges that all the cases had to overcome. By sharing information about the creation of new policies and practices, we hope the case studies will serve as a source of inspiration for institutions seeking to review or improve academic career assessment….

Policy progress for research assessment reform continued to gain momentum in 2020. A new national policy on research assessment in China announced in February prohibits cash rewards for research papers and indicates that institutions can no longer exclusively hire or promote researchers based on their number of publications or citations. In June, Wellcome published guidance for research organizations on how to implement responsible and fair approaches for research assessment that are grounded i

How should Dora be enforced? – Research Professional News

“One lesson is that the declaration’s authors did not consider redundancy as a possible outcome of research assessment, focusing instead on hiring, promotion and funding decisions. However, in my view, redundancy processes should not be delegated to crude metrics and should be informed by the principles of Dora. 

That said, it is not Dora’s job as an organisation to intervene in the gritty particulars of industrial disputes. Nor can we arbitrate in every dispute about research assessment practices within signatory organisations. …

Recently, we have re-emphasised that university signatories must make it clear to their academic staff what signing Dora means. Organisations should demonstrate their commitment to Dora’s principles to their communities, not seek accreditation from us. In doing so, they empower their staff to challenge departures from the spirit of the declaration. Grant conditions introduced by signatory funders such as the Wellcome Trust and Research England buttress this approach. 

Dora’s approach to community engagement taps into the demand for research assessment reform while acknowledging the lack of consensus on how best to go about it. The necessary reforms are complex, intersecting with the culture change needed to make the academy more open and inclusive. They also have to overcome barriers thrown up by academics comfortable with the status quo and the increasing marketisation of higher education. In such a complex landscape, Dora has no wish to be prescriptive. Rather, we need to help institutions find their own way, which will sometimes mean allowing room for course corrections….”

Boost for academic recognition and reward revolution

“Dutch academics are putting their foot on the gas in the rebellion against the uncritical use of journal impact factors to recognise and reward researchers, which was set in motion by the 2012 San Francisco Declaration on Research Assessment, or DORA.

From early next year, Utrecht University in the Netherlands will officially stop using the so-called ‘impact factor’ in all its hiring and promotions and judge its researchers by their commitment to open science, teamwork, public engagement and data sharing.

And despite opposition from some Dutch professors, the sweeping changes are gathering pace, with Leiden University among the Dutch institutions also pledging their support with their Academia in Motion paper….”

Dashboard will track hiring and promotion criteria

“A US$1.2 million grant will fund an effort to identify and publicize the criteria that universities around the world use to hire and promote researchers. The Declaration on Research Assessment (DORA), a global initiative to reform the evaluation of researchers, will use part of the funds to create an interactive dashboard that will shine much-needed light on a process that is often opaque and controversial, says programme director Anna Hatch, who is based in Washington DC. “When criteria are visible and transparent, universities can be held accountable,” she says. “Researchers will know how their contributions will be measured, so they can make a better case for themselves.”

DORA, conceived in 2012 at the annual meeting of the American Society for Cell Biology, called for improvements to the evaluation of researchers and the outputs of scholarly research. The declaration specifically calls for doing away with impact factors as a way to judge the merit of academics. So far, it has been signed by more than 20,000 individuals and institutions around the world.

The grant is from the Arcadia Fund, a UK-based charity that has supported many academic initiatives since its founding in 2001….”

TU Berlin unterzeichnet San Francisco Declaration on Research Assessment (TU Berlin signs DORA)

Influence of the journal impact factor in research assessment is criticized

By signing the San Francisco Declaration on Research Assessment (DORA) on July 14, 2021, Technische Universität (TU) Berlin joins an international movement of researchers* and institutions advocating for more equality and transparency in the evaluation of scientific research results. As of mid-July 2021, 2,251 organizations and 17,721 individuals worldwide have signed the declaration, including the German Research Foundation (DFG).
 

 

Einfluss des Journal Impact Factors bei der Forschungsbewertung wird kritisiert

Mit der Unterzeichnung der San Francisco Declaration on Research Assessment (DORA) am 14. Juli 2021 schließt sich die Technische Universität (TU) Berlin einer internationalen Bewegung von Forscher*innen und Institutionen an, die sich für mehr Gleichberechtigung und Transparenz bei der Evaluation wissenschaftlicher Forschungsergebnisse einsetzt. Stand Mitte Juli 2021 haben weltweit 2.251 Organisationen und 17.721 Personen die Erklärung unterzeichnet, darunter die Deutsche Forschungsgemeinschaft (DFG).

ERA Portal Austria – ERC announces its plans for 2022

On the occasion of the adoption of the ERC’s 2022 work programme, the ERC has also announced its formal endorsement of the San Francisco Declaration on Research Assessment (DORA), in line with its long-standing adherence to the highest standards of research assessment. The ERC is convinced that the broad implementation of research assessment procedures that integrate the DORA principles is the key to an equitable transition to Open Science.

Luxembourg National Research Fund develops an action plan for responsible research assessment | DORA

“Research culture is influenced by the ways in which research and researchers are assessed. For example, funding criteria such as journal prestige can cause researchers to focus their efforts on publishing in a small subset of scientific journals. Though this increases competition, it can inadvertently lead assessors to overlook other types of research outputs, contributions, and achievements. Because of this, the Luxembourg National Research Fund (FNR) is using research assessment as a key mechanism to improve research culture nationally. Sean Sapcariu, programme manager at FNR, shared their strategy at DORA’s Funder Discussion on December 16, 2020. For FNR, Sapcariu says, “The vision is to ensure that research maintains its attractiveness for current and all future generations.” 

A major part of FNR’s strategy is the introduction of a narrative CV format in all funding programs. According to Sapcariu, the goal is to shift how quality is evaluated by adopting a more holistic perspective of good research. The narrative CV format can help achieve this by framing funding proposals in the context of researchers’ wide variety of accomplishments.

FNR’s narrative CV format is based on the Résumé for Researchers, which was developed by the Royal Society in 2019, and consists of a personal statement, academic profile and relevant skillset, and key outputs, contributions, and achievements. Similar to the intent of the Résumé for Researchers, Sapcariu says, FNR is placing a premium on contributions to the generation of knowledge, development of individuals, and contributions to the wider research community and broader society….”

Why Do We Need to Change Research Evaluation Systems? — Observatory | Institute for the Future of Education

“Can we break out of this vicious cycle? Are there alternatives? Yes, there are. For some years now, various movements worldwide have sought to change the system for evaluating research. In 2012, the “San Francisco Declaration” proposed eliminating metrics based on the impact factor. There was also the Charte de la désexcellence  (“Letter of Dis-Excellence”) mentioned above. In 2015, a group of academicians signed the Leiden Manifesto, which warned of the “widespread misuse of indicators in evaluating scientific performance.” Since 2013, the group Science in Transition has sought to reform the science evaluation system. Finally, since 2016, the Collectiu InDocentia, created at the University of Valencia (Spain), has also been doing its part. …”

DORA receives $1.2M grant from Arcadia to accelerate research assessment reform | DORA

“Research assessment reform is part of the open research movement in academia that asks the question: Who and what is research for? The San Francisco Declaration on Research Assessment (DORA), an initiative that operates under the sponsorship of the American Society for Cell Biology, has been awarded a 3-year, $1.2M grant from Arcadia – a charitable fund of Lisbet Rausing and Peter Baldwin. The generous funding will support Tools to Advance Research Assessment (TARA), a project to facilitate the development of new policies and practices for academic career assessment. Project TARA is a collaboration with Sarah de Rijcke, Professor in Science and Evaluation Studies and director of the Centre for Science and Technology Studies (CWTS) at Leiden University, and Ruth Schmidt, Associate Professor at the Institute of Design at the Illinois Institute of Technology.

The grant for Project TARA will help DORA to identify, understand, and make visible the criteria and standards universities use to make hiring, promotion, and tenure decisions. This information will be used to create resources and practical guidance on the reform of research assessment for academic and scholarly institutions. The grant provides DORA with crucial support to create the following outputs:

An interactive online dashboard that tracks criteria and standards academic institutions use for hiring, review, promotion, and tenure.
A survey of U.S. academic institutions to gain a broad understanding of institutional attitudes and approaches to research assessment reform.
A toolkit of resources informed by the academic community to support academic institutions working to improve policy and practice….”

ERC plans for 2022 announced | ERC: European Research Council

“On the occasion of the adoption of this work programme, the ERC is also announcing its formal endorsement of the San Francisco Declaration on Research Assessment (DORA), in line with its long-standing adherence to the highest standards of research assessment. The ERC is convinced that the broad implementation of research assessment procedures that integrate the DORA principles is the key to an equitable transition to Open Science.”

Impact factor abandoned by Dutch university in hiring and promotion decisions

“A Dutch university says it is formally abandoning the impact factor — a standard measure of scientific success — in all hiring and promotion decisions. By early 2022, every department at Utrecht University in the Netherlands will judge its scholars by other standards, including their commitment to teamwork and their efforts to promote open science, says Paul Boselie, a governance researcher and the project leader for the university’s new Recognition and Rewards scheme. “Impact factors don’t really reflect the quality of an individual researcher or academic,” he says. “We have a strong belief that something has to change, and abandoning the impact factor is one of those changes.” …”

Incorporating Preprints into Academic Assessment | DORA

“Join DORA and ASAPbio on Tuesday Day, June 29, for a joint webinar on preprints and academic assessment….

Speakers will discuss important topics surrounding the incorporation of preprints into academic assessment: the value of considering preprints in academic assessment, how preprints can be included in existing assessment processes, and what challenges may arise along the way. Participants will have the opportunity to engage in the dialogue and ask questions of the speakers in the last section of the webinar.

 

This webinar is free to attend and open to everyone interested in improving research assessment. In particular, this webinar will aim to equip early career researchers, faculty, and academic leadership with the knowledge to advocate for the use of preprints at their institutions.”

Job Opening: DORA Program Manager | DORA

“Since 2013, more than 19,000 individuals and organizations in 145 countries have signed the San Francisco Declaration on Research Assessment (DORA) and committed to improving the ways research and researchers are assessed for hiring, promotion, and funding decisions. As an initiative, DORA raises awareness and facilitates the implementation of good practice in research assessment to catalyze change and improve equity in academia. As the DORA Program Manager, you will be working with the Program Director to ensure the execution of DORA activities and projects. The DORA Program Manager assists, plans, initiates, and executes various projects ensuring that goals are met, projects are completed on time and within budget. …”

SPACE to evolve academic assessment: A rubric for analyzing institutional conditions and progress indicators | DORA

“This is part of DORA’s toolkit of resources to support academic institutions that are improving their policies and practices. Find the other resources in the toolkit here.

Improving research and scholarship assessment practices requires the ability to analyze the outcomes of efforts and interventions. However, when conducted only at the unit level of individual interventions, these evaluations and reflections miss opportunities to understand how institutional conditions themselves set the table for the success of new efforts, or how developing institutional capabilities might improve the effectiveness and impact of these new practices at greater scale. The SPACE rubric was developed to help institutions at any stage of academic assessment reform gauge their institutional ability to support interventions and set them up for success.

Organizations can use the SPACE rubric to support the implementation of fair and responsible academic career assessment practices in two ways: First, it can help establish a baseline for the current state of infrastructural conditions, to gauge an institution’s ability to support the development and implementation of new academic assessment practices and activities. Second, the rubric can be used to retroactively analyze how strengths or gaps in these institutional conditions may have impacted the outcomes of concrete interventions targeted to specific types of academic assessment activities—such as hiring, promotion, tenure, or even graduate student evaluation—either helping or hindering progress toward those goals.

The SPACE rubric is a result of DORA’s partnership with Ruth Schmidt, Associate Professor at the Institute of Design of the Illinois Institute of Technology, who led the iterative participatory design process. The creation of the rubric was informed by nearly 75 individuals in 26 countries and 6 continents, and benefited from multiple rounds of feedback….”