Defining, Measuring, and Rewarding Scholarly Impact: Mind the Level of Analysis

Abstract:  We address the pervasive practice of using a journal’s impact factor (JIF) as a proxy to assess individual researcher and article scholarly impact, and the grossly incorrect inferences this process generates. This invalid practice occurs because of confusion about the definition and measurement of impact at different levels of analysis. We illustrate the severity of the errors that occur when using JIF to evaluate individual scholarly impact, and advocate for an immediate moratorium on the exclusive use of JIF and other journal-level (i.e., higher level of analysis) measures when assessing the impact of individual researchers and individual articles (i.e., lower level of analysis). Given the importance and interest in assessing the scholarly impact of researchers and articles, we delineate measures that are level-appropriate and readily available. We discuss implications for the careers of researchers and educators and the administration and future of business schools, and conclude with actionable recommendations for internal and external stakeholders regarding the assessment of scholarly impact that can drive positive change in academic environments.

[Undated]

UNESCO Recommendation on Open Science – Free Knowledge Advocacy Group EU

“In late 2021, the UNESCO General Assembly approved a new Recommendation on Open Science. All the member states agreed on a final version, that for the first time provides an official definition of what open science is, and that calls for legal and policy changes in favor of open science. As a recommendation is the strongest policy tool of UNESCO, “intended to influence the development of national laws and practices”, this is important news for the entire scientific community. 

The recommendation presents a framework on, and principles for, open science. It aims to build a common understanding on the topic, and calls for publicly funded research to be aligned with the principles: transparency, scrutiny, critique and reproducibility; equality of opportunities; responsibility, respect and accountability; collaboration, participation and inclusion; flexibility, and sustainability. 

It asks for more dialogue between the public and the private sector, and for new, innovative means and methods to be developed for open science. Finally, the recommendation stresses the importance of citizen science and crowdsourcing, and the need for cooperation between different kinds of actors, nationally and internationally.

In Sweden, the recommendation is currently being discussed with stakeholders. A few weeks ago, Wikimedia Sverige was invited by the Swedish National UNESCO Commission to a round table conversation on the subject. Other than Wikimedia Sverige, organisations and institutions such as the Association of Swedish Higher Education Institutions, the Swedish Research Council, the Ministry for Education and the National Library, took part – many of those who will bear the largest responsibility for putting the recommendations in practice. …”

It’s Time to Terminate Social Work’s Relationship with the Impact Factor | Social Work | Oxford Academic

“As a journal-level metric, the IF is unable to assess the value of any given article or author. To make this inference, one would need to read the article and assess its claims, scientific rigor, methodological soundness, and broader implications. What’s more, the IF (which represents the average number of citations across a finite set of eligible articles) is vulnerable to the skewness in citation rates among articles (Nature, 2005) and to the manipulation, negotiation, and gaming of its calculation among stakeholders (Ioannidis & Thombs, 2019). At a more fundamental level the IF does not capture journal functioning such as improvements to (or worsening of) internal evaluative processes (e.g., effectiveness of peer review, changes to submission instructions and policies, use and adherence to reporting guidelines, etc.; Dunleavy, 2022). These and other issues are explored in more depth by Seglen (1997)….

In light of these limitations, social work should de-emphasize the IF and instead embrace a new set of evaluative tools. The San Francisco Declaration on Research Assessment (American Society for Cell Biology, 2013)—and more recently the Leiden Manifesto (Hicks et al., 2015)—typify such efforts. They encourage stakeholders (i.e., academic institutions, journals, funders, researchers) to consider using a multitude of qualitative and quantitative alternative metrics (i.e., “altmetrics”; Priem et al., 2012; see also https://metrics-toolkit.org/metrics/) when judging scholarly output—whether it be a journal article, a grant proposal, or even a hiring or tenure packet. …”

 

Recommendations for repositories and scientific gateways from a neuroscience perspective | Scientific Data

“Digital services such as repositories and science gateways have become key resources for the neuroscience community, but users often have a hard time orienting themselves in the service landscape to find the best fit for their particular needs. INCF has developed a set of recommendations and associated criteria for choosing or setting up and running a repository or scientific gateway, intended for the neuroscience community, with a FAIR neuroscience perspective….”

Management and maintenance of research data by researchers in Zimbabwe | Emerald Insight

Abstract:  Purpose

The concept of research data management (RDM) is new in Zimbabwe and other developing countries. Research institutions are developing research data repositories and promoting the archiving of research data. As a way of creating awareness to researchers on RDM, the purpose of this paper is to determine how researchers are managing their research data and whether they are aware of the developments that are taking place in RDM.

Design/methodology/approach

A survey using a mixed method approach was done and an online questionnaire was administered to 100 researchers in thirty research institutions in Zimbabwe. Purposive sampling was done by choosing participants from the authors of articles published in journals indexed by Google Scholar, Scopus and Web of Science. Interviews were done with five top researchers. The data was analysed using NVIVO. The results were presented thematically. The questionnaire was distributed using the research offices of the selected 30 research institutions. There was a 75% response rate.

Findings

The findings indicated that all the researchers are aware of the traditional way of managing research data. A total of 70% of the respondents are not aware of the current trends in RDM services, as they are keeping their data on machines and external hard drives, while 97.3% perceive RDM services as useful, as it is now a requirement when applying for research grants. Librarians have a bigger role to play in creating awareness on RDM among researchers and hosting the data repositories for archiving research data.

Practical implications

Research institutions can invest in research data services and develop data repositories. Librarians will participate in educating researchers to come up with data management plans before they embark on a research project. This study also helps to showcase the strategies that can be used in awareness creation campaigns. The findings can also be used in teaching RDM in library schools and influence public policy both at institutional and national level.

Social implications

This study will assist in building capacity among stakeholders about RDM. Based on the findings, research institutions should prioritise research data services to develop skills and knowledge among librarians and researchers.

Originality/value

Few researches on RDM practices in Zimbabwe were done previously. Most of the papers that were published document the perception of librarians towards RDM, but this study focused mainly on researchers’ awareness and perception. The subject is still new and people are beginning to research on it and create awareness amongst the stakeholders in Zimbabwe.

MIT Open Access Task Force | Massachusetts Institute of Technology

“The MIT Ad Hoc Task Force on Open Access to MIT’s Research, chaired by Class of 1922 Professor of Electrical Engineering and Computer Science Hal Abelson and Director of Libraries Chris Bourg, will lead an Institute-wide discussion of ways in which current MIT open access policies and practices might be updated or revised to further the Institute’s mission of disseminating the fruits of its research and scholarship as widely as possible.”

An Open Science Roadmap for Swedish Higher Education Institutions | Nordic Perspectives on Open Science

Abstract:  In the spring of 2021, a National Open Science Roadmap for Swedish Higher Education Institutions (HEI) was adopted by The Association of Swedish HEIs. The roadmap’s eight principles aim to guide the HEIs’ development of local structures and processes, speed up their concrete actions and encourage their collaboration in the shift to Open Science. The recommendations are concentrated on specific measures for open access to research data and research publications at HEIs. The primary target group for the roadmap is university management at Swedish HEIs. In the spring of 2022 the roadmap is to be supplemented by an action plan for Open Science.

 

Need for universal acceptance of preprinting by editors of journals of health professional education | SpringerLink

“While publishers in multiple fields are adopting preprints [2], we have discovered a great deal of confusion about the pros and cons of preprinting as well as disparity in publishers’ policies regarding preprinting in health professions education (HPE). In seeking to resolve this confusion, we documented preprint policies at 74 journals within HPE (e.g. nursing, medicine, pharmacy, dentistry, rehabilitation sciences, nutrition). We culled preprint policies for 43 (58%) journals using journal websites, JISC’s Sherpa Romeo tool, and Wikipedia’s list of academic publishers by preprint policy. We then obtained information from email solicitations for an additional 27 (36%), leaving us without information for 4 (5%). Of the 70 journals for which we have information, 53 (76%) will review/accept preprinted manuscripts; 11 (16%) do not, and 6 (9%) are unclear or make decisions on a case-by-case basis. (For a link to our list of HPE journals and our understanding of their policies regarding preprinted manuscripts, see https://jahse.med.utah.edu/submission/ and select “Where to Publish”.) No wonder there is confusion.

We encourage our colleagues across the health professions to join our call to eliminate this confusion by encouraging all HPE journals to support and promote preprinting. The value of preprinting has only become more important during the COVID-19 pandemic [3]. Being able to preprint scholarship prior to formal submission enhances formative review and revision, augments the benefits of peer coaching, and promotes higher quality publications. Preprinting also makes work available to others more quickly, which can enhance collaboration and uptake of new ideas without compromising the eventual copyright of the final published product.”

SocArXiv Papers | Targeted, actionable and fair: reviewer reports as feedback and its effect on ECR career choices

Abstract:  Previous studies of the use of peer review for the allocation of competitive funding agencies have concentrated on questions of efficiency and how to make the ‘best’ decision, by ensuring that successful applicants are also the more productive or visible in the long term. This paper examines the function of peer review by examining how it can be used as a participatory research governance tool by focusing on the function feedback plays in assisting in the development of ECR applicants. Using a combination of survey, interviews and linguistic-based coding of reviewer reports, this study explores how reviewer reports provided to unsuccessful applicants as an artefact of the peer-review decision making process, can be considered as a method of feedback. Specifically, it examines which components of this feedback underpinned their decisions to re-submit their grant applications following first-failure; change their research topics or withdraw from academia entirely. Peer review feedback, we argue, sends signals to applicants to encourage them to persist (continue) or switch (not continue) even when the initial application has failed. The results lead to identification of standards of feedback for funding agencies and peer-reviewers to promote when providing reviewer feedback to applicants as part of their peer review process. The results also highlight a function of peer review overlooked by current research which is not concentrated solely on the development of an outcome, to one that can be used effectively to support the development of individuals and their future research plans.

 

Rethinking Research Assessment for the Greater Good: Findings from the RPT Project – Scholarly Communications Lab | ScholCommLab

“The review, promotion, and tenure (RPT) process is central to academic life and workplace advancement. It influences where faculty direct their attention, research, and publications. By unveiling the RPT process, we can inform actions that lead towards a greater opening of research. 

Between 2017 and 2022, we conducted a multi-year research project involving the collection and analysis of more than 850 RPT guidelines and 338 surveys with scholars from 129 research institutions across Canada and the US. Starting with a literature review of academic promotion and tenure processes, we launched six studies applying mixed methods approaches such as surveys and matrix coding.

So how do today’s universities and colleges incentivize open access research? Read on for 6 key takeaways from our studies….”

Dismantling the ivory tower’s knowledge boundaries

“The major shift to open access during the pandemic began with the Free Read initiative, which launched the petition “

Unlock Coronavirus Research” for scientists in early February of 2020 and to which highly reputable medical publishers quickly responded. Before the pandemic, up to 75 percent of scholarly publications were behind a paywall. By comparison, a preliminary study of over 5,600 articles on PubMed suggests that more than 95 percent of scholarly articles related to COVID-19 are now freely available. This increase in accessibility resulted from the rapid adaptation by biomedical journals and publishers, including Elsevier, Springer Nature, Cell Press, New England Journal of Medicine, and The Lancet. These journals and publishers granted open access to research on COVID-19 research, often making it 

immediately accessible on the platform PubMed Central and similar public repositories. Free and open access to COVID-19 research quickly became the new normal for biomedicine, with available findings directly impacting the development of treatment protocols and vaccines. Yet the pandemic became more than a health crisis. Understanding the social, psychological, and economic implications of the pandemic were imperative to its continued management.

Social science research, which delivers insights into human behaviors, relationships, and institutions, was instrumental to policymaking and healthcare solution development during the COVID-19 pandemic. The importance of social science research to pandemic management was demonstrated by the 

shift in the topic of COVID-19 papers, from the initial focus on disease modeling, hospital mortality, diagnostics, and testing to an increasing focus on topics such as business closure, remote work, geographic mobility and migration, inequality, managerial decision-making, as well as accelerating innovation. Once the basic science on the virus were established, research on creating societal and economic resilience played an even larger role for beating the COVID-19 pandemic. One clear area that demonstrated the importance of social science research in informing COVID-19 management was the rollout of vaccines. Psychological, marketing, and information systems research played a central role in vaccine uptake across communities. A recent report by the National Institutes of Health called for the use of evidence-based strategies, such as 

behavioral nudges and strategic social norms, to increase vaccine uptake….”

 

 

UNESCO’s open science recommendations: A publisher action plan

“To help make sense of UNESCO’s proposed model for accelerating an equitable vision of open science, this post will offer a simple overview of the recommendations and unpack specific actions any publisher can set in motion today to collaborate in scaling open access. Publishers have the opportunity to adopt new business models and implement a range of operational and cultural changes necessary to open up the research lifecycle. Let’s start with a brief look at how UNESCO envisions the future of open scholarship….”

Usability and Accessibility of Publicly Available Patient Sa… : Journal of Patient Safety

Abstract:  Objectives 

The aims of the study were to identify publicly available patient safety report databases and to determine whether these databases support safety analyst and data scientist use to identify patterns and trends.

Methods 

An Internet search was conducted to identify publicly available patient safety databases that contained patient safety reports. Each database was analyzed to identify features that enable patient safety analyst and data scientist use of these databases.

Results 

Seven databases (6 hosted by federal agencies, 1 hosted by a nonprofit organization) containing more than 28.3 million safety reports were identified. Some, but not all, databases contained features to support patient safety analyst use: 57.1% provided the ability to sort/compare/filter data, 42.9% provided data visualization, and 85.7% enabled free-text search. None of the databases provided regular updates or monitoring and only one database suggested solutions to patient safety reports. Analysis of features to support data scientist use showed that only 42.9% provided an application programing interface, most (85.7%) provided batch downloading, all provided documentation about the database, and 71.4% provided a data dictionary. All databases provided open access. Only 28.6% provided a data diagram.

Conclusions 

Patient safety databases should be improved to support patient safety analyst use by, at a minimum, allowing for data to be sorted/compared/filtered, providing data visualization, and enabling free-text search. Databases should also enable data scientist use by, at a minimum, providing an application programing interface, batch downloading, and a data dictionary.

Guest Post: Open Access and the Direction Moving Forward – The Scholarly Kitchen

“Perhaps in recognition of this state of play, the BOIA Steering Committee used their recent 20th Anniversary Recommendations to help clarify the goal of open, stating that “OA is not an end in itself, but a means to other ends, above all,” the document continues, “to the equity, quality, usability, and sustainability of research.” …

After six years of thinking about scholarly communication, I’ve come to think about what needs to happen to improve this system in ways that I believe are compatible with the high-level summary recommendations of BOAI20. What I believe is that:

 

no author should be asked to pay
no reader should be unable to access the record
the idea of “excellence” should be incompatible with exclusivity, artificial scarcity, or any other device not pertaining directly to the soundness of a scholarly activity
authors should be rewarded for behavior such as making usable data available whenever appropriate, for engaging with flourishing modes of experiential reporting or communication, or for exhibiting a history of collegial peer feedback….”

Opening Up to Open Science

“This way of sharing science has some benefits: peer review, for example, helps to ensure (even if it never guarantees) scientific integrity and prevent inadvertent misuse of data or code. But the status quo also comes with clear costs: it creates barriers (in the form of publication paywalls), slows the pace of innovation, and limits the impact of research. Fast science is increasingly necessary, and with good reason. Technology has not only improved the speed at which science is carried out, but many of the problems scientists study, from climate change to COVID-19, demand urgency. Whether modeling the behavior of wildfires or developing a vaccine, the need for scientists to work together and share knowledge has never been greater. In this environment, the rapid dissemination of knowledge is critical; closed, siloed knowledge slows progress to a degree society cannot afford. Imagine the consequences today if, as in the 2003 SARS disease outbreak, the task of sequencing genomes still took months and tools for labs to share the results openly online didn’t exist. Today’s challenges require scientists to adapt and better recognize, facilitate, and reward collaboration….

This tension between individual and institutional incentives and the progress of science must be recognized and resolved in a manner that contributes to solving the great challenges of today and the future. To change the culture, researchers must do more than take a pledge; they must change the game—the structures, the policies, and the criteria for success. In a word, open science must be institutionalized….

A powerful open science story can be found in the World Climate Research Programme’s Coupled Model Intercomparison Project (CMIP), established in 1995. Before CMIP, with the internet in its infancy, climate model results were scattered around the world and difficult to access and use. CMIP inspired 40 modeling groups and about 1,000 researchers to collaborate on advancing modeling techniques and setting guidelines for how and where to share results openly. That simple step led to an unexpected transformation: as more people were able to access the data, the community expanded, and more groups contributed data to CMIP. More people asking questions and pointing out issues in their results helped drive improvements. In its assessment reports, the Intergovernmental Panel on Climate Change relied on research publications using CMIP data to assess climate change. As a platform, CMIP enabled thousands of scientists to work together, self-correct their work, and create further ways to collaborate—a virtuous circle that attracted more scientists and more data, and increased the speed and usefulness of the work….

The most important message from these reports is that all parts of science, from individual researchers to universities and funding agencies, need to coordinate their efforts to ensure that early adopters aren’t jeopardizing their careers by joining the open science community. The whole enterprise has to change to truly realize the full benefits of open science. Creating this level of institutional adoption also requires updating policies, providing training, and recognizing and rewarding collaborative science….”