Challenges of scholarly communication: bibliometric transparency and impact

Abstract:  Citation metrics have value because they aim to make scientific assessment a level playing field, but urgent transparency-based adjustments are necessary to ensure that measurements yield the most accurate picture of impact and excellence. One problematic area is the handling of self-citations, which are either excluded or inappropriately accounted for when using bibliometric indicators for research evaluation. In this talk, in favour of openly tracking self-citations, I report on a study of self-referencing behaviour among various academic disciplines as captured by the curated bibliometric database Web of Science. Specifically, I examine the behaviour of thousands of authors grouped into 15 subject areas like Biology, Chemistry, Science and Technology, Engineering, and Physics. In this talk, I focus on the methodological set-up of the study and discuss data science related problems like author name disambiguation and bibliometric indicator modelling. This talk bases on the following publication: Kacem, A., Flatt, J. W., & Mayr, P. (2020). Tracking self-citations in academic publishing. Scientometrics, 123(2), 1157–1165. https://doi.org/10.1007/s11192-020-03413-9

 

A comparison of scientometric data and publication policies of ophthalmology journals

Abstract: Purpose: 

This retrospective database analysis study aims to present the scientometric data of journals publishing in the field of ophthalmology and to compare the scientometric data of ophthalmology journals according to the open access (OA) publishing policies.

Methods: 

The scientometric data of 48 journals were obtained from Clarivate Analytics InCites and Scimago Journal & Country Rank websites. Journal impact factor (JIF), Eigenfactor score (ES), scientific journal ranking (SJR), and Hirsch index (HI) were included. The OA publishing policies were separated into full OA with publishing fees, full OA without fees, and hybrid OA. The fees were stated as US dollars (USD).

Results: 

Four scientometric indexes had strong positive correlations; the highest correlation coefficients were observed between the SJR and JIF (R = 0.906) and the SJR and HI (R = 0.798). However, journals in the first quartile according to JIF were in the second and third quartiles according to the SJR and HI and in the fourth quartile in the ES. The OA articles published in hybrid journals received a median of 1.17-fold (0.15–2.71) more citations. Only HI was higher in hybrid OA; other scientometric indexes were similar with full OA journals. Full OA journals charged a median of 1525 USD lower than hybrid journals.

Conclusion: 

Full OA model in ophthalmology journals does not have a positive effect on the scientometric indexes. In hybrid OA journals, choosing to publish OA may increase citations, but it would be more accurate to evaluate this on a journal basis.

Becoming metrics literate: An analysis of brief videos that teach about the h-index | PLOS ONE

Abstract:  Introduction

Academia uses scholarly metrics, such as the h-index, to make hiring, promotion, and funding decisions. These high-stakes decisions require that those using scholarly metrics be able to recognize, interpret, critically assess and effectively and ethically use them. This study aimed to characterize educational videos about the h-index to understand available resources and provide recommendations for future educational initiatives.

Methods

The authors analyzed videos on the h-index posted to YouTube. Videos were identified by searching YouTube and were screened by two authors. To code the videos the authors created a coding sheet, which assessed content and presentation style with a focus on the videos’ educational quality based on Cognitive Load Theory. Two authors coded each video independently with discrepancies resolved by group consensus.

Results

Thirty-one videos met inclusion criteria. Twenty-one videos (68%) were screencasts and seven used a “talking head” approach. Twenty-six videos defined the h-index (83%) and provided examples of how to calculate and find it. The importance of the h-index in high-stakes decisions was raised in 14 (45%) videos. Sixteen videos (52%) described caveats about using the h-index, with potential disadvantages to early researchers the most prevalent (n = 7; 23%). All videos incorporated various educational approaches with potential impact on viewer cognitive load. A minority of videos (n = 10; 32%) displayed professional production quality.

Discussion

The videos featured content with potential to enhance viewers’ metrics literacies such that many defined the h-index and described its calculation, providing viewers with skills to recognize and interpret the metric. However, less than half described the h-index as an author quality indicator, which has been contested, and caveats about h-index use were inconsistently presented, suggesting room for improvement. While most videos integrated practices to facilitate balancing viewers’ cognitive load, few (32%) were of professional production quality. Some videos missed opportunities to adopt particular practices that could benefit learning.

Wiley Signs Declaration on Research Assessment, Deepens Commitment to Responsible Research Assessment | John Wiley & Sons, Inc.

“Global research and education leader Wiley today announced it has signed the Declaration on Research Assessment (DORA), which is a world-wide initiative designed to improve the ways in which the outputs of scholarly research are evaluated. 

As the publisher of nearly 2,000 academic journals, Wiley will deliver more ways to assess and recognize research outputs, which in turn supports healthy scholarship and allows more researchers to thrive in their careers. To this end, Wiley will roll out a broad range of journal and article metrics across its journal portfolio with the aim of providing a holistic, well-rounded view of the value and impact of any author’s research. This includes metrics that measure levels of impact beyond citation value, including usage, re-use, reproducibility, peer review assessment, geographic reach, and public recognition via references in media outlets….”

Revolutions in science: The proposal of an approach for the identification of most important researchers, institutions, and countries based on Reference Publication Year Spectroscopy (RPYS)

Abstract:  RPYS is a bibliometric method originally introduced in order to reveal the historical roots of research topics or fields. RPYS does not identify the most highly cited papers of the publication set being studied (as is usually done by bibliometric analyses in research evaluation), but instead it indicates most frequently referenced publications – each within a specific reference publication year. In this study, we propose to use the method to identify important researchers, institutions and countries in the context of breakthrough research. To demonstrate our approach, we focus on research on physical modeling of Earth’s climate and the prediction of global warming as an example. Klaus Hasselmann and Syukuro Manabe were both honored with the Nobel Prize in 2021 for their fundamental contributions to this research. Our results reveal that RPYS is able to identify most important researchers, institutions, and countries. For example, all the relevant authors’ institutions are located in the USA. These institutions are either research centers of two US National Research Administrations (NASA and NOAA) or universities: the University of Arizona, Princeton University, the Massachusetts Institute of Technology (MIT), and the University of Stony Brook.

 

Dismantling the ivory tower’s knowledge boundaries

“The major shift to open access during the pandemic began with the Free Read initiative, which launched the petition “

Unlock Coronavirus Research” for scientists in early February of 2020 and to which highly reputable medical publishers quickly responded. Before the pandemic, up to 75 percent of scholarly publications were behind a paywall. By comparison, a preliminary study of over 5,600 articles on PubMed suggests that more than 95 percent of scholarly articles related to COVID-19 are now freely available. This increase in accessibility resulted from the rapid adaptation by biomedical journals and publishers, including Elsevier, Springer Nature, Cell Press, New England Journal of Medicine, and The Lancet. These journals and publishers granted open access to research on COVID-19 research, often making it 

immediately accessible on the platform PubMed Central and similar public repositories. Free and open access to COVID-19 research quickly became the new normal for biomedicine, with available findings directly impacting the development of treatment protocols and vaccines. Yet the pandemic became more than a health crisis. Understanding the social, psychological, and economic implications of the pandemic were imperative to its continued management.

Social science research, which delivers insights into human behaviors, relationships, and institutions, was instrumental to policymaking and healthcare solution development during the COVID-19 pandemic. The importance of social science research to pandemic management was demonstrated by the 

shift in the topic of COVID-19 papers, from the initial focus on disease modeling, hospital mortality, diagnostics, and testing to an increasing focus on topics such as business closure, remote work, geographic mobility and migration, inequality, managerial decision-making, as well as accelerating innovation. Once the basic science on the virus were established, research on creating societal and economic resilience played an even larger role for beating the COVID-19 pandemic. One clear area that demonstrated the importance of social science research in informing COVID-19 management was the rollout of vaccines. Psychological, marketing, and information systems research played a central role in vaccine uptake across communities. A recent report by the National Institutes of Health called for the use of evidence-based strategies, such as 

behavioral nudges and strategic social norms, to increase vaccine uptake….”

 

 

Societal impact of university research in the written press: media attention in the context of SIUR and the open science agenda among social scientists in Flanders, Belgium | SpringerLink

Abstract:  Transferring scientific knowledge to non-academic audiences is an essential aspect of the open science agenda, which calls for scholars to pursue a popularization of their research. Accordingly, purposefully introducing scientific insights to the public at large is almost univocally deemed commendable. Indeed, in today’s models of research evaluation, the objects and activities considered are being extended beyond peer-reviewed journal articles to include non-scholarly popular communication. Although altmetrics offer one instrumental way to count some interactions with lay audiences, their reliance on social media makes them susceptible to manipulation, and mostly reflect circulation among niche audiences. In comparison, attention from non-scholarly media like newspapers and magazines seems a more relevant pathway to effectuate societal impact, due to its recognition in qualitative assessment tools and its broad, societal reach. Based on a case study of social scientists’ attention by newspapers and magazines in Flanders (northern Dutch-speaking region of Belgium) in 2019, this paper highlights that frequent participation in the public debate is reserved for high-status researchers only. Results show highly skewed media appearance patterns in both career position and gender, as eight male professors accounted for almost half of all 2019 media attention for social scientists. Because media attention is highly subject-dependent moreover, certain disciplines and fields offer easier pathways to popularization in media than others. Both the open science agenda and research assessment models value presence of researchers in popular media, adding written press attention to existing evaluation assessments however would disproportionately disadvantage early career researchers and exacerbate existing inequalities in academia.

 

 

At what point do academics forego citations for journal status? | Impact of Social Sciences

“The limitations of journal based citation metrics for assessing individual researchers are well known. However, the way in which these assessment systems differentially shape research practices within disciplines is less well understood. Presenting evidence from a new analysis of business and management academics, Rossella Salandra and Ammon Salter and James Walker¸ explore how journal status is valued by these academics and the point at which journal status becomes more prized than academic influence….”

At what point do academics forego citations for journal status? | Impact of Social Sciences

“The limitations of journal based citation metrics for assessing individual researchers are well known. However, the way in which these assessment systems differentially shape research practices within disciplines is less well understood. Presenting evidence from a new analysis of business and management academics, Rossella Salandra and Ammon Salter and James Walker¸ explore how journal status is valued by these academics and the point at which journal status becomes more prized than academic influence….”

What are the benefits of open access? TIB study confirms advantages and dispels reservations – Technische Informationsbibliothek (TIB)

“Open access – free access to scholarly publications – offers many advantages. As surveys show, however, some researchers still have reservations. In the past decade, numerous empirical studies have been published providing substantiated results on the hopes and concerns regarding open access….

To conduct this review, TIB identified a total of 318 scientific studies that empirically examine various effects of open access. From this corpus, the authors selected 61 particularly relevant studies for a systematic comparison; these were then analysed thoroughly and the various results were compared in detail.

The effects studied relate to seven major aspects of open access:

Attention in the scientific community
Quality of scientific publications
Knowledge transfer
Productivity of the publishing system
Use of publications
Inequality in the science system
Economic impact on the publishing system…

Dr. David Hopf, lead author of the study, reported the key findings: “The literature reviewed confirms several advantages of open access: open access leads to increased usage and to a professionally and geographically more diverse readership. At the same time, open access publications make a greater contribution to knowledge transfer than traditionally published research results, and the publishing process – the time between the submission and acceptance or publication of articles – is shorter. What is more, a number of negative concerns assumed in relation to the effects of open access – for example, that open access publications are of an inferior quality and lead to disadvantages in print edition sales – have been dispelled.”

However, one partial result came as a surprise: the fact that open access publications are cited more frequently than publications that are not freely available is often mentioned as an advantage of open access – and is also confirmed by most empirical studies. However, a substantial proportion of the empirical literature deviates from this result, which means that an OA citation advantage cannot be conclusively confirmed empirically. In light of a high level of plausibility and methodological difficulties in this area, however, it can still be assumed that such an advantage exists.

Just one finding indicates a negative effect of open access: where so-called article processing charges (APCs) – publication costs incurred by many open access publications – exist, authors with fewer resources may be discouraged from publishing open access, e.g. due to low income levels in some regions of the world or a lack of institutional funding. However, this is not an effect of open access per se, but rather an effect of a particular business model for financing open access publications….”

Hopf, D., Dellmann, S., Hauschke, C. and Tullney, M. (2022) Wirkungen von Open Access. Literaturstudie über empirische Arbeiten 2010-2021. Hannover : Technische Informationsbibliothek (TIB).

Hopf, D., Dellmann, S., Hauschke, C. and Tullney, M. (2022) Wirkungen von Open Access. Literaturstudie über empirische Arbeiten 2010-2021. Hannover : Technische Informationsbibliothek (TIB).

Open Access – the free access to scientific publications – intuitively offers many advantages. At the same time, some scientists, members of university administration, publishers, and policymakers continue to have reservations against open access. In the last decade, numerous empirical studies on the effects of open access were conducted. This report provides an overview of the state of research from 2010 to 2021. The empirical findings presented help determine the advantages and disadvantages of open access and serve as a knowledge base for scholars, publishers, academic institutions, and policy makers. An overview of the state of knowledge informs decisions on open access and publishing strategies. In addition, this report identifies aspects of open-access effects that are potentially highly relevant but have not yet been adequately studied. Overall, various advantages of open access can be considered empirically confirmed at the current state of research. These advantages include improved knowledge transfer, increased speed of the publication process, and increased usage by a more diverse readership, both in terms of profession and location. In addition, some presumed negative effects – such as lower quality of publications and disadvantages in the sale of print editions – can be considered empirically refuted. The empirical results on effects of open-access publishing therefore support the goal of a far-reaching transformation to open access, to which the German science organisations, among others, have committed themselves.

Correlation Between Altmetric Attention Scores and Citations for Articles Published in High–Impact Factor Ophthalmology Journals From 2018 to 2019 | Medical Journals and Publishing | JAMA Ophthalmology | JAMA Network

Importance  The Altmetric attention score (AAS) provides new information to gauge the impact of a research article not found through typical metrics, such as impact factor or citation counts. Objective  To explore the association between AAS and common impact markers among high-impact ophthalmology journals from 2018 to 2019. Design, Setting, and Participants  All articles published in the American Journal of Ophthalmology (AJO), JAMA Ophthalmology (JAMAO), and Ophthalmology (OPH) from January 1, 2018, to December 31, 2019, were collected for this cross-sectional study. Excluded articles were those missing Altmetric data at the time of data collection. The AAS and associated social media impact for each article were collected with the AAS calculator bookmarklet. Spearman rank correlation analyses and analysis of variance tests were conducted to assess differences in various metrics between AJO, JAMAO, and OPH. The study included articles published of all document types (article, conference paper, editorial, erratum, letter, note, retracted, review, and short survey) and access status (open access and not open access). Main Outcomes and Measures  The correlation between citation counts and Altmetric variables including AAS. Results  A total of 2467 articles were published in the study period. There were 351 articles excluded owing to missing Altmetric data. Of the 2116 articles included in the analysis, 1039 (49.1%) were published in 2018, and 1077 (50.9%) were published in 2019; the mean number of citations was 8.8 (95% CI, 7.9-9.6) for AJO, 6.2 (95% CI, 5.3-7.1) for JAMAO, and 15.1 (95% CI, 13.3-17.0) for OPH. The mean AAS was 4.5 (95% CI, 3.3-5.6) for AJO (723 publications), 27.4 (95% CI, 22.1-32.8) for JAMAO (758 publications), and 15.1 (95% CI, 10.9-19.3) for OPH (635 publications). Citation rate was moderately correlated with AAS across the 3 journals (AJO, ??=?0.39; P?<?.001; JAMAO, ??=?0.41; P?<?.001; OPH, ??=?0.40; P?<?.001), as well as minimally or moderately correlated with engagement or mention by Facebook posts (AJO, ??=?0.38; P?<?.001; JAMAO, ??=?0.24; P?<?.001; OPH, ??=?0.20; P?<?.001), news outlet reporting (AJO, ??=?0.12; P?<?.001; JAMAO, ??=?0.38; P?<?.001; OPH, ??=?0.19; P?<?.001), and Twitter posts (AJO, ??=?0.40; P?<?.001; JAMAO, ??=?0.38; P?<?.001; OPH, ??=?0.42; P?<?.001). Conclusions and Relevance  Results of this cross-sectional study suggest that citation rate has a moderate positive correlation with online and social media sharing of research in ophthalmology literature. Peer-reviewed journals may increase their reach and impact by sharing their literature through social media and online platforms.

Numbers Speak for Themselves, or Do They? On Performance Measurement and Its Implications – Berend van der Kolk, 2022

van der Kolk, B. (2022). Numbers Speak for Themselves, or Do They? On Performance Measurement and Its Implications. Business & Society. https://doi.org/10.1177/00076503211068433
 

Abstract; Performance measurement systems have the potential to improve organizational outcomes, but they often come at a cost. This commentary highlights the individual, organizational, and societal costs of performance measurement systems and explores how such costs could be reduced.

 

An open science argument against closed metrics

“In the Open Scientist Handbook, I argue that open science supports anti-rivalrous science collaborations where most metrics are of little, or of negative value. I would like to share some of these arguments here….

Institutional prestige is a profound drag on the potential for networked science. If your administration has a plan to “win” the college ratings game, this plan will only make doing science harder. It makes being a scientist less rewarding. Playing finite games of chasing arbitrary metrics or ‘prestige’ drags scientists away from the infinite play of actually doing science….

As Cameron Neylon said at the metrics breakout of the ‘Beyond the PDF’ conference some years ago, “reuse is THE metric.” Reuse reveals and confirms the advantage that open sharing has over current, market-based, practices. Reuse validates the work of the scientist who contributed to the research ecosystem. Reuse captures more of the inherent value of the original discovery and accelerates knowledge growth….”

Measures of Impact for Journals, Articles, and Authors | SpringerLink

“Journals and authors hope the work they do is important and influential. Over time, a number of measures have been developed to measure author and journal impact. These impact factor instruments are expanding and can be difficult to understand. The varying measures provide different perspectives and have varying strengths and weaknesses. A complete picture of impact for individual researchers and journals requires using multiple measures and does not fully capture all aspects of influence. There are only a few players in the scholarly publishing world that collect data on article citations: Clarivate Analytics, Elsevier, and Google Scholar (Table 1). Measures of influence for authors and journals based on article citations use one of these sources and may vary slightly because of differing journal coverage….”