[2006.14830] Metrics and peer review agreement at the institutional level

Abstract:  In the past decades, many countries have started to fund academic institutions based on the evaluation of their scientific performance. In this context, post-publication peer review is often used to assess scientific performance. Bibliometric indicators have been suggested as an alternative to peer review. A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review based on a sample of publications submitted for evaluation to the national Italian research assessment exercise (2011–2014). In particular, we study the agreement between bibliometric indicators and peer review at a higher aggregation level, namely the institutional level. Additionally, we also quantify the internal agreement of peer review at the institutional level. We base our analysis on a hierarchical Bayesian model using cross-validation. We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two reviewers for certain fields of science in this particular context. This suggests that for some fields, bibliometric indicators may possibly be considered as an alternative to peer review for the Italian national research assessment exercise. Although results do not necessarily generalise to other contexts, it does raise the question whether similar findings would obtain for other research assessment exercises, such as in the United Kingdom.


Twenty years of Wikipedia in scholarly publications: a bibliometric network analysis of the thematic and citation landscape | SpringerLink

Abstract:  Wikipedia has grown to be the biggest online encyclopedia in terms of comprehensiveness, reach and coverage. However, although different websites and social network platforms have received considerable academic attention, Wikipedia has largely gone unnoticed. In this study, we fill this research gap by investigating how Wikipedia is used in scholarly publications since its launch in 2001. More specifically, we review and analyze the intellectual structure of Wikipedia’s scholarly publications based on 3790 Web of Science core collection documents written by 10,636 authors from 100 countries over two decades (2001–2021). Results show that the most influential outlets publishing Wikipedia research include journals such as Plos one, Nucleic Acids Research, the Journal of the Association for Information Science and Technology, the Journal of the American Society for Information Science and Technology, IEEE Access, and Information Processing and Management. Results also show that the author collaboration network is very sparsely connected, indicating the absence of close collaboration among the authors in the field. Furthermore, results reveal that the Wikipedia research institutions’ collaboration network reflects a North–South divide as very limited cooperation occurs between developed and developing countries’ institutions. Finally, the multiple correspondence analysis applied to obtain the Wikipedia research conceptual map reveals the breadth, diversity, and intellectual thrust of the Wikipedia’s scholarly publications. Our analysis has far-reaching implications for aspiring researchers interested in Wikipedia research as we retrospectively trace the evolution in research output over the last two decades, establish linkages between the authors and articles, and reveal trending topics/hotspots within the broad theme of Wikipedia research.


Escaping ‘bibliometric coloniality’, ‘epistemic inequality’

“Africa’s scholarly journals compete on an unequal playing field because of a lack of funding and the struggle to sustain academic credibility.

“These inequalities are exacerbated by the growing influence of the major citation indexes, leading to what we have called bibliometric coloniality,” say the authors of the book, Who Counts? Ghanaian academic publishing and global science, published by African Minds at the start of 2023.

“The rules of the game continue to be defined outside the continent. We hope that, in some small way, this book contributes to the renaissance and renewal of African-centred research and publishing infrastructures,” the authors say….”

“Superior identification index – Quantifying the capability of academic journals to recognize good research

Abstract:  In this paper we present “superior identification index” (SII), a metric to quantify the capability of academic journals to recognize top papers restricted by specific time window and study field. Intuitively, SII is the percentage of papers from a journal in the top p% papers in the field. SII provides flexible framework to make trade-offs on journal quality and quantity, as p rises it puts more weight on quantity and less weight on quality. Concerns on the p selection are discussed, and extended metrics of SII, including superior identification efficiency (SIE) and paper rank percentile (PRP), were proposed to sketch other dimensions of journal performance. Based on bibliometric data from ecological field, we find that as p increases, the correlation between SIE and JIF first rises then drops, indicating that JIF might most likely reflect “how well a journal identifies the top 26~34% papers in the field”. Hopefully, the new proposed SII metric and its extensions could promote the quality awareness and provide flexible tools for research evaluation.

Promoting Open Science through bibliometrics | LIBER Quarterly: The Journal of the Association of European Research Libraries

Abstract:  In order to assess the progress of Open Science in France, the French Ministry of Higher Education, Research and Innovation published the French Open Science Monitor in 2019. Even if this tool has a bias, for only the publications with a DOI can be considered, thus promoting article-dominant research communities, its indicators are trustworthy and reliable. The University of Lorraine was the very first institution to reuse the National Monitor in order to create a new version at the scale of one university in 2020. Since its release, the Lorraine Open Science Monitor has been reused by many other institutions. In 2022, the French Open Science Monitor further evolved, enabling new insights on open science. The Lorraine Open Science Monitor has also evolved since it began. This paper details how the initial code for the Lorraine Open Science Monitor was developed and disseminated. It then outlines plans for development in the next few years.


Challenges of scholarly communication: bibliometric transparency and impact

Abstract:  Citation metrics have value because they aim to make scientific assessment a level playing field, but urgent transparency-based adjustments are necessary to ensure that measurements yield the most accurate picture of impact and excellence. One problematic area is the handling of self-citations, which are either excluded or inappropriately accounted for when using bibliometric indicators for research evaluation. In this talk, in favour of openly tracking self-citations, I report on a study of self-referencing behaviour among various academic disciplines as captured by the curated bibliometric database Web of Science. Specifically, I examine the behaviour of thousands of authors grouped into 15 subject areas like Biology, Chemistry, Science and Technology, Engineering, and Physics. In this talk, I focus on the methodological set-up of the study and discuss data science related problems like author name disambiguation and bibliometric indicator modelling. This talk bases on the following publication: Kacem, A., Flatt, J. W., & Mayr, P. (2020). Tracking self-citations in academic publishing. Scientometrics, 123(2), 1157–1165. https://doi.org/10.1007/s11192-020-03413-9


Revolutions in science: The proposal of an approach for the identification of most important researchers, institutions, and countries based on Reference Publication Year Spectroscopy (RPYS)

Abstract:  RPYS is a bibliometric method originally introduced in order to reveal the historical roots of research topics or fields. RPYS does not identify the most highly cited papers of the publication set being studied (as is usually done by bibliometric analyses in research evaluation), but instead it indicates most frequently referenced publications – each within a specific reference publication year. In this study, we propose to use the method to identify important researchers, institutions and countries in the context of breakthrough research. To demonstrate our approach, we focus on research on physical modeling of Earth’s climate and the prediction of global warming as an example. Klaus Hasselmann and Syukuro Manabe were both honored with the Nobel Prize in 2021 for their fundamental contributions to this research. Our results reveal that RPYS is able to identify most important researchers, institutions, and countries. For example, all the relevant authors’ institutions are located in the USA. These institutions are either research centers of two US National Research Administrations (NASA and NOAA) or universities: the University of Arizona, Princeton University, the Massachusetts Institute of Technology (MIT), and the University of Stony Brook.


Handbook on Research Assessment in the Social Sciences (2022) – Emanuel Kulczycki

This Handbook provides a comprehensive overview of current developments, issues and good practices regarding assessment in social science research. It pays particular attention to the challenges in evaluation policies in the social sciences, as well as to the specificities of publishing in the area. The Handbook discusses the current societal challenges facing researchers, from digital societies, to climate change and sustainability, to trust in democratic societies. Chapters provide ways to strengthen research assessment in the social sciences for the better, by offering a diverse range of experiences and views of experts from all continents. The Handbook also outlines major data sources that can be used to assess social sciences research, as well as looking at key dimensions of research quality in the social sciences including journal peer review, the issue of identifying research quality, and gender disparities in social science research. This book will be an essential read for scholars interested in research assessment in the social sciences. It will also be useful to policy makers looking to understand the key position of the social sciences in science and society and provide appropriate frameworks for key societal challenges.

Laakso & Multas (2022, preprint) European scholarly journals from small- and mid-size publishers in times of Open Access: Mapping journals and public funding mechanisms | Zenodo

Mikael Laakso, & Anna-Maija Multas. (2022). European scholarly journals from small- and mid-size publishers in times of Open Access: Mapping journals and public funding mechanisms (Version 1). Zenodo. https://doi.org/10.5281/zenodo.5909512


Open Access (OA) publishing omits reader-side fees, which requires that resources to sustain journals can not originate from the reader-side. Large international publishers have had capability to monetize OA publishing from national consortia and institutions, but small- and mid-sized publishers have not succeeded to the same degree. There is currently a lack of information concerning to what degree small- and mid-sized publishers are present in European countries, to what degree their journals are already OA, and how the countries are supporting these journals financially or technically to publish their materials OA. The methods for this study include bibliometric analysis, document analysis of web information, inquiries to OA experts in European countries, and web-survey to a small sample of journals in each country. The study found that there are 16387 journals from small-and mid-sized publishers being published in European countries (incl. transcontinental states) of which 36% are already publishing OA. The vast majority of journals published in Europe are by single-publisher journals (77% of all publishers publish only one journal). Journals from smalland mid-sized publishers were found to be multilingual or non-English to a higher degree than journals from large publishers (44% and 43% vs 6% and 5%). According to our observations there is large diversity in how (and if) countries reserve and distribute funds to journals active in the countries, ranging from continuous inclusive subsidies to competitive grant funding or nothing at all. Funding information was often difficult to discover and efforts to make such information more easily available would likely facilitate policy development in this area. We call out for additions and corrections to journal funding instrument information in order to make the data as comprehensive and accurate as possible.

Mapping Science: Tools for Bibliometric and Altmetric Studies

Introduction. The study investigates whether online attention, carried out on social media or by video tutorials, affects the popularity of these tools in the research community.
Method. We collected data from the Web of Science, Scopus, YouTube, Facebook, Twitter, and Instagram, using web-scraping tools. Bibliometrics, altmetrics and webometrics were applied to process the data and to analyse Gephi, Sci2 Tool, VOSviewer, Pajek, CiteSpace and HistCite.
Analysis. Statistical and network analyses, and YouTube analytics, were used. The tools’ interfaces were assessed in the preliminary stage of the comparison. The results were plotted on charts and graphs, and compared.
Results. Social media and video tutorials had minimal influence on the popularity of different tools, as reflected by the number of papers within the Web of Science and Scopus where they featured. However, the small but constant growth of publications mentioning Gephi could be a result of Twitter promotion and a high number of video tutorials. The authors proposed four directions for further comparisons of science mapping software.
Conclusions. This work shows that biblio- and scientometricians are not influenced by social media visibility or accessibility of video tutorials. Future research on this topic could focus on evaluating the tools, their features and usability, or the availability of workshops.

Surveillance Publishing

Abstract:  This essay develops the idea of surveillance publishing, with special attention to the example of Elsevier. A scholarly publisher can be defined as a surveillance publisher if it derives a substantial proportion of its revenue from prediction products, fueled by data extracted from researcher behavior. The essay begins by tracing the Google search engine’s roots in bibliometrics, alongside a history of the citation analysis company that became, in 2016, Clarivate. The point is to show the co-evolution of scholarly communication and the surveillance advertising economy. The essay then refines the idea of surveillance publishing by engaging with the work of Shoshana Zuboff, Jathan Sadowski, Mariano-Florentino Cuéllar, and Aziz Huq. The recent history of Elsevier is traced to describe the company’s research-lifecycle data-harvesting strategy, with the aim to develop and sell prediction products to universities and other customers. The essay concludes by considering some of the potential costs of surveillance publishing, as other big commercial publishers increasingly enter the predictive-analytics mark. It is likely, I argue, that windfall subscription-and-APC profits in Elsevier’s “legacy” publishing business have financed its decade-long acquisition binge in analytics, with the implication that university customers are budgetary victims twice over. The products’ purpose, I stress, is to streamline the top-down assessment and evaluation practices that have taken hold in recent decades, in tandem with the view that the university’s main purpose is to grow regional and national economies. A final pair of concerns is that publishers’ prediction projects may camouflage and perpetuate existing biases in the system—and that scholars may internalize an analytics mindset, one already encouraged by citation counts and impact factors.

The OAPEN Library and the origin of downloads – libraries & academic institutions – OAPEN – supporting the transition to open access for academic books

On a regular basis, we look at the download data of the OAPEN Library and where it comes from. While examining the data from January to August 2021, we focused on the usage originating from libraries and academic institutions. Happily, we found that more than 1,100 academic institutions and libraries have used the OAPEN Library.

Of course, we do not actively track individual users. Instead we use a more general approach: we look at the website from which the download from the OAPEN Library originated. How does that work? For instance, when someone in the library of the University of Leipzig clicks on the download link of a book in the OAPEN library, two things happen: first, the book is directly available on the computer that person is working on, and second, the OAPEN server notes the ‘return address’: https://katalog.ub.uni-leipzig.de/. We have no way of knowing who the person is that started the download, we just know the request originated from the Leipzig University Library. Furthermore, some organisations choose to suppress sending their ‘return address’, making them anonymous.

What is helpful to us, is the fact that aggregators such as ExLibris, EBSCO or SerialSolutions use a specific return address. Examples are “west-sydney-primo.hosted.exlibrisgroup.com” – pointing to the library of the Western City University – or “sfx.unibo.it”– coming from the library of the Università di Bologna. And in this way, many academic libraries can also be identified from their web address. Some academic institutions only display their ‘general’ address.


Author productivity pattern and applicability of Lotka’s inverse square law: a bibliometric appraisal of selected LIS open access journals | Emerald Insight

Abstract:  Purpose

The purpose of this paper is to know whether the authors’ productivity pattern of library and information science (LIS) open access journals adheres to Lotka’s inverse square law of scientific productivity. Since the law was introduced, it has been tested in various fields of knowledge, and results have varied. This study has closely followed Lotka’s inverse square law in the field of LIS open access journals to find a factual result and set a baseline for future studies on author productivity of LIS open access journals.


The publication data of selected ten LIS open access journals pertain to authorship, citations were downloaded from the Scopus database and analysed using bibliometric indicators like authorship pattern, collaborative index (CI), degree of collaboration (DC), collaborative coefficient (CC) and citation counts. This study has applied Lotka’s inverse square law to assess authors’ productivity pattern of LIS open access journals and further Kolmogorov-Smirnov (K-S) goodness-of-fit test applied for testing of observed and expected author productivity data.


Inferences were drawn for the set objectives on authorship pattern, collaboration trend and authors’ productivity pattern of LIS open access journals covered in this study. The single authorship pattern is dominant in LIS open access journals covered in this study. The CI, DC and CC are found to be 1.95, 0.47 and 0.29, respectively. The expected values as per Lotka’s law (n = ?2) significantly vary from the observed values as per the chi-square test and K-S goodness-of-fit test. Hence, this study does not adhere to Lotka’s inverse square law of scientific productivity.

Practical implications

Researchers may find an idea about the authors’ productivity patterns of LIS open access journals. This study has used the K-S goodness-of-fit test and the chi-square test to validate the authors’ productivity data. The inferences found out from this study will be a baseline for future research on author productivity of LIS open access journals.


This study is significant from the viewpoint of the growing research on open access journals in the field of LIS and to identify the authorship pattern, collaboration trend and author productivity pattern of such journals.