Post-Publication Metrics Versus Pre-Publication Peer Review

Patterson, Mark (2009) PLoS Journals ? measuring impact where it matters writes:

“[R]eaders tend to navigate directly to the articles that are relevant to them, regardless of the journal they were published in… [T]here is a strong skew in the distribution of citations within a journal ? typically, around 80% of the citations accrue to 20% of the articles… [W]hy then do researchers and their paymasters remain wedded to assessing individual articles by using a metric (the impact factor) that attempts to measure the average citations to a whole journal?

“We?d argue that it?s primarily because there has been no strong alternative. But now alternatives are beginning to emerge… focusing on articles rather than journals… [and] not confining article-level metrics to a single indicator… Citations can be counted more broadly, along with web usage, blog and media coverage, social bookmarks, expert/community comments and ratings, and so on…

“[J]udgements about impact and relevance can be left almost entirely to the period after publication. By peer-reviewing submissions purely for scientific rigour, ethical conduct and proper reporting before publication, articles can be assessed and published rapidly. Once articles have joined the published literature, the impact and relevance of the article can then be determined on the basis of the activity of the research community as a whole… [through] [a]rticle-level metrics and indicators…”

Merits of Metrics. Of course direct article and author citation counts are infinitely preferable to — and more informative than — just a journal average (the journal “impact factor”).

And yes, multiple postpublication metrics will be a great help in navigating, evaluating and analyzing research influence, importance and impact.

But it is a great mistake to imagine that this implies that peer review can now be done on just a generic “pass/fail” basis.

Purpose of Peer Review. Not only is peer review dynamic and interactive — improving papers before approving them for publication — but the planet’s 25,000 peer-reviewed journals differ not only in the subject matter they cover, but also, within a given subject matter, they differ (often quite substantially) in their respective quality standards and criteria.

It is extremely unrealistic (and would be highly dysfunctional, if it were ever made to come true) to suppose that these 25,000 journals are (or ought to be) flattened to provide a 0/1 pass/fail decision on publishability at some generic adequacy level, common to all refereed research.

Pass/Fail Versus Letter-Grades. Nor is it just a matter of switching all journals from assigning a generic pass/fail grade to assigning its own letter grade (A-, B+, etc.), despite the fact that that is effectively what the current system of multiple, independent peer-reviewed journals provides. For not only do journal peer-review standards and criteria differ, but the expertise of their respective “peers” differs too. Better journals have better and more exacting referees, exercising more rigorous peer review. (So the 25,000 peer-reviewed journals today cannot be thought of as one generic peer-review filter that accepts papers for publication in each field with grades between A+ and E; rather there are A+ journals, B- journals, etc.: each established journal has its own independent standards, to which its submissions are answerable)

Track Records and Quality Standards. And users know all this, from the established track records of the journals they consult as readers and publish in as authors. Whether or not we like to put it that way, this all boils down to selectivity across a gaussian distribution of research quality in each field. There are highly selective journals, that accept only the very best papers — and even those often only after several rounds of rigorous refereeing, revision and re-refereeing. And there are less selective journals, that impose less exacting standards — all the way down to the fuzzy pass/fail threshold that distinguishes “refereed” journals from journals whose standards are so low that they are virtually vanity-press journals.

Supplement Versus Substitute. This difference (and independence) among journals in terms of their quality standards is essential if peer-review is to serve as the quality enhancer and filter that it is intended to be. Of course the system is imperfect, and, for just that reason alone (amongst many others) a rich diversity of post-publication metrics are an invaluable supplement to peer review. But they are certainly no substitute for pre-publication peer review, or, most importantly, its quality triage.

Quality Distribution. So much research is published daily in most fields that on the basis of a generic 0/1 quality threshold, researchers simply cannot decide rationally or reliably what new research is worth the time and investment to read, use and try to build upon. Researchers and their work differ in quality too, and they are entitled to know a priori, as they do now, whether or not a newly published work has made the highest quality cut, rather than merely that it has met some default standards, after which users must wait for the multiple post-publication metrics to accumulate across time in order to be able to have a more nuanced quality assessment.

Rejection Rates. More nuanced sorting of new research is precisely what peer review is about, and for, and especially at the highest quality levels. Although authors (knowing the quality track-records of their journals) mostly self-select, submitting their papers to journals whose standards are roughly commensurate with their quality, the underlying correlate of a journal’s refereeing quality standards is basically their relative rejection rate: What percentage of annual papers in their designated subject matter would meet their standards (if all were submitted to that journal, and the only constraint on acceptance were the quality level of the article, not how many articles the journal could manage to referee and publish per year)?

Quality Ranges. This independent standard-setting by journals effectively ranges the 25,000 titles along a rough letter-grade continuum within each field, and their “grades” are roughly known by authors and users, from the journals’ track-records for quality.

Quality Differential. Making peer review generic and entrusting the rest to post-publication metrics would wipe out that differential quality information for new research, and force researchers at all levels to risk pot-luck with newly published research (until and unless enough time has elapsed to sort out the rest of the quality variance with post-publication metrics). Among other things, this would effectively slow down instead of speeding up research progress.

Turn-Around Time. Of course pre-publication peer review takes time too; but if its result is that it pre-sorts the quality of new publications in terms of known, reliable letter-grade standards (the journals’ names and track-records), then it’s time well spent. Offloading that dynamic pre-filtering function onto post-publication metrics, no matter how rich and plural, would greatly handicap research usability and progress, and especially at its all-important highest quality levels.

More Value From Post-Publication Metrics Does Not Entail Less Value From Pre-Publication Peer Review. It would be ironic if today’s eminently valid and timely call for a wide and rich variety of post-publication metrics — in place of just the unitary journal average (the “journal impact factor”) — were coupled with an ill-considered call for collapsing the planet’s wide and rich variety of peer-reviewed journals and their respective independent, established quality levels onto some sort of global, generic pass/fail system.

Differential Quality Tags. There is an idea afoot that peer review is just some sort of generic pass/fail grade for “publishability,” and that the rest is a matter of post-publication evaluation. I think this is incorrect, and represents a misunderstanding of the actual function that peer review is currently performing. It is not a 0/1, publishable/unpublishable threshold. There are many different quality levels, and they get more exacting and selective in the higher quality journals (which also have higher-quality and more exacting referees and refereeing). Users need these differential quality tags when they are trying to decide whether newly published work is worth taking the time to ready and making the effort and risk to try to build upon (at the quality level of their own work).

User/Author/Referee Experience. I think both authors and users have a good idea of the quality levels of the journals in their fields — not from the journals’ impact factors, but from their content, and their track-records for content. As users, researchers read articles in their journals; as authors they write for those journals, and revise for their referees; and as referees they referee for them. They know that all journals are not equal, and that “peer-reviewed” can be done at a whole range of quality levels.

Metrics As Substitutes for User/Author/Referee Experience? Is there any substitute for this direct experience with journals (as users, authors and referees) in order to know what their peer-reviewing standards and quality level are? There is nothing yet, and no one can say yet whether there will ever be metrics as accurate as having read, written and refereed for the journals in question. Metrics might eventually provide an approximation, though we don’t yet know how close, and of course they only come after publication (well after).

Quality Lapses? Journal track records, user experiences, and peer review itself are certainly not infallible either, however; the usually-higher-quality journals may occasionally publish a lower-quality article, and vice versa. But on average, the quality of the current articles should correlate well with the quality of past articles. Whether judgements of quality from direct experience (as user/author/referee) will ever be matched or beaten by multiple metrics, I cannot say, but I am pretty sure they are not matched or beaten by the journal impact factor.

Regression on the Generic Mean? And even if multiple metrics do become as good a joint predictor of journal article quality as user experience, it does not follow that peer-review can then be reduced to generic pass/fail, with the rest sorted by metrics, because (1) metrics are journal-level, not article-level (though they can also be author-level) and, more important still, (2) if journal-differences are flattened to generic peer review, entrusting the rest to metrics, then the quality of articles themselves will fall, as rigorous peer review does not just assign articles a differential grade (via the journal’s name and track-record), but it improves them, through revision and re-refereeing. More generic 0/1 peer review, with less individual quality variation among journals, would just generate quality regression on the mean.

REFERENCES

Bollen J, Van de Sompel H, Hagberg A, Chute R (2009) A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE 4(6): e6022. doi:10.1371/journal.pone.0006022

Brody, T., Harnad, S. and Carr, L. (2006) . Journal of the American Association for Information Science and Technology (JASIST) 57(8) pp. 1060-1072.

Garfield, E., (1955) Citation Indexes for Science: A New Dimension in Documentation through Association of Ideas. Science 122: 108-111

Harnad, S. (1979) Creative disagreement. The Sciences 19: 18 – 20.

Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.

Harnad, S. (1984) Commentaries, opinions and the growth of scientific knowledge. American Psychologist 39: 1497 – 1498.

Harnad, Stevan (1985) Rational disagreement in peer review. Science, Technology and Human Values, 10 p.55-62.

Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry Psychological Science 1: 342 – 343 (reprinted in Current Contents 45: 9-13, November 11 1991).

Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A difficult balance: Peer review in biomedical publication.) Nature 322: 24 – 5.

Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT Press. Pp 103-118.

Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292.

Harnad, S. (1998/2000/2004) The invisible hand of peer review. Nature [online] (5 Nov. 1998), Exploit Interactive 5 (2000): and in Shatz, B. (2004) (ed.) Peer Review: A Critical Inquiry. Rowland & Littlefield. Pp. 235-242.

Harnad, S. (2008) Validating Research Performance Metrics Against Peer Rankings. Ethics in Science and Environmental Politics 8 (11) Special Issue: The Use And Misuse Of Bibliometric Indices In Evaluating Scholarly Performance

Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1)

Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, N., Eds. Open Access: Key Strategic, Technical and Economic Aspects. Chandos.

Call to Register Universities’ Open Access Mandates in ROARMAP

ROAR is the Registry of Open Access Repositories

ROARMAP is the Registry of Open Access Repository Material Archiving Policies

The purpose of ROARMAP is to register and record the openaccess policies of those institutions and funders who are putting the principle of Open Access (as expressed by the Budapest Open Access Initiative and the Berlin Declaration) into practice as recommended by Berlin 3 (as well as the UK Government Science and Technology Committee).

Universities, research institutions and research funders:

If you have adopted a mandate to provide open access to your own peer-reviewed research output you are invited to click here to register and describe your mandate in ROARMAP. (For suggestions about the form of policy to adopt, see here.)

Registering your OA mandate in ROARMAP will:

(1) record your own institution’s commitment to providing open access to its own research output,

(2) help the research community measure its progress (see Figure below, courtesy of Alma Swan and Oasis) in providing open access worldwide, and

(3) encourage further institutions to adopt openaccess mandates (so that your own institution’s users can have access to the research output of other institutions as well).

Sample Institutional Self-Archiving Mandate

“For the purposes of institutional record-keeping, research asset management, and performance-evaluation, and in order to maximize the visibility, accessibility, usage and impact of our institution’s research output, our institution’s researchers are henceforth to deposit the final, peer-reviewed, accepted drafts of all their journal articles (and accepted theses) into our institution’s institutional repository immediately upon acceptance for publication.”

To register and describe your mandate, please click here.


Arabic (please return to English version to sign) [Many thanks to Chawki Hajjem for the translation]
Chinese (please return to English version to sign) [Many thanks to Chu Jingli for the translation]
French (s.v.p. revenir sur cette version anglaise pour signer) [Beaucoup de remerciements a H. Bosc.]
German (Unterzeichnung bitte in der englischen Version) [Vielen Dank an K. Mruck.]
Hebrew (please return to English version to sign) [Many thanks to Miriam Faber and Malka Cymblista for the translation]
Italian (please return to English version to sign) [Many thanks to Susanna Mornati for the translation]
Japanese(please return to English version to sign) [Many thanks to Koichi Ojiro for the translation]
Russian (pozsaluista vozvratite k angliskomu variantu k dannym po zalemi) [Spasibo bolshoia Eleni Kulaginoi dla perevoda]
Spanish (ver tambien) (por favor volver a la version inglesa para firmar) [Muchas gracias a Hector F. Rucinque para la traduccion espanola]

Hungary’s First Green OA Mandate; Planet’s 93rd: OTKA

HUNGARY funder-mandate

Hungarian Scientific Research Fund (OTKA)

Institution’s/Department’s OA Eprint Archives: [growth data]

Institution’s/Department’s OA Self-Archiving Policy:

All scientific publications resulting from support by an OTKA grant are required to be made available for free according to the standards of Open Access, either through providing the right of free access during publication or through depositing the publication in an open access repository. The deposit can be in any institutional or disciplinary repository, as well as in the Repository of the Library of the Hungarian Academy of Sciences ? REAL: http://real.mtak.hu/

Added by: Dr. Elod Nemerkenyi (Assistant for International Affairs) nemerkenyi.elod AT otka.hu on 22 Jul 2009

UK’s 24th Green Open Access Mandate, Planet’s 92nd: Coventry University

OA Self-Archiving Policy: Coventry University: Department of Media and Communication

Full list of institutions

Coventry University: Department of Media and Communication
(UK
departmental-mandate)
http://www.coventry.ac.uk/cu/schoolofartanddesign/mediaandcommunication

Institution’s/Department’s OA Eprint Archives
[growth data] https://curve.coventry.ac.uk/cu/logon.do

CURVE, which stands for Coventry University Repository Virtual Environment, is Coventry University’s existing institutional repository.

Institution’s/Department’s OA Self-Archiving Policy
Open Access Self-Archiving Policy: Coventry University, Department of Media and Communication

This Open Access Self-Archiving Policy requires all researchers in the Department of Media and Communication at Coventry University to deposit copies of their research outputs in CURVE (which stands for Coventry University Repository Virtual Environment), in order to make these outputs freely accessible and easily discoverable online, and so increase the visibility, dissemination, usage and impact of the Department?s research. This Open Access Self-Archiving Policy will be mandatory from 1st September 2009 onwards.

The Department?s Open Access Self-Archiving Policy makes it obligatory for each researcher in the Department to supply an electronic copy of the author?s final version of all peer-reviewed research outputs for deposit in CURVE immediately upon their acceptance for publication.

The policy also endorses the depositing in CURVE of an electronic copy of the author?s final accepted version of all non-peer reviewed research outputs, especially those that are likely to contribute to any future REF, as well as of research outputs published before the introduction of this policy.

Researchers will make these research outputs available for deposit in CURVE together with the relevant bibliographic metadata (name of author, title of publication, date and place of publication and so on). When doing so they will indicate whether a particular research output can be made publicly visible.

Where it is possible to do so all researchers in the Department of Media and Communication at Coventry University are required to designate outputs deposited in CURVE as being publicly available Open Access. This will enable the full text of the output and the associated metadata to be easily found, accessed, indexed and searched across a range of global search engines, archives and databases.

In those instances where it is not possible to do so ? because it is necessary to comply with the legal requirements of a publisher?s or funder?s copyright policy or licensing agreement, for example – researchers can define outputs deposited in CURVE as being ?closed access? and for use only within Coventry University as an aid to the administration, management and reporting of research activity. In such cases only the metadata of the research output will be visible publicly, with Open Access to the full text being delayed for that period specified by the publisher or funder, often in the form of an eighteen, twelve or (preferably, at most) six month embargo. The full text of the output can then be made publicly available under Open Access conditions at a later date, immediately the period of the embargo has come to an end or permission to do so has otherwise been granted.

To keep such cases to an absolute minimum, from 1st September 2009 researchers in the Department of Media and Communication at Coventry University are expected, as much as is possible and appropriate, to avoid signing copyright or licensing agreements that do not allow electronic copies of the author?s final, peer reviewed and accepted version of their research outputs to be deposited in an institutional Open Access repository such as CURVE.

17 July 2009

Science Magazine: Letters About the Evans & Reimer Open Access Study

Five months after the fact, this week’s Science Magazine has just published four letters and a response about Evans & Reimer’s Open Access and Global Participation in Science, Science 20 February 2009: 1025.

You might want to also take a peek at these three rather more detailed critiques that Science did not publish…:

Open Access Benefits for the Developed and Developing World: The Harvards and the Have-Nots

The Evans & Reimer OA Impact Study: A Welter of Misunderstandings

Perils of Press-Release Journalism: NSF, U. Chicago, and Chronicle of Higher Education

Stevan Harnad
American Scientist Open Access Forum

OA in High Energy Physics Arxiv Yields Five-Fold Citation Advantage

Gentil-Beccot, Anne; Salvatore Mele, Travis Brooks (2009) Citing and Reading Behaviours in High-Energy Physics: How a Community Stopped Worrying about Journals and Learned to Love Repositories


This is an important study, and most of its conclusions are valid:

(1) Making research papers open access (OA) dramatically increases their impact.

(2) The earlier that papers are made OA, the greater their impact.

(3) High Energy Physics (HEP) researchers were among the first to make their papers OA (since 1991, and they did it without needing to be mandated to do it!)

(4) Gold OA provides no further impact advantage over and above Green OA.

However, the following caveats need to be borne in mind, in interpreting this paper:

(a) HEP researchers have indeed been providing OA since 1991, unmandated (and computer scientists have been doing so since even earlier). But in the ensuing years, the only other discipline that has followed suit, unmandated, has been economics, despite the repeated demonstration of the Green OA impact advantage across all disciplines. So whereas still further evidence (as in this paper by Gentil-Beccot et al) confirming that OA increases impact is always very welcome, that evidence will not be sufficient to induce enough researchers to provide OA; only mandates from their institutions and funders can ensure that they do so.

(b) From the fact that when there is a Green OA version available, users prefer to consult that Green OA version rather than the journal version, it definitely does not follow that journals are no longer necessary. Journals are (and always were) essentially peer-review service-providers and cerifiers, and they still are. That essential function is indispensable. HEP researchers continue to submit their papers to peer-reviewed journals, as they always did; and they deposit both their unrefereed preprints and then their refereed postprints in arxiv (along with the journal reference). None of that has changed one bit.

(c) Although it has not been systematically demonstrated, it is likely that in fields like HEP and astrophysics, the journal affordability/accessibility problem is not as great as in many other fields. OA’s most important function is to provide immediate access to those who cannot afford access to the journal version. Hence the Early Access impact advantage in HEP — arising from making preprints OA well before the published version is available — translates, in the case of most other fields, into the OA impact advantage itself, because without OA many potential users simply do not have access even after publication, hence cannot make any contribution to the article’s impact.

(d) Almost no one has ever argued (let alone adduced evidence) that Gold OA provides a greater OA advantage than Green OA. The OA advantage is the OA advantage, whether Green or Gold. (It just happens to be easier and more rigorous to test and demonstrate the OA advantage through within-journal comparisons [i.e Green vs. non-Green articles] than between-journal comparisons [Gold vs. non-Gold journals].)

Stevan Harnad


EXCERPTS: from Gentil-Beccot et al:

ABSTRACT: Contemporary scholarly discourse follows many alternative routes in addition to the three-century old tradition of publication in peer-reviewed journals. The field of High- Energy Physics (HEP) has explored alternative communication strategies for decades, initially via the mass mailing of paper copies of preliminary manuscripts, then via the inception of the first online repositories and digital libraries.

This field is uniquely placed to answer recurrent questions raised by the current trends in scholarly communication: is there an advantage for scientists to make their work available through repositories, often in preliminary form? Is there an advantage to publishing in Open Access journals? Do scientists still read journals or do they use digital repositories?

The analysis of citation data demonstrates that free and immediate online dissemination of preprints creates an immense citation advantage in HEP, whereas publication in Open Access journals presents no discernible advantage. In addition, the analysis of clickstreams in the leading digital library of the field shows that HEP scientists seldom read journals, preferring preprints instead….

…arXiv was first based on e-mail and then on the web, becoming the first repository and the first ?green? Open Access5 platform… With the term ?green? Open Access we denote the free online availability of scholarly publications in a repository. In the case of HEP, the submission to these repositories, typically arXiv, is not mandated by universities or funding agencies, but is a free choice of authors seeking peer recognition and visibility… The results of an analysis of SPIRES data on the citation behaviour of HEP scientists is presented… demonstrat[e] the ?green? Open Access advantage in HEP… With the term ?gold? Open Access we denote the free online availability of a scholarly publication on the web site of a scientific journals…. There is no discernable citation advantage added by publishing articles in ?gold? Open Access journals…

Figure (Gentil-Beccot et al. 2009): Cumulative citation count as a function of the age of the paper relative to its publication date. 4839 articles from 5 major HEP journals published in 2005 are considered.

7. Conclusions

Scholarly communication is at a cross road of new technologies and publishing models. The analysis of almost two decades of use of preprints and repositories in the HEP community provides unique evidence to inform the Open Access debate, through four main findings:

1. Submission of articles to an Open Access subject repository, arXiv, yields a citation advantage of a factor five.

2. The citation advantage of articles appearing in a repository is connected to their dissemination prior to publication, 20% of citations of HEP articles over a two-year period occur before publication.

3. There is no discernable citation advantage added by publishing articles in ?gold? Open Access journals.

4. HEP scientists are between four and eight times more likely to download an article in its preprint form from arXiv rather than its final published version on a journal web site.

Taken together these findings lead to three general conclusions about scholarly communication in HEP, as a discipline that has long embraced green Open Access:

1. There is an immense advantage for individual authors, and for the discipline as a whole, in free and immediate circulation of ideas, resulting in a faster scientific discourse.

2. The advantages of Open Access in HEP come without mandates and without debates. Universal adoption of Open Access follows from the immediate benefits for authors.

3. Peer-reviewed journals have lost their role as a means of scientific discourse, which has effectively moved to the discipline repository.

HEP has charted the way for a possible future in scholarly communication to the full benefit of scientists, away from over three centuries of tradition centred on scientific journals. However, HEP peer-reviewed journals play an indispensable role, providing independent accreditation, which is necessary in this field as in the entire, global, academic community. The next challenge for scholarly communication in HEP, and for other disciplines embracing Open Access, will be to address this novel conundrum. Efforts in this direction have already started, with initiatives such as SCOAP3…