Should altmetrics take a step back and reconsider what the main purpose / research question is? I should suggest that what we need is an alternative to the current power of the impact factor in assessing the work of scholars. This may or may not involve metrics of any kind. My suggestion for starters is that we need a system that is not as reliant on metrics of any kind.
Having said that, some metrics studies that might actually be useful:
– does an emphasis on quantity of publication increase duplication of content and/or reduce quality? With respect to the latter, this is what I have heard from senior experts in scholarly publishing and I think both Brown and Harley touch on this in their reports – at least with respect to books, pushing scholars to publish two books rather than one to get tenure means pressure to publish in less time than it takes to write a good book. So pushing for quantity seems likely to correlate with reduced quality (a hypothesis worth testing?)
One advantage to studying the disadvantages of pushing for quantity is that if the hypothesis (quantity correlates negatively with quality) is correct, then that is evidence that can reduce the workload of scholars – something I expect that scholars are likely to support
Other possibilities:
– scholars might want to know about journals:
– average and range of time from submission to decision
– level of “peer” doing the peer review (grad student? senior professor?)
– extent and quality of contents (this has to be qualitative analysis; sampling makes sense)
Shifting from a print-based scholarly communication system to an open access knowledge commons, while retaining or increasing quality and reducing costs, is possible – but it’s not easy. It is worth taking the time to think things through and get at least some stuff right.
