One of the main sessions was on ‘Scientometrics, bibliometrics and quantitative evaluation of research: evaluating articles, people and institutes’. Anthony van Raan, Director of the Centre for Science and Technology Studies at Leiden University, gave a fact-packed presentation about his work on measuring citation patterns and impact, on the bases of articles, individual scientists and research groups and institutions. Different institutions can be assessed in a comparative way, for example, for the impact their astronomy programmes enjoy.

The tools that Anthony’s group have developed over the past two decades are extremely powerful. Until recently, the base data he and his colleagues have used came from Thomson Reuters (the ISI databases) but they are now also including data from Elsevier’s Scopus service. Anthony’s conclusions were that bibliometric analysis is a very useful, informative and penetrating methodology for assessing research effort, but that it should never be used in isolation, only in conjunction with other assessment regimens, particularly peer review.

He was followed on the programme by Mary Van Allen, Manager of the Research Services Group at Thomson Reuters (aka ISI). Mary’s talk title was ‘Beyond Impact Factors’, a topic that has long been awaited! She ran through the metrics produced by her organisation, familiar to frequent users of the Thomson Web of Science. Mary demonstrated some new functionality of the Web of Science service, including some neat displays – such as being able to track and display a paper’s ‘children’ and ‘grandchildren’, all tracked via citation analysis. A new website called Researcher ID allows researchers to create an authority file of their own papers and get a real-time display of their citations, h-index and so forth. The big idea here is to give the collaboration network diagram, displaying by geographic region or down to individual institution. Clearly, Web of Science is working hard on developing new metrics from its databases. A question from Matt Cockerill as to whether Web of Science is intending to give citations from different journals a different ‘importance’ was answered in the negative, partly because weighting is a subjective issue and partly because, as Anthony pointed out in the ensuing discussion, there is not a consistency to apply – many papers published in Nature and Science are never cited (apart from self-citation).

Richard Gedye, Chair of COUNTER and the UKSG Working Group on Usage Factors and Research Director in the journals division at Oxford University Press. Richard spoke on measuring usage of articles – or rather, of journals, since that is the base point that COUNTER has employed. He described the research programme that has been carried out by the UKSG on usage, including qualitative surveying of librarians and authors and large-scale online surveying of the same constituencies. The more metrics that can be brought into play were advisable. Richard reported that both UKSG’s survey and the previous one by the CIBER Group showed that 70% of authors are enthusiastic about a usage-based measure for assessing research journals. Librarians has a slightly different perspective, of course, but still exhibited a high degree of support for the notion, ranking the putative usage factor below only ‘feedback from library users’ as away of assessing journals. Richard also reported that plans are underway for a study to outline the metrics currently being assessed, whether any of them are suitable and how publishers can establish a consistency over how they report usage. There is a paper on this work here. There was some discussion over the significance of download numbers

Then we heard from Howard Browman, Principal Research Scientist at the Institute of Marine Research, Storebo in Norway, speaking on the use and misuse of bibliometric indices in evaluating scholarly performance. Howard gave an overview of existing metrics, of 21 ‘problems’ with the Journal Impact Factor, and emphasised that ALL bibliometric indices have such limitations. Moreover, the practical application (i.e. assessment of an individual for promotion or tenure) of these indices should only be done by people with a thorough understanding of their limitations and never by uninformed panels of assessors. Howard showed data that confirm that the Pareto Principle holds for any individual scientist’s citations (i.e. a minority of articles get the majority of the scientist’s citations) and this also holds when whole journals are studied. He also showed that almost 50% of articles in the Web of Science database have never been cited at all. Journals with high JIFs have a high degree of editorial pre-screening (editors screen before manuscripts are sent out for review) and a relatively low acceptance rate. Howard’s questions: are we saying that 80% of articles published are of low quality? Are 80% of journals of little significance? Or is there something there that is not captured by citations? We are accustomed to focusing on the ‘quality’ (i.e. highly-cited) end of the published corpus, but what about the rest? Authors, according to the CIBER study, tend to agree that too much emphasis is given to impact measures based on citations, and other commentators too are recommending a more balanced approach to assessing research ‘quality’.

The final speaker was Ed Pentz, Executive Director of CrossRef, the facilitator of the reference-linking system of scholarly publishers (currently with 550 publishers and 15,000 journals; CrossRef now has 35 million items including, as well as journal articles, book, book chapters and so forth). Ed questioned whether journals and articles will continue to retain the significance and brand importance that they presently enjoy. The rise of informal ‘Web 2.0’ tools for communication and the linking to new kinds of content are changing the paradigm. So are new kinds of ‘publication’, such as databases (e.g. protein sequence databanks) that are already assigning DOIs to items and wikis as a platform for (almost-formal) publishing . The latter are not yet assigning DOIs but the indications are that they are moving in this direction. Blogs are citing DOIs, even if they are not assigning any, and we are now seeing aggregations of blogs (e.g. Science Blogs), and scientists looking to such developments to give recognition to their work outside of the traditional mechanism of citing journal articles.