Information Research Weblog

I have posted the following to the JESSE discussion list and one or two more, so forgive me if you've already seen it.

Three years ago I commented on the extent to which Web citation was growing as a means of assessing the impact of research outputs. There was an interesting discussion and, eventually, prompted by the debate, a paper on the subject in JASIST by Vaughan and Shaw (Volume 54, Issue 14, Pages 1313 - 1322).

I'm returning to the topic because of the emergence of Scholar Google as an interesting new venture in the field of bibliographic control, and by my testing of frequently hit papers in Information Research. The results suggest that for university administrations to rely upon the Web of Science citation counts and the associated journal impact measures as a means of assessing faculty for promotion is rather flawed. The same might be said of the decisions by the Research Assessment panels of the Higher Education Funding Councils in the UK to rely upon ranked lists of journals in assessing research outputs.

I'm aware, of course, of Peter Jasco's very interesting paper on Scholar and Web of Science in Current Science, v.89, no. 9, 1537-1547 and no doubt his criticisms of Scholar will provoke some changes. However, my strategy was not affected by the system's shortcomings, since I was looking for citations of known items and I weeded out the occasional 'false drops'.

I looked at the papers in Information Research that had received 10 or more citations, according to Scholar, and the results can be seen at my Weblog.

To take one example: The effect of query complexity on Web searching results, by BJ Jansen (Information Research, Volume 6 No. 1 October 2000) was said by Scholar to have 24 citations (actually, 26 were recorded) - WoS gave it zero citations - and yet the citations found by Scholar were as follows:

Peer reviewed journals - 9 citations (4 journals, all of which are covered by WoS)

Conference papers - 8 citations

University department paper archive - 7 citations

Research group report - 1 citation

Thesis - 1 citation

The paper also had about 200 Web citations, as measured by searching on Google.com

What then, are we to regard as "impact"? It would be wise for any candidate for promotion to press the case for a wider definition than measurement by WoS citations provides. There are problems with other measures but, for example, the international impact is likely to be measured better by either Scholar or Web citation generally. Similarly, in relation to the UK's Research Assessment exercise, it would be sensible for the Higher Education Funding Councils to offer better guidance on how the impact of research outputs should be assessed.