Charles Oppenheim's lecture on the REF, Bibliometrics and suchlike!

Writing about web page http://www.hefce.ac.uk/Research/ref/

I'm catching up on myself after a busy couple of weeks: I attended a lecture by Charles Oppenheim, here at the University of Warwick way back in March. Here follows a summary of the sense I gained from Charles' lecture...

Charles' talk covered his involvement with bibliometrics and how HEFCE came to look at bibliometrics owing to Gordon Brown's announcement that the REF would include them, back when he was Chancellor of the Exchequer. Ever since that announcement, it seems that the REF has become less and less about bibliometrics, and that is perhaps as a result of the findings of their pilot. Although it's fair to say that we still don't know exactly what the REF will measure and how... or whether there will be one at all, depending on which party is in government in 2011.

The REF framework is described on the page I've linked to, and it includes three elements: the research environment, outputs and some kind of measure of impact. Charles suggested that one way to use bibliometrics for the REF, should you wish to, would be to use them to add evidence of impact, regardless of however the REF itself might use bibliometrics. I guess that will depend on what is meant by impact.

There are many ways to calculate bibliometrics. The REF pilot was only about Science, Technology and Medicine subjects and even that identified that there was no single method which correlated as consistently across all the disciplines. It is for this reason that there can be no cheap and reliable method to evaluate research by numbers! Even if there were a reliable method, it would be a backwards-looking measure, and future research strategy is important to a University. So bibliometrics can only ever be a part of the answer.

Charles admitted that he has published papers which argued that the RAE should be replaced by bibliometrics, but his papers also said that analyses of such measures would need to be done by subject experts. Correlations between the bibliometrics for research outputs and RAE scores can be found when looking at the bibliometrics on all a researcher's outputs, or on only the outputs put forward for the RAE. Only one study of UK RAE ratings and bibliometrics has not found a correlation of any kind, and that was one based at Cranfield which looked at impact factors of the journals in which outputs were published, rather than at the metrics for the individual articles, apparently.

Charles went on to bust some myths about bibliometrics, one of which was that there is no evidence to suggest that writing an article that is deliberately flawed will earn you more citations as people cite you to publish corrections and refutations of your work! Apparently, people are far more likely to ignore your work if you publish work full of mistakes... which makes sense to me. Another myth is that self-citations and citation syndicates would affect bibliometric measurements, but apparently the effects of these are minimal as they are statistically insignificant. In any case, whatever measurement is used for research measurement, people will alter their behaviour to game the system, and no measurement is entirely without flaws. At this point I wanted to ask Charles what his top tip for gaining most citations would be... and I suspect that the answer would be to publish the best quality of research on the most important topics, but I did want to put my tongue in my cheek and ask!

There are various sources of data. Charles refered to Thomson Reuter's data in Web of Knowledge and to Elsevier's SCOPUS data. At this point I wanted to ask Charles whether he saw repositories as a source of potential citations data. I did ask my question at the end of his talk, and he was very keen on repositories as a source of data for research outputs, but there wasn't really time to discuss more about it as a source of references and therefore citation measurements. Someone else asked about Google Scholar as a potential source of data, also at the end of Charles' talk and Charles more or less said that the measures on Google Scholar are of little use. In GScholar, the same article might be listed more than once in GScholar's results, but from different sources and the citation counts for each source might need to be added together, and in any case, GScholar will pick up on a citation in someone's slides or a student essay. Although Charles did say that even with such weaknesses there may still be correlations to be found with the RAE ratings, except that there is no study to prove such. (NB my own interpretation of why GScholar doesn't index all of Warwick's repository's articles is that they do conflate records for the same article in at least some instances, and present the publisher record only: see my WRAP blog for more info on that!)

However citations are measured, they can be used in different ways to inform the REF, and no-one is yet certain how they might be used. They might be used to inform judgements about particular papers. They might be used to ensure the consistency of panel review ratings, so that departments with similar citation rates will not be scored with vastly different star ratings, unless with good reason. And a new government might scrap the REF altogether and do something entirely different.

Some bibliometric measurement techniques that Charles has used:

1) Look at the total number of citations received by a department in the same period as covered by the former RAE and then correlate that with the RAE ratings.

2) Look at the average number of citations per member of staff within a department, and then to correlate this with the RAE outcomes.

3) Look at the citations of just the outputs reported for the RAE or the total outputs of the staff members.

At any rate, since Charles' talk there has been yet another article in the THES explaining further back-tracking on the part of HEFCE about the importance of bibliometrics and the timing of the REF and we are now all awaiting the outcome of the general election and any impact that may have...

Jenny Delasalle : 07 Apr 2010 15:59 |  Tags: Bibliometrics Publishing Ref |  Comments (0) |  Report a problem