The UK REF 2014 – my notes on what citation measurement we might expect

Writing about web page http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf

These are my notes upon reading the REF 2014 document: “Assessment framework and guidance on submissions” - http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/02_11.pdf page 22 onwards.


Each member of staff who is eligible for inclusion in the REF submission will have up to four outputs submitted.


WHAT CAN BE SUBMITTED:


In summary, each output must be:

- “The product of research, briefly defined as a process of investigation leading to new insights, effectively shared.” There is a full definition of research in the documentation.

- “First brought into the public domain during the publication period, 1 January 2008 to 31 December 2013 or, if a confidential report, lodged with the body to whom it is confidential during this same period”

NB if you get an online pre-print available within these dates but the official publication date is after 31 December 2013, you can still submit the item so long as you have evidence to prove the public domain availability of the item. One form of evidence relating to web content which is acceptable is: “a date-stamped scanned or physical printout or evidence derived from web-site archiving services.” (See Paragraph 111) Likewise though, if your publication was actually in the public domain prior to 1 January 2008 then it won’t be eligible.

- “Produced or authored solely, or co-produced or co-authored, by the member of staff against whom the output is listed, regardless of where the member of staff was employed at the time they produced that output.”

Although the official documentation expands on this more. (Paragraph 105 onwards)


Examples of output types given are:

“new materials, devices, images, artefacts, products and buildings; confidential or technical reports; intellectual property, whether in patents or other forms; performances, exhibits or events; work published in non-print media”, and other types can be included. The documentation goes on to say: “Reviews, textbooks or edited works (including editions of texts and translations) may be included if they embody research…”


“A confidential report may only be submitted if the HEI has prior permission from the sponsoring organisation that the output may be made available for assessment.” (para 115)


WHAT IS NOT ELIGIBLE:

Editorships or other activities are not outputs, and so should not be included in the submission. Theses and “items submitted for a research degree” won’t count, although it does seem that published items and other eligible outputs based on your research degree can be listed.

Panels might choose to assess two outputs which are based on the same research and so have “significant material in common” as a single output, or to assess just the content which is distinct in each. Panels own judements will also be used in the instance of a publication which is a version of one published prior to 1 January 2008, as to whether the publication is eligible and how it is to be assessed: “Submissions should explain where necessary how far any work published earlier was revised to incorporate new material” (Paragraph 113)

“HEIs may not list as the output of a staff member any output produced by a research assistant or research student whom they supervised, unless the staff member co-authored or co-produced the output.” (para 110)

DATA ABOUT THE OUTPUTS

Output types are to be categorized in the institution’s submission, into:

i. Books (or parts of books).

ii. Journal articles and conference contributions.

iii. Physical artefacts.

iv. Exhibitions and performances.

v. Other documents.

vi. Digital artefacts (including web content).

vii. Other.

(Para 118) and there will be different data requirements for each of these categories.

Paragraph 119 says:

“Each of the following is required where applicable to the output:

a. Co-authors: the number of additional co-authors.

b. Interdisciplinary research: a flag to indicate to the sub-panel if the output embodies interdisciplinary research.

c. The research group to which the research output is assigned, if applicable. This is not a mandatory field, and neither the presence nor absence of research group is assumed.

d. Request for cross-referral: a request to the sub-panel to consider cross-referring the output to another sub-panel for advice (see paragraph 75d).

e. Request to ’double weight’ the output: for outputs of extended scale and scope, the submitting institution may request that the sub-panel weights the output as two (see paragraphs 123-126).

f. Additional information: Only where required in the relevant panel criteria, a brief statement of additional information to inform the assessment (see paragraph 127).

g. A brief abstract, for outputs in languages other than English (see paragraph 128-130).”


And there is much more information about how information on each of these features can be provided, in the document.


CITATION DATA:

“Some sub-panels will consider the number of times that an output has been cited, as additional information about the academic significance of submitted outputs.” (Paragraph 131) They won’t be interested in the impact factors of the journals as such, but the number of citations accrued by the outputs themselves. The citation data is to be provided to the panels by the REF team, and submissions may not include details of citations in additional information for outputs.


“In using such data panels will recognise the limited value of citation data for recently published outputs, the variable citation patterns for different fields of research, the possibility of ‘negative citations’, and the limitations of such data for outputs in languages other than English.” (para 132)


I’ve not read criteria from each panel, but David Young’s blog entry at http://research.blogs.lincoln.ac.uk/2011/08/01/hefce-publishes-draft-ref-panel-criteria-and-working-methods/ neatly summarises the expected use of citation data across the different panels (although note that consultation is still underway at present):

* “Different panels and UoAs will use citation data to differing degrees, and they will also differ in what kinds of outputs are acceptable
* All sub-panels of Panel A (life sciences and allied health disciplines) will use citation data “where it is available, as an indicator of the academic impact of the outputs, to inform its assessment of output quality”
* Just under half of the sub-panels in Panel B (physical sciences) will use citation data: Earth systems & Environment, Chemistry, Physics, and Computer Science.
* Two sub-panels in Panel C (social sciences) will make use of citation data: Geography, Environmental Studies and Archaeology (although not all of this UoA will use citations) and Economics and Econometrics.
* None of Panel D (arts and humanities) will use citation data.
* Physical sciences will be able to submit a larger range of outputs, including patents, computer algorithms and software. In contrast life scientists will be more restricted, and can only include some kinds of outputs, such as databases or textbooks “exceptionally”.”

Where sub-panels are using citation data, it is being made available to them, matched to outputs by the REF team, using DOIs and other bibliographic data. Institutions will be able to verify these matches and to view the citation counts provided to the panels. Citations made after the submission deadline will continue to be counted and provided to the panels. (See para 133)

ACCESS TO SUBMITTED OUTPUTS

Journal articles and conference papers will be accessed by the REF team via the publishers, so will require DOIs in the submission. Other output types can be provided in an electronic format, or a physical copy, or as appropriate evidence, seemingly in that order of preference. We await the submission system software in autumn 2012.

Jenny Delasalle : 04 Oct 2011 12:00 |  Tags: Bibliometrics Ref Researchers |  Comments (0) |  Report a problem