Principles
47. The fundamental over-riding criterion for selecting outputs for submission to the REF and attributing those outputs to staff will be one of quality of the output, and maximising the quality profile of the outputs sub-profile within a UoA. This means that, in general, those outputs which are considered to be the highest quality will be submitted, subject to the Research England parameters of attributing between 1 to 5 outputs to each eligible staff member within a unit, and submitting the overall required number of outputs (generally 2.5 x staff FTE).
48. The final selection of outputs and attribution of outputs to individual staff will be obtained by viewing the UoA submission as a whole, and making a strategic decision about how the unit should best be presented to optimise the outcome in terms of the overall quality profile for the unit and the College as a whole. This will include a consideration of how the research environment of a submission may be assessed and, as such, may mean that the final pool of outputs selected for submission within a UoA may not necessarily coincide with the pool of outputs felt likely to achieve the best assessment within the outputs sub-profile alone.
49. The REF assesses the quality of a unit submission as a whole – the quality of individual staff members is not assessed. No inferences can be drawn on the quality of individual staff from the final selection and attribution of outputs to staff and these will not be published by Research England or the College. The attribution of outputs to staff within a REF submission will not be used for any other purpose beyond the REF. In particular the College will not take into account the attribution of outputs to staff within a REF submission in any future promotion, remuneration, career progression, extension of contract or performance management considerations.
Process
50. Each leading Faculty Committee will be responsible for initially reviewing outputs within their UoA and determining which outputs will be submitted in line with the parameters within the REF guidance and the principles described above. In making its decision, the leading Faculty committee may be informed by an assessment of the quality of outputs by departmental REF lead(s) most relevant to the UoA.
51. The Vice-Provost (Research and Enterprise), advised by the REF Strategy Group, will have the ultimate responsibility for the final decision on output selection, considering the best possible REF profile for the College, and for a given UoA.
52. The REF Equality Committee will have responsibility for ensuring that decisions around the selection of research outputs are made with due regard to equality and diversity, including in relation to where staff have voluntarily declared circumstances affecting their ability to research productively over the REF reporting period.
Detailed process for selecting outputs
53. Once the total number of outputs to be submitted within a UoA is known, the leading Faculty committee will select outputs for submission by the process outlined below.
54. All eligible staff will be invited to self-select for consideration to be submitted to the REF the one output they have authored over the REF submission period which they consider to be the highest quality, and to optionally put forward any further outputs which they consider to be of high quality, subject to a maximum number which will be set by the individual UoA.
55. Drawing from the pool of eligible outputs self-selected for consideration to be submitted by staff, each eligible member of staff, will initially be attributed to one eligible output (except in Research England approved cases where staff have circumstances meaning they have 0 outputs – see paragraphs 62-75). These will be attributed to staff in such a way that maximises the overall output quality sub-profile for the unit as a whole; the attribution of outputs to staff will not necessarily coincide with the self-selection of outputs by staff, nor will it necessarily result in each eligible staff member being attributed to the highest quality output they have put forward for submission.
56. The remaining outputs needed to obtain the total number of outputs to be submitted within a UoA will then initially be formed by the highest quality outputs which have not yet been attributed to staff via the process outlined in paragraph 54. These will be selected from the remaining pool of self-selected eligible outputs and attributed to staff in such a way that maximises the overall output quality sub-profile for the unit as a whole, subject to the maximum of 5 outputs that may be attributed to any one staff member who is REF eligible on the census date of 31 July 2020; the attribution of outputs to staff will not necessarily coincide with the self-selection of outputs by staff.
57. The final pool of outputs for submission will be formed, by additionally taking the following outputs into consideration:
a. Eligible outputs from employed staff who are REF eligible on the census date of 31 July 2020, but which have not been self-selected for submission consideration by any member of staff.
b. Eligible outputs from employed staff who are not REF eligible on the census date of 31 July 2020 but have been employed as REF eligible at some previous point over the REF reporting period.
c. Eligible outputs from staff who are not employed on the census date of 31 July 2020 but have been employed as REF eligible at some previous point over the REF reporting period. This may include outputs from staff not employed on 31 July 2020, but who have been previously employed as REF eligible on a fixed-term contract, which ended prior to 31 July 2020. Outputs will not be considered for submission from staff not employed on 31 July 2020, who were previously employed as REF eligible, but have subsequently been made redundant as a result of restructuring. Outputs may be considered from staff formerly on a fixed-term contract which has ended by the census date.
58. In forming the final pool of outputs, outputs from any of the three pools described in paragraph 56 may be added to or replace outputs from the pool initially obtained via the process outlined in paragraph 54-55. Such additions and\or replacements will take place where one or both of the following criteria are satisfied:
a. Quality. An addition\replacement increases the overall output quality sub-profile of the submission as a whole
b. Strategic. An addition\replacement enhances the presentation of the overall submission considered across the entire UoA, or for the College as a whole, such that the overall quality profile (including taking into account the environment sub-profile) is optimised.
59. Additions\Replacements must keep the overall output pool within the parameters of the attribution of outputs to REF eligible staff on the census date of 31 July 2020, specified within the Research England Guidance on Submission i.e. 1-5 outputs attributed to each REF eligible staff member1.
60. In all cases, judgements of quality will be formed by a combination of internal peer review, external expert review as considered appropriate, and committees responsible for REF decision making (see Annexes1-3).
61. In accordance with the College being a signatory of the San Francisco Declaration on Research Assessment (DORA), no reference will be made to journal-based metrics such as Journal Impact Factors when forming judgements on the quality of research outputs. Where available and appropriate, citation data will be considered as an indicator to inform judgements on output quality, for submissions to units of assessment, or for outputs likely to be cross-referred to units of assessment; where the sub-panels have confirmed that they will make use of citation data in their assessment (This applies to all sub-panels within Main Panel A, the following sub-panels within Main Panel B: 7 – Earth Systems and Environmental Sciences, 8 – Chemistry, 9 – Physics and 11 – Computer Science and Informatics and sub-panel 16 – Economics and Econometrics, within Main Panel C).
Appeals (output selection)
62. The REF assesses the quality of a unit submission as a whole and individual staff members are not assessed. No inferences will be drawn on the quality of individual staff from the final selection and attribution of outputs to staff and these will not be published by Research England or the College. Therefore, and in accordance with guidance issued by Research England, there will be no appeals process on the selection of outputs, or the attribution of outputs to individual staff members within a submission.
Staff circumstances
63. The College recognises that a range of individual circumstances may have had a material impact on the quantity of research outputs that staff have produced over the REF census period and does not expect every eligible staff member to contribute equally to the volume of outputs submitted.
64. All staff will be invited to self-select for consideration to be submitted to the REF the one output
they have authored over the REF submission period which they consider to be the highest quality, and to optionally put forward any further outputs which they consider to be of high quality. Although staff will be encouraged to provide more than the minimum of one output, and it is thought likely that many will choose to do so, there is no expectation for staff, regardless of their individual circumstances, to self-select for consideration to be submitted to the REF, more than the minimum of 1 output.
65. Staff will be given the opportunity to voluntarily and confidentially disclose any circumstances which they believe have constrained their ability to work productively throughout the REF census period, in particular if this means that, exceptionally, they have not authored any outputs over the REF census period. Some staff may however prefer to keep this information confidential and the College respects absolutely, their right to privacy over personal information.
66. For staff wishing to disclose circumstances, this will be via a confidential online or paper form (Annex 8) regarding any circumstances for submission to the Deputy Director, Organisational Development and Inclusion. Staff can also request a confidential ‘surgery’ with their Faculty HR representative to discuss any complex special circumstance.
67. The form is collected by the College for REF purposes only. Staff are in no way obliged to complete the form or disclose any circumstance if they do not wish to do so. By completing the form, the individual will be providing their written consent for the information to be considered, on a confidential and sensitive basis, by the College’s REF Equality Committe and the College central REF team.
Summary of applicable circumstances
68. In accordance with guidance issued by Research England, the following equality-related circumstances, in isolation or together, may significantly constrain the ability of staff to produce outputs or to work productively throughout the assessment period.
a. Straightforward circumstances.
i. Qualifying as an Early Career Researcher ECR; that is having become an independent researcher on or after 1 August 2016
ii. Absence from work due to secondments or career breaks outside the HE sector
iii. Qualifying periods of family-related leave
iv. Part-time working
v. Junior clinical academics. These are defined as clinically qualified academics who are still completing their clinical training in medicine or dentistry and have not gained a Certificate of Completion of Training (CCT) or its equivalent prior to 31 July 2020.
b. More complex circumstances with an equivalent effect to absence, that require a judgement:
i. Disability
ii. Ill health, injury, or mental health conditions
iii. Constraints relating to pregnancy, maternity, paternity, adoption or childcare that fall outside of those covered by 67a
iv. Other caring responsibilities (such as caring for an elderly or disabled family member)
v. Gender reassignment
vi. Other circumstances relating to the protected characteristics listed in Annex 6, or relating to activities protected by employment legislation
69. In accordance with Research England guidance, the following criteria may mean that staff have been unable to produce any eligible outputs over the REF assessment period:
a. an overall period of 46 months or more absence from research during the assessment period, due to one or more of the circumstances set out in paragraph 67 (such as an ECR who has only been employed as an eligible staff member for part of the assessment period)
b. circumstances equivalent to 46 months or more absence from research, where circumstances set out in paragraph 67 apply (such as mental health issues, caring responsibility, long-term health conditions) or two or more qualifying periods of family- related leave, as defined in Annex 7.
70. The information on staff circumstances will be considered by the REF Equality Committee who will then advise the relevant Faculty and Department of the total reduction in outputs which could be applied for within a UoA submission, making use of the algorithms provided by Research England (see Annex 7) for staff with straight-forward circumstances, and forming a judgement on more complex circumstances. This will be considered alongside the total pool of outputs collected and a decision made over whether to seek approval from Research England of any reduction in the required number of outputs to be submitted in a unit. The REF Equality Committee will also advise the relevant Faculty and Department on any staff for whom a request may be made to Research England for them to be submitted with 0 outputs, without the relevant UoA receiving a penalty score of ‘unclassified’ for 1 output.
71. The decision to seek reductions in the overall number of outputs submitted within a UoA will be subject to the final approval of the REF Strategy Group. Requests for reductions will generally only be considered where a unit has a critical mass of staff who have declared circumstances meaning that they have produced less than the average number of outputs required within a unit. In accordance with Research England guidance, reference will also be made to the total number of eligible outputs that may be submitted within the Unit of Assessment.
72. Where an overall reduction in the number of outputs to be submitted within a UoA has been approved by Research England as a result of staff circumstances, the number of outputs attributed to the relevant staff with circumstances will not be greater than the overall average number of outputs per staff within a UoA (or zero, where reductions have been approved for individual staff to be submitted with zero outputs).
73. Applications to Research England to reduce outputs attributed to individual staff below the minimum of 1, will generally be made if staff have disclosed circumstances meaning they meet the criteria specified in paragraph 68, and there are no eligible outputs over the REF submission period that may be attributed to them.
74. Decisions over applications for reduced outputs will only take into account information on individual circumstances which have been voluntarily declared by staff via the process outlined in paragraphs 64-66. No consideration will be given to information on individual staff circumstances disclosed or otherwise known outside of this process. Where staff have voluntarily declared circumstances affecting their research productivity, which the REF Equality Committee agrees meet the criteria set out in paragraphs 68-69, the College will adjust its expectations on their contribution to the output pool, regardless of whether this leads to a successful application to Research England to reduce the number of outputs within a UoA.
75. The expectation on such staff will be adjusted as follows:
a. For staff who have voluntarily declared circumstances but have produced at least one eligible output, it will not generally be necessary to adjust expectations as the default expectation on all staff is that a minimum of one output be self-selected for
consideration to be submitted to the REF. However, for such staff, the Head of Department will re-confirm to the staff member that they are expected to contribute no more than the minimum of one output.
b. For staff who have voluntarily declared circumstances and have produced no outputs, the Head of Department will confirm to the staff member that they are not expected to contribute any outputs for submission to the REF.
Adjustments to expectations will be made regardless of whether voluntarily declared circumstances lead to a successful application to Research England to reduce the number of outputs within a UoA.
Appeals for staff circumstances
76. Staff who have voluntary declared circumstances which have affected their ability to research productively over the REF assessment period may appeal if they feel that these circumstances have not been correctly taken into account in the expectations upon their contribution to the overall output pool within a unit of assessment submission.
77. Any grounds for appeal must focus on why the individual believes that, having voluntarily declared circumstances affecting their ability to research productively, he or she has, unjustly, been asked to contribute a minimum number of outputs to a submission. It would thus be appropriate for an appeal to be made on the grounds of:
i. Unfair discrimination
ii. Process (including if it is felt that procedure has not been followed)
iii. Previously unavailable evidence
78. Disagreement with the decision on the contribution to the output pool alone would not be appropriate grounds for an appeal.
79. Since the general expectation on all staff is that they contribute a minimum of one output to a submission, appeals would normally only be valid in cases where staff have voluntary declared circumstances meaning that they do not have any outputs to contribute to a submission within a unit of assessment (in accordance with paragraph 68). However appeals may also be valid if an individual who has voluntarily declared circumstances affecting their ability to research productively believes that they have expectations placed upon them to contribute more than the general minimum of one output.
80. See Annex 5 for the details of the appeals process.
Equality impact assessment
81. In accordance with guidance provided by Research England, an Equality Impact Assessment will be undertaken to compare and analyse the protected characteristics (where data is available) of staff falling into each of the 6 pools determined by the number of outputs attributed (5, 4, 3, 2, 1 or 0). The results of this analysis will be provided to the REF Equality Committee, who will then advise the REF Strategy Group as appropriate. Where there is a significant underrepresentation of staff with protected characteristics within the selected output pool, the potential reasons for this will be explored further. This may be caused by:
a. a bias within the REF procedures for selecting outputs from the overall output pool produced over the REF submission period. In this case, alternative procedures for selecting outputs will be explored, potentially resulting in a reselection of outputs.
b. an underrepresentation of outputs from staff with protected characteristics in the overall output pool produced over the REF submission period. This could potentially indicate a more fundamental issue of restricted opportunity for support within the research environment within a UoA, which would be outside of the remit of the REF code of practice. Such cases will be referred to the EDI forum for further investigation.
82. Protected characteristics considered are those outlined in Annex 6, where this data is held.
