Skip to content

The Research Excellence Framework and me

June 10, 2011

I have to start this post with a disclaimer: I am a departmental director of research (DoR), and so much of what I am going to talk about is something I have to work with every day.  The Research Excellence Framework (REF) is one part of the UK’s dual funding system for academic research, and as a DoR I am intimately involved in the development of our REF submission.  It is worth stating right now that not doing this is simply not an option for any department that considers itself research active.  But that doesn’t mean we are happy about it.  I’d like to use this post to get through some of my thoughts on the matter.  So this post is not intended to be a definitive critique of UK research funding.  It can’t be, because it only addresses about half of it by definition.  It will focus on what I think of various aspects of the REF, chiefly its lack of transparency.

The REF is also known in university circles as QR (quality-related) funding (to differentiate it from research grant capture).  Universities work very hard to achieve high ratings in the REF (or the RAE as was) because it provides a quantity with which universities compare each other, and the amount of money HE funding councils dish out is related to the “QR” university departments get.  How is the REF scored? It is relatively straightforward.  University departments submit a portfolio of staff work and a narrative of the research environment, along with figures for postgraduate research student numbers and grant income, and finally “impact case studies”.  The submissions are rated on the quality of outputs (65% of the total), the economic and social impact of our research (20%), and the research environment (15%).  The environment is scored from the figures and the narrative, the impact from case studies and supporting evidence and the quality of outputs from each member of staff submitting up to four research items (papers, for the most part).  That is about as much as I can tell you.  In fact, I’m pretty sure that is about as much as anybody can tell you with any certainty.  And this is my fundamental problem with the REF.  For all the consultation documents, guidance notes, and workshops offered, the REF remains scandalously opaque.  The feedback given to individual departments following the last RAE was usually shorter than the average research paper review, and certainly shorter than this blog post.  Bear in mind that this exercise decides the fate of about £1.6 Bn of funding annually.

The merits and problems of the REF are many and varied, but I’d like to briefly address the lack of transparency from the point of view of an individual researcher.  There is no doubt the REF will shape an academic’s career: you wouldn’t want to go to a department that wasn’t planning to submit to the next REF, and a department wouldn’t want to recruit somebody who isn’t “REF returnable”.  Sad but true.

Against this background, there is my personal involvement in the last RAE.  It was the first time I was included in a submission.  I submitted 4 papers, those I considered my best, and a few esteem indicators (mercifully cut from the REF!) and gave some words for the environment narrative context.  Our outcome, for a department that didn’t exist before 2002 and whose staff were almost all less than 5 years post-doc, was creditable (15% 3*, 50% 2* and 35% 1* – 4* being world class, 3* internationally excellent, 2* internationally recognised and 1* nationally recognised.  As an aside, Vince Cable recently considered anything less than 3* to be “mediocre”, which finally convinced me that rather than being a visionary he is in fact a massive tool).  I can’t say much beyond that because I don’t know anything more about it. Really. Here are the things I don’t know:

1. I don’t know what ratings any of my papers were given, or if the RAE panel members even read them. Nobody I work with does or ever has known this information.  It’s like doing an exam and never knowing the result.  Some of those papers will form the basis of an “impact case study” which might go into the next REF.  These papers have to be rated at least 2* for the case study to count as impact from excellent research.  But since I don’t know what my papers were previously rated, and don’t have any way of finding out, I am stuffed.  More generally, this lack of feedback is corrosive.  Every department in every university will have a range of RAE/REF scores associated with their submission.  In sport and exercise science, the best department in the country had 25% of its research rated at 4*.  Cue the “we are a world-class research department” fanfare, and they were not alone.  They skillfully ignored the obvious fact that the majority of the work that they do is not world class, and so did everybody else.  Moreover, it is a safe bet that every single member of staff submitted assumed that they were responsible (in part) for the 4* grading.  Nobody is likely to claim they were responsible for the 10% rated at 1*.  And nobody will ever find out either.  I don’t just find this unhelpful, I find it completely alien to the way I work, which is to base most of what I do on tangible evidence.  What is even more surprising is that the REF is constructed and assessed by senior academics.

It is possible that we could obtain the feedback we need by FoI requests to HEFCE.  I’d be interested to know if anybody does this routinely or has ever tried it.  The alternative is to invite senior people who have been panel members before to review what we have and rate it.  This we have done in the past, and the exercise was reassuring but still unsatisfying, because the scores dished out often correlated surprisingly well with journal impact factors, which gave a pretty big hint as to the method of assessment.  Moreover, there is no way of ensuring the accuracy of those scores as we have no validated measure in the first place (that is, scores from previous assessments).  Beyond that, why should senior people have to perform this function?  Why should FoI requests be necessary?  We surely have the right to access this information, especially as it is informing public funding decisions.

2. Why do we submit 4 outputs?  Why not all of them published in the census period?  Why not 3, or 5 or even only one if the panels are struggling to assess everything and are trying to select excellence in research?

3. Why are departments still allowed to select staff?  If we are funding departments based on the work they’ve done, surely we need to declare all of it?  I’ve seen cases where less than 15% of a department gets submitted to boost the score.  Given the huge multipliers used at 3* and 4* this time around, this will only get worse.

4. How on earth is the environment assessed?  There is only one environment, so how does its score get spread out (e.g., 10% 4*, 30% 3*, etc.)?

5. Which idiot decided that departments who support a member of staff for years get no credit whatsoever if that member of staff moves on a year before the census date?  Was this the same idiot who decided that impact was a departmental thing, meaning that a member of staff that moves cannot take their impact with them, even though they did all the work, it’s their IP and it contributes at least 20% to the REF score?  In exactly the same way as the “outputs stay with the researcher” issue decimated smaller departments before the last RAE as bigger departments offered lucrative employment packages to star researchers, the structure of the impact assessment has the potential to cause sector-wide stagnation as senior people may be reluctant to move if it means losing demonstrable impact.

6. How much money is 3* and 4* going to be worth (2* – internationally recognised research – is not good enough to be funded anymore, and with the cuts that is unlikely to change in the medium-term)?  This for me is a scandal few outside academia know about.  A year after submission the ratings are published.  About 3 months after that, the submissions are published (without feedback), then some weeks or months after that, the universities are told how much money they receive.  This is at the discretion of the regional funding councils and absolutely nobody knows how much this will be until it is announced.  The amount of money you receive can also change when new regional budgets are announced: what we get now bears no relationship to what we got in 2009/10.  Little wonder, then, that most university income is secured from undergraduate students: this is a much bigger pot and the amount in said pot is largely safe for the next 3 years.  Research income is far too volatile to be relied upon unless you are department already securing tens of millions of pounds in research council grants.

There is little in the way of good news here.  I have often said (off the record) “to hell with the RAE/REF!”  If you are motivated to do research, then get on and do it, as the REF will take care of itself.  Unfortunately, I no longer have the luxury of thinking that ignorance is bliss.  The major problem, as it always has been, is that even if you work really hard to find out how it works, and even if your job depends upon it, you will still be largely ignorant of the internal workings of the REF.

3 Comments leave one →
  1. July 5, 2013 5:03 pm

    If your late-night eating is causing you to always go over your desired calorie
    intake for the dieting or exercise day, avoid eating after
    your evening meal. Yo-yo dieting or exercise has also been
    found to be associated with increased risks of diabetes, heart, and
    stomach. So it doesn’t help people lose weight, but the real challenge is keeping the weight off, because of the lack of these nutrients. There is no way to lose weight is going on a diet.

  2. Rudy permalink
    May 31, 2016 4:01 pm

    Thanks for this. Roughly how much money would you reckon each 4star submission brings into a department. I know this overly complex, but I was wondering if you had any insights. Kind regards,


  1. Reflecting on the Research Excellence Framework 2014 | Dr Burnley's Third Eye

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: