Cheap at any price?
In the United Kingdom at least there now appears to be a belief that assuring quality means measuring things. This, as we have noted previously in this blog, lies at the heart of the Research Excellence Framework (REF – previously the Research Assessment Exercise), and it appears increasingly likely it will also be at the heart of the proposed Teaching Excellence Framework (TEF). In fact these exercises tend to consider an intriguing jumble of inputs and outputs and put a relative value on them. The result is seen as a kind of gold standard. The REF in particular is viewed not just as a table of research excellence, but also as some sort of indicator of wider institutional quality. Few seem to think, as perhaps they should, that such massive exercises will often prompt worthy mediocrity much more than intellectual creativity. And nobody much seems to want to ask why virtually no other country thinks this sort of thing is a good idea.
But maybe our fatalism that this is inevitably our destiny might be shaken a little if we thought more about the cost of it all. The journal Times Higher Education has recently referred to studies suggesting that the cost of the most recent REF may have been anything from £214 million to £1 billion. To put that into some sort of perspective, even the smaller of these figures is nearly as much as the entire annual funding of all of Ireland’s universities (including tuition fees paid by the state). For this kind of cost to be worthwhile it would have to guarantee an enormous explosion of research excellence producing massive educational and financial benefits to the institutions and to society. There is really no evidence to suggest this is the case. The history of the RAE and REF does show they prompted a much greater volume of publication, but there is no evidence at all that this generated a greater amount of innovative discovery or scholarly insight. In passing it can be said with some assurance that research funding does have such an impact, and competing for it produces benefits – but no such claims can be proven to be true for REF. And now we are apparently about to load another huge cost on to the system in TEF, almost certainly with similarly uncertain benefits.
We do need to secure high quality teaching and research. But we also need to display much more sophistication as to how this can be assured.
Explore posts in the same categories: higher educationTags: REF, Research Excellence Framework, TEF
You can comment below, or link to this permanent URL from your own site.
University Blog 
May 31, 2016 at 4:35 am
I am minded of the debate over whether Aberdeen College Library should adopt an electronic system to enable searching of its database. The naysayers said that you could buy a lot of books for the price of the electronic wizardry. FE/HE, like the military, has to move with the times.
Those in charge of Advertising spend have a similar problem with justifying their budgets: 50% of their spend is valuable the rest is waste. The problem is that they are not sure which is which.
May 31, 2016 at 10:49 am
From the evidence you cite above of the cost of the REF and its impact on research surely the money would be better spent on research grants to increase research. However that this not to say that the present research culture is perfect – far from it there needs to be more of an audit of the research, more quality assurance and not just a points system based on publications. There also needs to be more accountability built in and more equity across institutions and topics.
And as for the TEF – we already have a costly, inefficient system for research with the REF and now the UK government want the same for teaching. Yet we already have quality assurance procedures for teaching, with the HEA fellowship system. Perhaps if the HEA was financed properly, monies given for education research and a willingness to accept that research into education can be as valid as any other research with accountability put in place there would be no need for a TEF. Or am I just looking at the evidence and using it as a basis for my thoughts – now isn’t that an idea ….
June 1, 2016 at 3:52 pm
“In the United Kingdom at least there now appears to be a belief that assuring quality means measuring things.”
A propensity to categorize and count is nothing new – evolution theories might explain why such a propensity has come about – the problem is that measuring things has become an obsession, a “consuming love” as W.H. Auden put it in 1935:
And still they come, new from those
nations to which the study of that
which can be weighed and measured
is a consuming love.
In more recent times historian Alfred W. Crosby in The Measure of Reality (1997) has examined the epochal shift from qualitative to quantitative perception in Western Europe during the late Middle Ages and Renaissance. This is what made modern science, technology, business practice and, crucially for the purpose of this comment, *bureaucracy* possible.
No one can deny the usefulness of measuring as a strategy to make sense and confer a sense of reality upon our world, however what we witness in many organizations, including universities, these days is that measuring defines what is meaningful (or impactful to use REF jargon), rather than the other way around, i.e. the meaning of the work undertaken to define the measuring. Measuring has become an idol, a false god to worship on the altar of a rampant managerial culture. And how ironic that the word ‘management’ – deriving from the Italian (1561) maneggiare meaning “restrain” – is used in conjunction with innovation! Universities should relish the latter and not become servants of the former, they should be mindful, as George Herbert wrote in his poem Agonie (1633) that:
Philosophers have measur’d mountains,
Fathom’d the depths of the seas, of states, and kings,
Walk’d with a staff to heav’n, and traced fountains:
And yet,
…there are two vast, spacious things,
The which to measure it doth more behove:
Yet few there are that sound them; Sin and Love.
http://www.ccel.org/h/herbert/temple/Agonie.html
We cannot measure the immeasurable.
June 2, 2016 at 10:03 am
I have to say I somewhat agree with Anna, although I don’t believe the situation is quite as immeasurable as suggested. I think we can learn something from the private sector.
An investor in Apple is going to care about sales relative to Samsung. These are two massive companies going head-to-head in the market. But an investor in a new start-up producing phones isn’t going to worry too much about the current sales. They know that in a head-to-head comparison, the new company isn’t going to be shifting as many products as Apple. But what they will care about is the plan by management – how are things going to *change* over the next few years and are they getting value for money in terms of those projected changes.
Let’s take ‘Teaching and Learning’ in a university. How do you measure the effectiveness? Well, what you do is send your exam papers off to be reviewed by other universities, as is currently the case. You try to ensure that the exams you are setting are of a similar academic standard to other institutions. You then examine the progression of students through your degree programmes. Students who are academically strong when entering the university will have an average progression curve showing their marks as a function of year. Similarly, weaker students entering the university will have a different progression curve. The difference between them shows how effective your teaching is at improving the performance of weaker students. You look at these curves and you set a goal of narrowing them. You invest in maths training centres and additional supports. And as these initiatives come on stream, you demonstrate a change in the statistics, hopefully with more students achieving better grades relative to their stronger academic peers.
The point is that rather than national frameworks, all that is really required is for universities to find metrics they believe are important to that institution, establish that those metrics are what they are going to measure, that these can be compared from year to year, and that the plan for the institution is to change this metric. That’s what we’d expect of private sector managers – we don’t expect them to adhere to universal metrics that aren’t necessarily applicable to every company. It is the *change* in institutional metrics, rather than a comparative metric between institutions that has actual value to an institution and to the public funding them.
June 3, 2016 at 5:57 am
At the examiner´s meeting there was a spreadsheet put up displaying student results by course. The only person to worry was the lecturer who graded 1st class students other than 1st class and vice versa.
June 3, 2016 at 8:42 am
Always good when one *somewhat* agrees with you 🙂
For the record I did not argue that ‘the situation was immeasurable’, I argued that the situation in HE has become skewed towards an obsessive use of metrics, which in turn reflects a societal trend towards measuring everything in the misplaced belief that is the best way to assess quality (as the post argues).
We live in an age that has developed an unhealthy passion for quantifying everything, people count their steps as they count their money and their friends.We compartmentalize, reduce, and deconstruct the human experience into something we can quantify and analyze ourselves against. This is what lies behind “Big Data” (there is even a university by the same name http://bigdatauniversity.com/) which aims for everything to be quantified and tracked in order to enable smarter decisions, better products, greater knowledge, optimal solutions, customer-centric products, increased customer loyalty, more automated processes, more accurate predictive and prescriptive analytics, better models of future behaviors and outcomes in business, government, security, science, healthcare, education. All this “smartness” provides us with confidence and certitude but isn’t it the role of knowledge exactly the opposite, i.e. to rob us of our certainties?
June 5, 2016 at 5:07 am
As the chart shows there are trends which with the benefit of hindsight were inevitable; so I am undecided as to what the purpose of knowledge is. I remember being taught as an absolute certainty that England was last invaded in 1066 by the French. It turns out that the Dutch gloriously popped in later and lately hordes from other parts of the EU. We will soon see how certain people are armed with ¨facts¨ at the referendum ballot box.
June 4, 2016 at 12:47 pm
Well whatever may have been learnt in Britain, in Ireland there is a new found enthusiasm for the RAE type measures, perhaps even more extreme as they are based on complying with the rather science and North Atlantic skewed SCOPUS list of journals, deemed the only ones that count as ‘research’. Having seen the negative impact of the RAE up close (and No I am not of course against research capacity building!) I do wonder if academics are slow learners. Why on earth would anyone who cares for intellectual discovery and creativity want to introduce this in Ireland, not even by demand from above, really beats me.