Cheap at any price?

In the United Kingdom at least there now appears to be a belief that assuring quality means measuring things. This, as we have noted previously in this blog, lies at the heart of the Research Excellence Framework (REF – previously the Research Assessment Exercise), and it appears increasingly likely it will also be at the heart of the proposed Teaching Excellence Framework (TEF). In fact these exercises tend to consider an intriguing jumble of inputs and outputs and put a relative value on them. The result is seen as a kind of gold standard. The REF in particular is viewed not just as a table of research excellence, but also as some sort of indicator of wider institutional quality. Few seem to think, as perhaps they should, that such massive exercises will often prompt worthy mediocrity much more than intellectual creativity. And nobody much seems to want to ask why virtually no other country thinks this sort of thing is a good idea.

But maybe our fatalism that this is inevitably our destiny might be shaken a little if we thought more about the cost of it all. The journal Times Higher Education has recently referred to studies suggesting that the cost of the most recent REF may have been anything from £214 million to £1 billion. To put that into some sort of perspective, even the smaller of these figures is nearly as much as the entire annual funding of all of Ireland’s universities (including tuition fees paid by the state). For this kind of cost to be worthwhile it would have to guarantee an enormous explosion of research excellence producing massive educational and financial benefits to the institutions and to society. There is really no evidence to suggest this is the case. The history of the RAE and REF does show they prompted a much greater volume of publication, but there is no evidence at all that this generated a greater amount of innovative discovery or scholarly insight. In passing it can be said with some assurance that research funding does have such an impact, and competing for it produces benefits – but no such claims can be proven to be true for REF. And now we are apparently about to load another huge cost on to the system in TEF, almost certainly with similarly uncertain benefits.

We do need to secure high quality teaching and research. But we also need to display much more sophistication as to how this can be assured.

Advertisement

In this game is the REF to blame?

Anyone working in and around higher education in the United Kingdom will have been obsessing about the ‘Research Excellence Framework’ (REF) over the past week. According to the REF website, it is ‘the new system for assessing the quality of research in UK higher education institutions’. A total of 154 institutions made 1,911 submissions to this exercise, and last week they found out how they had fared. The results will influence a number of things, including league table positions of universities and public funding. They will also have reinforced a trend to focus research attention and funding on a smaller number of institutions.

REF is the successor to the Research Assessment Exercise, which in turn had been around since the 1980s. The first one of these I had to deal with was conducted in 1992, when I was Dean of the Law School of the University of Hull. While I believe I was rather successful in managing the RAE, in that my department improved hugely between 1992 and the next exercise in 1998, I now believe that most of the decisions we took were good for the RAE and bad for research. In fact, that could be the overall summary for the whole process across the country from the beginnings right up to last week’s REF.

And here are three reasons.

1. The RAE and REF have, despite claims to the contrary, punished interdisciplinarity, because the units of assessment overwhelmingly focus on outputs within rather than between disciplines. The future of research is interdisciplinary – but academics worried about REF will be wary of focusing too much on such work.
2. Despite the way in which it aims to reward international recognition, a key impact of the RAE/REF framework is to promote mediocrity. For funding and related reasons, many institutions will try to drive as many academics as possible into published research, spending major resources on pushing average researchers to perform – resources that should really be devoted to supporting those who have the most promise. Of course some excellent researchers have been able to thrive, but in many institutions the RAE/REF process has hindered rather than supported real excellence. On top of that it has diverted some staff from doing what they do really well into doing things they don’t much like. One of the casualties of that, incidentally, is collegiality.
3. The RAE/REF has produced a stunning bureaucratisation of research. A key difference between research management in my last university in Ireland (where there is no such exercise) and in my current one is the extraordinary amount of time staff have to put into the tactical, operational and administrative maintenance of the REF industry. Also, I shudder to think how much time and resources institutions will have spent last week managing the news of the results. Industrial-scale bureaucracy of course also produces huge costs.

Other equally good reasons for doubting the value of REF have been given by Professor Derek Sayer of Lancaster University, writing in the Guardian.

I am not against competition in research, nor do I believe that research performance should not be monitored. But the RAE/REF process is about ranking universities rather than promoting research. I have no reason to think that anyone who matters is listening, but it is time to think again about this process.

Making an impact – or not …

One of the defining questions being asked about universities around the world is what impact they should have on society. Should the benefit of a university education, or of university research, be that it will have created capacity for independent thinking and evaluation, or facilitated discovery and innovation that will ultimately produce technological, business, social or cultural improvements? Or should there be something much more direct, whereby students learn skills that are needed under current economic conditions, or whereby research is focused on problems to which society wants urgent solutions?

The problem for universities has been that as public investment in higher education has risen exponentially over the past century (regardless of whether that investment has been sufficient to meet the desired ends), expectations have risen that the investment, or at least a good part of it, will be directed towards supporting public policy as identified either by politicians, or by the media, or by various interest groups and stakeholders. This in turn has chipped away at the traditional expectations of academic autonomy and freedom.

Some of this has come to a head in the United Kingdom as the planned new Research Excellence Framework (REF) is being debated. Under this framework (which is intended to replace the former Research Assessment Exercise) research performance will be evaluated in line with a number of criteria, one of which will be its ‘impact’. This will be assessed by asking whether the research in question has been able to ‘deliver demonstrable benefits to the economy, society, public policy, culture and quality of life.’

Ever since the inclusion of impact as a yardstick has been revealed, it has produced a significant backlash. Most recently Ralph Wedgwood, Professor of Philosophy at Oxford University, has written an article in the Daily Telegraph newspaper in which he calls the use of impact ‘clumsy’ and ‘ill-judged’. He takes this view in part at least because he feels that his own discipline would not be judged to have the kind of ‘impact’ that policy-makers want to measure. Other academics are taking a similar view, and it is possible that attempts will be made to boycott the REF.

Leaving that specific UK context aside, other university systems will have to address this issue also. And once again, this requires us to look more closely at what the principles of higher education ought to be. Universities are still often presented as institutions that have stood the test of time and that have perfected their ethos and working methods over the centuries, and that should therefore be left to get on with what they have always done. Others express impatience with this attitude and say that a big public investment entitles the taxpayer to expect specific actions and solutions. Right now various working groups and committees are assessing these matters, but the universities themselves are rather silent. It is time that a more open and audible debate should be taking place within higher education, and I am hoping to organise a conference around these themes in DCU before I end my term of office. When this happens, I hope some readers of this blog will want to participate.

Assessing the value of research

I was at a function recently when I was accosted  by one of the other persons attending. How, she wanted to know, could I justify all that ‘useless research’ that was going on in my university?  She wasn’t against research – not at all, in fact: she wanted us to find the cure for cancer, the answer to Dublin’s traffic problems, and a solution to all those under-funded pension schemes. And instead, what were we working on? Well, she had heard someone say that research was being funded by the taxpayer to analyse the ‘syntax of Wordsworth’s poetry’! I mean, can you imagine?

My first response when she paused to draw breath was that DCU was working on the three topics she mentioned (well actually, I don’t think we’re working on Dublin’s traffic, but I wasn’t going to admit that). But, I pointed out, it was important for society that there would be some researchers who were not working to a particular practical agenda, because they might well discover things that nobody had yet anticipated but which would change our lives. OK, she conceded, but Wordsworth’s syntax? I had no idea who if anyone really was working on this, but I pointed out that such research might produce valuable insights into the effectiveness of communication (well, I had to think of something quickly…).

But even if I found this conversation a little annoying, she was raising an issue with which we do need to come to grips: what is university research for? Why do we do it, and why should it be funded? And how many strings should be attached to the funding? And how do we measure whether it has all been worthwhile? A good friend of mine, a very respected academic who is one of the global leaders in his discipline, argues from time to time that the only worthwhile research is useless research; once we are subjecting it to an impact assessment, he suggests, we are cheapening it.

All of this is at the heart of the new system to be introduced in Britain for evaluating research, the Research Excellence Framework (REF). This will be used (as a successor to the Research Assessment Exercise) to evaluate a university’s research performance and determine how much general research funding it should receive. One of the key criteria to be used will be ‘impact’. This is explained as follows: ‘significant … recognition will be given where researchers build on excellent research to deliver demonstrable benefits to the economy, society, public policy, culture and quality of life.’ In other words, this will assess whether the research can satisfy my friend at the function. And this has drawn some strong criticism from the academic community, as has been reported in the most recent issue of Times Higher Education. Academics have not been persuaded that the ‘impact’ of their research is always a relevant or fair criterion, not least because it may not be known when a research project is first planned.

I have some sympathy with this resistance. And yet, as society (and other funders) are being asked to provide the resources for research, it is not unreasonable that they should ask what it is for. So maybe we should resist a little less, and just get better at explaining the purpose of research, even research that is at first sight functionally ‘useless’. We are probably no longer in an era where we can answer ‘mind your own business’ to such questions and still hope to get resources, but equally we should be able to explain convincingly that, sometimes, research is justified because it will engage an intellectual agenda and because the pursuit of such an agenda is right for a civilised society, and for a society that wants to train the best minds to do the best they can. And sometimes it is justified because it cures cancer and makes the traffic flow.