Posted tagged ‘university research’

Higher education: measuring success

February 10, 2010

On June 11, 2009, the Oireachtas Public Accounts Committee assessed (amongst other things) the allocation of money to and by Science Foundation Ireland. At the end of the session the Comptroller and Auditor General, Mr John Buckley, reflected briefly on how one might assess whether the state is getting an adequate return from the investment in research carried out in universities. This is how he summarised the issue:

The challenge is to manage commercialisation such that there is a return on the State investment. This applies to the university sector where there is a need to ensure the process of protecting intellectual property rights and licensing and so on are in place and exploiting research outputs to ensure they are optimised. Then there is the level of industry or industrial promotion. The challenge here is to ensure there is a pay-off for State assistance and investment in research and development. The point is that this pay-off will be increased to the degree that it is linked with commercialisation and market awareness.

There is very little in that statement to argue with, but it needs to be said that in political debates in particular there is often an impatience with what some might regard as the rather vague metrics offered as justification for research investment. The politician’s instinct is always to look for jobs, and to look for them in the here and now. Having got used to sums invested in the IDA producing employment in a short time frame, they expect university research to do the same. But as I have noted previously in this blog, the economic impact of research is often in the job creation that it encourages third parties to make in the shorter term, and in the commercialisation impact in the much longer term. But it is dangerous to look for a ‘pay-off’ in any direct way in the short term. To that extent, universities should not encourage such misleading analysis by themselves offering the prospects of jobs in larger numbers directly created by university research, as this promotes a misunderstanding of why research should be funded.

But even when we assess university degree programmes, how do we measure success? The number of students admitted into higher education? The number of successful graduates? The number of access students? The ‘value added’ of improving results (which are sometimes taken as evidence of ‘dumbing down’, perhaps perversely)? Student satisfaction? Foreign direct investment in areas where graduates provide skills?

It seems clear to me that we are living in an age where everything that is funded is assessed in terms of whether the funded activities satisfy key performance indicators and can therefore be seen to provide value for money. This is an understandable approach, but its application to higher education is complex. We probably cannot – and probably should not – avoid this movement, but we need to develop a much clearer approach as to what indicates that investment in higher education has provided an appropriate return to the taxpayer. We may want to say that the return is vital but (literally) immeasurable: for example that it is demonstrated in an enlightened, skilled, intelligent, entrepreneurial, cultured and adaptable population. But that may not satisfy the spirit of the age that wants greater accountability in a more direct sense. So as a sector we should lead in developing an understanding of how what we do can be justified in such a spirit. This may now be one of our most urgent tasks.


Miscreant universities

March 23, 2009

I have a terrible, terrible confession to make: I began my professional life as a banker. There, I have been courageous enough to say it. Admitting to being a banker seems these days to be akin to confessing to a life as a drug dealer or a time share salesman. Right now it seems likely that, having made this confession, I shall be shunned by all right-thinking people. In my defence I can say that I abandoned that path of roguery in the 1970s, so it is all so terribly long ago; and I never claimed, nor did I receive, any bonus while I pursued that particular life.

In any case it hardly matters, because there are days when I cannot help feeling that the profession I chose instead has turned out to be equally shady – I became an academic. Today I bought three Sunday newspapers – two Irish and one British – and each one of them has articles and news items that suggest universities are full of under-performing and over-paid layabouts; and that as institutions they pursue completely useless activities that are a drain on the taxpayer.

One particular bit of commentary that caught my eye was in the Sunday Independent, in an article by the economics editor of the radio station Newstalk 106 Marc Coleman. The article generally was about cronyism in government and the public sector. And thrown into the mix, à propos of nothing as far as I could see, was the following:

Will universities be forced to publish accounts? Will incompetent lecturers be sacked, as incompetent bankers have been? Will recalcitrant ones be forced to do what they’re paid for, ie, lecture their students? Will the Government stop wasting money on useless third-level research and divert the saved funds to creating more primary-school teaching posts?

This is extraordinary stuff, and full of ill-informed innuendo. As far as I am aware, all universities do publish their accounts. Certainly DCU does. It is, I agree, very hard to sack a lecturer, and in many ways it should be, not least because it is necessary to maintain protection of academic freedom; but all universities now have performance management systems. Will lecturers be forced to teach students? They do teach students, and I have hardly ever come across one who refuses (and in those very rare cases where that might happen, we have acted). Useless third level research? As the IDA has recently confirmed, third level research is what is now mainly attracting foreign direct investment; almost every significant recent investment has been in some way linked to research or R&D.

I suspect that the last paragraph sounds defensive. But unless we understand as a country that the excellence of our universities is what will, perhaps more than any other single item, determine our way out of the recession, we are doomed to go back to national mediocrity.

I don’t doubt at all that there are still many things that need to change and many improvements that need to be made in our university sector. But to suggest, as is so frequently done right now, that there is something fundamentally wrong with our third level sector is ludicrous. Except that it is dramatically under-funded, and that the means to produce some of the reforms still needed are denied to the institutions.

Of course universities must operate in a transparent and accountable manner. But that of itself won’t make them world class. And if Marc Coleman would like us to have an economy that warrants an economics editor at Newstalk 106, he might want to take a closer look at what universities really do.

UK Research Assessment – some additional observations

December 18, 2008

From the perspective of the UK universities, the outcome of the Research Assessment Exercise is significant in two different ways. First, the RAE has perhaps the greatest impact on reputation of all metrics used to compile league tables; there is evidence that event student choices are heavily influenced by them. Secondly, they help to determine state funding (though the precise impact of these latest results on money is not yet known). In past RAEs there has been some conflict between these two factors, as institutions struggled to work out how best to maximise both money and reputation in the decisions about how many staff to enter: if you entered more staff and scored well, the financial benefits were highest – but if the gamble failed and you scored badly, the negative impact on reputation could be immense. It was all essentially a gamble.

In previous RAE outcomes, the exercise tended also to reinforce the – formally abolished but still alive and kicking – binary divide between ‘old’ universities and ‘new’ ones (the former polytechnics). This appears to have been undermined by the 2008 RAE results. The lowest place ‘old’ university appears to be the University of Wales at Lampeter, at No 83. The highest placed ‘new’ university is the University of Hertforshire, at No 58. Between these two there is a mix between the old and the new. So while all universities above 58 are old universities, and all below 83 are new ones or various specialist institutions, the boundary has become more fluid. And in some subject areas this fluidity is more pronounced, with new universities reaching the top ranks in some areas. In Ireland of course it is different anyway, with the two ‘new’ universities (in chronological terms), DCU and the University of Limerick, either performing to the same standard as the older ones in research, or sometimes out-performing them.

So what do we make of all this? Are such exercises useful? I have no direct experience of the preparations for the UK’s 2008 RAE, but I was heavily involved in managing subject units for the University of Hull (which in the whole has not done well in 2008) for the previous two RAEs. On the positive side, I found that the RAE had driven home in the academic community the importance of research performance as a way of building up an international reputation – so vital for any national university sector that wants to compete in quality internationally. On the negative side, the focus of the earlier RAE cycles was to combat research non-performance amongst academics, and the result of this tended to be heavy-handed methods to compel performance on the part of staff who, even when they did produce output, were never going to set the world on fire. A lot of very mediocre published output resulted, and arguably truly excellent staff were neglected.

I believe it is right to assess research, and I suspect that we shall be moving in that direction in Ireland. But I also think we should learn from the UK’s RAE and ensure that we avoid the mistakes, and the extraordinary bureaucracy, that accompanied the earlier cycles. If we can do that, we may be able to develop our own system in a way that genuinely adds transparency to the system.