Rank confusion

For the last few years the late summer and early autumn has been the season for university world rankings. This season kicks off in August with the Academic Ranking of World Universities (published by Shanghai Jiao Tong University), and a month or so later we have two sets of  World University Rankings (one published by Quacquarelli Symonds, and the other by Times Higher Education).

There are also others that, at least for now, we can disregard; and there is the EU’s U-Multirank project, which describes itself as a ‘multi-dimensional ranking of higher education institutions’, and which says that its first (2014) publication will look like this:

‘U-Multirank is a new multidimensional, user-driven approach to international ranking of higher education institutions. The dimensions it includes are teaching and learning, research, knowledge transfer, international orientation and regional engagement. Based on empirical data U-Multirank will compare institutions with similar institutional profiles and allow users to develop personalised rankings by selecting indicators in terms of their own preferences.’

The purpose appears to be the production of a variable ranking system that users program to reflect their own priorities; meaning perhaps that most institutions will find a way of extracting from this a league table that has them in an attractive position.

But back to the existing autumn show of rankings. What do they tell us? One of the problems with them is that they seem to tell very different stories. All of them agree on one thing: that US universities still clearly lead the field, followed by British institutions. But when you get to the detail, there is little agreement. Each has a different leading university. The California Institute of Technology (Caltech) is global number 1 in one league table, number 6 in another, and number 10 in the third one. And when you get just a little further down the list, the variations are much greater. And as the Irish universities have shown this year, in one league table they can go up significantly while, in the same year, dropping like a stone in another.

So are league tables really just unreliable? Are the U-Multirank folks right, and the best thing is for you and me to compile our own rankings?

The point is that, like it or not, we are in the age of rankings. People want to have an objective view of quality and merit, and they will go for something that looks as if it offers that.  Even when we criticise the league tables, as at some point we all do, we still play the game they set us. And in truth, that’s what we have to do. So then, choose your favourite league table, and see how you can use it to best effect. But don’t be mesmerised by it, and for goodness sake don’t construct your strategy around it.

Advertisement

Are our universities really destined for long term decline?

The latest university world rankings have prompted another round of questions about the future global distribution of higher education excellence and strength. The Times Higher Education 2012-13 rankings have seen a little slippage in the position of some universities in the western hemisphere, with Ireland and Scotland both experiencing this phenomenon. Ireland no longer has any university in the top 100 universities globally; Scotland still has one university (Edinburgh) in the top 100, at number 32 (and rising), but other Scottish institutions have fallen, in some cases significantly.

Speaking more generally about UK universities, the Times Higher‘s rankings editor, Phil Baty, remarked:

‘Outside the golden triangle of London, Oxford and Cambridge, England’s world-class universities face a collapse into global mediocrity, while investment in top research universities in Asia is starting to pay off.’

While there is indeed some drift, the prediction of a British collapse into mediocrity may be a bit premature. It is absolutely right that some of the emerging countries have made major investments in higher education, as one would expect. But this has not produced any instant challenge for global leadership. The Times Higher top 20 contains only one university not in the United States or the UK, and it is in Switzerland. The top Asian universities (from Japan and Singapore, respectively) come in at number 27 and 29. The top university from a BRIC country (if you exclude Hong Kong) is Peking University, at number 46, and the only other institution from that grouping in the top 100 is also from China, at number 52. And while there are some slight changes from last year in those positions, they are actually not hugely significant. Peking University rose by three places.

Furthermore, the success of universities in China, Singapore, Korea and Japan – and only a very small number of the thousands of Asian universities make it into any rankings at all – is largely based on these institutions ‘westernising’ their educational and research methods and pouring in money. But that hasn’t just started last year; it has been a phenomenon of the past decade, and while the results are certainly there, they are not startling.

It is obvious enough that as some countries make a transition into a more developed economic state, their universities will benefit from more investment and higher levels of ambition. But actually, it is rather remarkable that this has not had a much greater impact on the rankings. Then again, this is not to say that there aren’t issues here to be addressed. The uncertainty about university funding in these islands has certainly had an impact, but so have other factors, including the inconsistencies and peculiarities of migration policies as they affect student movement, and the trend for major companies to seek university links away from the more traditional set.

What may be much more interesting, however, is this: there may be a hint in the rankings that the university of the future is no longer necessarily the ancient, classical, blue-skies-research institution. The new leader, as exemplified in the world’s number 1 university, the California Institute of Technology, may be a more focused, networked and translational university. Apart from Caltech, other institutions that also reflect this profile have climbed up the rankings. As we try to work out what the role of higher education is to be in the future, that may be the more interesting trend.

Times Higher rankings

The world university rankings issued by Times Higher Education – which over recent years have been accepted as the most authoritative international league table – are published today. The global number 1 spot has now been taken by the California Institute of Technology (Caltech), followed by Harvard, Stanford and Oxford. The top 20 contain 14 American universities, 4 British, 1 Swiss and 1 Canadian. There is no sign in the rankings of any visible decline of US institutions.

The highest ranked non-British EU university is the Karolinska Institute from Sweden at number 32, followed by the Ludwig-Maximilians-Universität in Munich at 45. The University of Edinburgh leads the Scottish universities at number 36. Ireland’s highest rated university is Trinity College Dublin at number 117, followed by University College Dublin at 159.

So, what makes a great university?

In anticipation of the publication shortly of the Times Higher Education World University Rankings, the THE has invited Twitter followers to declare succinctly what makes a university great. The responses, which I believe are mainly from younger and mid-career academics, are interesting. Here’s a sample:

• A place about minds, not behinds. Now try + measure that!
• Being called The University of Sheffield, of course 😉
• Cheese.
• A great University is the community of great teachers and scholars.
• A great univ = place that is intellectually risk tolerant
• The students.
• GreatUni=A Uni where professors hav balance between passion for subject, ability to teach it 2 others & facilities to do so.
• A great univ = place that is intellectually risk tolerant.
• 1 that stimulates creativity, tantalizes intellect thus inspiring students to author theculture of their time.

In many ways our ability to discern what constitutes ‘greatness’ in higher education is becoming more and more important. As universities become more vulnerable to funding cuts and bureaucratisation, their ability to persuade the government (and the public) that what they do matters and needs to be protected will become vital. This is not just a responsibility of university managers, but of the whole academic community.

If there is to be a persuasive case, it needs to be less general (and banal) than some of the attempts above. The academy needs to be able to respond better to people’s expectations of higher education, and this means that we must understand those expectations better. A great university is more than a place with great and even tolerant minds: it is a place that does something with these. It is the difference we makes that makes us great.

Taking charge of your own university rankings

Whenever I raise the topic of university rankings, I always get readers who, either in comments made here or in emails sent offline, will suggest that I really shouldn’t be paying so much attention to them or encouraging their authors. I know very well that many academics are very sceptical about league tables and don’t believe that they reflect any sort of reality; or they suspect that rankings prompt inappropriate behaviour by university managers, or in some other way undermine academic integrity.

In reality, however, league tables are part of the landscape, and this is so in part because those who want to enter into any kind of relationship with universities – whether as students or as faculty or as business partners or as donors – take them seriously and want to have them as a guide. We may wish that it were otherwise, but it isn’t. This being so, we need to engage with them, and in that way help to ensure that they are reasonable and accurate and transparent. So for example, the transformation over the past year or two of the Times Higher Education world rankings owes a lot to academic interaction with the journal and with the company they selected to manage the project.

The best known world rankings – those run by Times Higher Education and by Shanghai Jiao Tong University – have one important thing in common: the global top 10 universities are exclusively American and British. This is tolerated by Asian institutions that believe they are rising up the tables and are biding their time, but it disturbs the European Union and its member states.  In both rankings the top non-British EU university only comes in at number 39 (French in each table, but not the same university).

Because of this the EU has set out to design its own rankings, to be known as U-Multirank. The thinking behind this is that the established league tables are too much focused on research outputs, and in particular on science research; they neglect teaching and don’t encourage diversity of mission, and they drive universities into strategies that they don’t have the means to deliver. So the new rankings are to be weighted differently, so that the resulting table would be more balanced; and moreover they are to allow users to design and weight their own criteria, so that students (say) can create their own league table that more accurately reflects the strengths they are looking for in considering universities.

Can this work? In my view, no – probably not. Rankings are not really meant to provide a method of institutional profiling, but rather are designed to set out a kind of reputational gold standard. They are not the answer to the question ‘what kind of institution is this?’ – rather, they answer the question ‘what does the world think of this institution?’ This may not be a scientific answer, or else all rankings would give us the same results, but it is an attempt at standardising external evaluation. Also, too many people will think of U-Multirank as an attempt to support the somewhat lesser known European universities and design the rules to suit them.

Still, if you’re interested, the U-Multirank project is coming to the end of a feasibility evaluation and, if this supports the idea (as it will), it will be rolled out some time over the next year or two. It will be interesting to see whether it attracts support. I suspect that it will not displace the pre-eminence of Times Higher.

Assessing the student experience

Once a year the journal Times Higher Education conducts and then publishes a survey of students in British universities, recording the criteria that students believe are important in assessing  the quality of their experience and the performance of their institution in relation to these. The results are then compiled in the form of a league table, the most recent of which has just been published by the journal.

As with every survey and every league table, it is wise to  enter a health warning or two. There will always be a margin of error, and in the case of each institution only a relatively small sample of the student body is involved. Also, a subjective experience on the part of a student may not necessarily translate into an objective statement of quality or the lack of it. All that having been said, the survey and the results from it can be used to gather some insights.

One interesting observation would have to be that while the results do not absolutely mirror university rankings produced in other league tables, they are not wholly incompatible either. So interestingly, research intensive universities on the whole appear to offer a better student experience than those that focus largely on teaching. Post-1992 universities – with a couple of exceptions, including the one I am about to join, RGU in Aberdeen (one of only two to be in the top 40, at 25) – do not do as well as older ones. Oxford and Cambridge, while showing up in the top 10, are not right at the top (they are at 6 and 4, respectively, out of a total of 113): that distinction goes to Loughborough University, for the second year running. The highest placed Scottish university is the University of Dundee (number 5), the highest placed Welsh institution is Bangor University (number 14), now led by ex Maynooth president John Hughes. Overall, the availability of good facilities, including good sports facilities, makes an impression on students, though ‘good teaching, enthusiastic staff and a well-structured course’ are seen as the most important.

In England surveys such as this may become an important selection tool for student applicants, who may balance the cost of degree programmes in the new tuition fee environment against student satisfaction recorded here. It is in any case right that universities should take seriously the impression they make on their students, and the level of satisfaction that these students feel. It is interesting also that students do not view what they find that differently from how it might be assessed in other processes. And finally, it is fairly clear and probably unsurprising that a well resourced institution will seem more attractive to students, as much as to anyone else.

And now, the Times Higher Education rankings

Well, I did say that it was the rankings season. And here’s another, and perhaps the one most likely to be seen as definitive: the World University Rankings issued by the journal Times Higher Education. It’s hard to know whether to describe these as new or the latest in a series. Times Higher have been publishing rankings for a few years now, but previously these were prepared in collaboration with Quacquarelli Symonds. In 2009 they parted company with QS, and this year the Times Higher rankings have been prepared in collaboration with Thomson Reuters. The methodology used is different, and so it may be better to see this as a completely new exercise, rather than a new edition of the old one. In any case, QS have continued with their own rankings, as we noted last week.

Before getting to the actual outcomes, the following rather eccentric element should be noted. It had been announced before today that the Times Higher rankings would only list the top 200 global universities. And so indeed it is, and you can find the table here. However, they are also selling an iPhone app through iTunes, and with this (and only this) you can access the top 400. This allows us to list more of the Irish universities, because only two appear in the top 200, while a further three are in the 200-400 range.

Unlike the QS rankings last week, Times Higher has the United States leading the field. The top university is Harvard, and it is followed by four other US universities (though not Yale, which comes in at number 10). The top non-US universities are Cambridge and Oxford, coming in together at number 6=. The top non-US/UK university is the Swiss Federal Institute of Technology Zürich at number 15. The top non-UK European Union university is France’s École Polytechnic at number 39. Overall, while there are a good few European Universities in the rankings, they are not challenging the US/UK dominance. Some Asian universities make make an appearance, as do Canadian and Australian ones.

And what about the Irish institutions? Here is the table:

TCD    76
UCD   94
UCC   243
NUI Galway  299
DCU    313
DIT   347

NUI Maynooth and the University of Limerick are not in the top 400.

So while Trinity College still leads the field, UCD is catching up. No Irish university has made it into the top 50. And as I have noted before, these positions are likely to slip further as the funding cuts start to bite even more.

And what’s next? The Sunday Times league table for Irish universities is due out next weekend.

Global rankings: the QS version

At midnight Quacquarelli Symonds (QS) released their university world rankings. Just to recap, QS were until last year the partners of Times Higher Education in their world rankings. Times Higher then parted company with QS and entered into a partnership with Thomson Reuters – their rankings are due out next week on September 16, and they yesterday released details of the criteria they have used.

But back to QS. The first thing to note is that their international number 1 university is the University of Cambridge (UK), which has moved ahead of Harvard and thus occupy the top spot for the first time. Indeed it is the first time in any league table that the highest position has not gone to an American university. However, in the top 20 the general distribution has not changed: the US is there 13 times, the UK 4 times, and the remaining three slots go to Switzerland, Canada and Australia. The highest placed continental European university is ETH Zürich at number 18. After that, European universities are fairly well represented in the top 50, but Germany doesn’t make an appearance until number 51 (Heidelberg).

The latter is just one place above the highest placed Irish university, Trinity College, which has dropped 9 places to number 52. There is also a drop for UCD, down from 89 last year to 114 in 2010. On the other hand, UCC has improved its position and has entered the top 200 for the first time at number 184.

It is hard to know for sure whether we are witnessing a trend, but the signs are that Ireland’s universities are, in terms of global rankings, in decline. That this is so is not unexpected, and I suspect that when the details are analysed we will find that one of the key factors will be the student-staff ratio. For two years now student numbers have grown, while due to government rules in the ’employment control framework’ staff numbers have dropped. The necessary impact of this is a decline in the international standing of Irish universities, and the consequences could be serious for Ireland in its plans for a ‘smart economy’.

During the week in which Ireland has also been found by the OECD to be under-investing in education more generally, we are facing a crisis that needs to be addressed positively and urgently. No matter how unpleasant this may seem to some politicians and some others, we need to grasp the nettle of university funding – and at least from my perspective I son’t see how we can succeed in this while we rule out tuition fees; the taxpayer simply does not have the resources to solve this problem on their own.

If we want to arrest an increasingly apparent and potentially long-term decline of our education system, and with it the erosion of any ambitions to be a knowledge society and economy, we had better act now.

The month of league tables

In the course of this month (September) two league tables will be published that, whatever we may think of them, will determine some people’s views of Irish higher education. They are the Times Higher Education world rankings (now compiled with Thomson Reuters), which will be issued on September 16, and the Sunday Times University Guide, which is typically published around September 20. Just to complicate the picture, Quacquarelli Symonds Limited  (QS), who until 2009 were the Times Higher partners for the world rankings, will continue to publish their own league table, expected to be out in October.

What can we expect to see this year? The Sunday Times rankings are extremely unpredictable; the only university to have maintained the same position, as Ireland’s number 1, in every year is Trinity College. All other universities have jumped round the numbers; DCU for example has over the years occupied every position except top and bottom. But if you look at the precise points used, you will see that the Irish universities are not far apart, which may explain the erratic positioning.

The Times Higher rankings may look completely different this year. For a start, they will only publish the list up to number 200. On present performance, that would mean that only three universities (TCD, UCD and UCC) will appear. But in fact they have also completely changed the performance indicators used and their weightings, with more emphasis on teaching than before and less on stakeholder and peer opinions; over 60 per cent of the score will however derive in one way or another from research. In the light of the tricky funding issues facing Irish higher education and the quality impact this has, we must in any case expect Ireland’s universities to begin slipping in these rankings, though it is possible that this will not become fully visible until 2011.

The QS rankings may be the ones that work best for Irish universities, though at this stage it is difficult to know what standing they will have globally in the light of the parting of ways between QS and Times Higher.

One way or another, we will hear a lot more about league tables over the weeks ahead.

Clinical medicine research league

If you were asked to guess which countries are the most influential in clinical medicine research, measured in terms of published output and citations, which constitute your top 5? Well, the journal Times Higher Education has just published rankings based on data collected between 2000 and 2010 by Thomson Reuters, the journal’s new partners in the global rankings exercise. Before I looked, I guessed the top 5 would be, in this order: the United States, the UK, Germany, Japan and France. I had reasons for guessing this way.

But how wrong I was. According to the table, the top 5 are Finland, Denmark, the Netherlands, Belgium and Scotland. Really? Belgium beats the US? Well actually no, they don’t. I mean, at all. What the compilers have done is to rank the countries in terms of citations per paper. So Finland, with 21,085 papers over the decade, has 386,878 citations, or 18.35 citations per paper. The United States however, weighing in at number 7, only managed 17.21 citations per paper. But hang on, that was based on 692,884 papers and 11,927,881 citations – and so I am wondering whether you could really say that Finland (or Belgium) had influenced medical research more than the United States.

Clearly academics from Finland have been doing a great job in attracting attention for their published work. But on this particular methodology, if you wanted to be the most ‘influential’ country you would choose your very best, world leading academic and enter him or her, and sack all the others so they would not pollute the data. I am mentioning this not in order to make some smart little point, but because if league tables are to tell us something we need to be careful to work out exactly what that is.

Actually, my own guess turned out to be 100 per cent correct, if you equate influence with the actual numerical output. And so I hope that as the next generation of rankings is revealed shortly the data will be genuinely useful and will be used responsibly. The Times Higher has an honorable record on this regard, and I hope that this will be re-affirmed.