Rank confusion
For the last few years the late summer and early autumn has been the season for university world rankings. This season kicks off in August with the Academic Ranking of World Universities (published by Shanghai Jiao Tong University), and a month or so later we have two sets of World University Rankings (one published by Quacquarelli Symonds, and the other by Times Higher Education).
There are also others that, at least for now, we can disregard; and there is the EU’s U-Multirank project, which describes itself as a ‘multi-dimensional ranking of higher education institutions’, and which says that its first (2014) publication will look like this:
‘U-Multirank is a new multidimensional, user-driven approach to international ranking of higher education institutions. The dimensions it includes are teaching and learning, research, knowledge transfer, international orientation and regional engagement. Based on empirical data U-Multirank will compare institutions with similar institutional profiles and allow users to develop personalised rankings by selecting indicators in terms of their own preferences.’
The purpose appears to be the production of a variable ranking system that users program to reflect their own priorities; meaning perhaps that most institutions will find a way of extracting from this a league table that has them in an attractive position.
But back to the existing autumn show of rankings. What do they tell us? One of the problems with them is that they seem to tell very different stories. All of them agree on one thing: that US universities still clearly lead the field, followed by British institutions. But when you get to the detail, there is little agreement. Each has a different leading university. The California Institute of Technology (Caltech) is global number 1 in one league table, number 6 in another, and number 10 in the third one. And when you get just a little further down the list, the variations are much greater. And as the Irish universities have shown this year, in one league table they can go up significantly while, in the same year, dropping like a stone in another.
So are league tables really just unreliable? Are the U-Multirank folks right, and the best thing is for you and me to compile our own rankings?
The point is that, like it or not, we are in the age of rankings. People want to have an objective view of quality and merit, and they will go for something that looks as if it offers that. Even when we criticise the league tables, as at some point we all do, we still play the game they set us. And in truth, that’s what we have to do. So then, choose your favourite league table, and see how you can use it to best effect. But don’t be mesmerised by it, and for goodness sake don’t construct your strategy around it.
Explore posts in the same categories: higher education, universityTags: Academic Ranking of World Universities, Quacquarelli Symonds, Times Higher Education, university rankings
You can comment below, or link to this permanent URL from your own site.
University Blog 
October 8, 2013 at 7:42 am
When sporting units are ranked, individuals or teams, it is with respect to an objective measure, like points scored or games won. There is a congruence between the objective measure in the ranking and the nature of the sport.
Why are all of administrators, politicians, academics and generalist spectators susceptible to the illusion that universities may be meaningfully ranked with respect to a classification scheme that is not objective and aggregates many factors not all congruent with institutional goals? It must relate to the illusion that universities share a singular purpose that provides a meaningful metric.
A university may have a critical mass of researchers who wish to cure dementia and another group within it working on solutions to Hilbert’s problems, among any number of other topics. Universities differ in their spread of researchers and overall goals. Once the researchers reach one goal, they move to another. Publication is merely an archive of the way of achieving goals, albeit an important mode of sharing the knowledge acquired. Certainly, generalist spectators care more that cures exist than that papers about them appear in one journal over another. However, publication statistics, whether publication counts or citation counts, serve as a proxy for measuring achievement of research goals and their impact, rather than as a direct measurement of the achievements themselves. In sports, this would be like measuring the quality of a scored point in relation to the stadium cheers or on-field post-score antics of the players.
Different disciplines are not directly comparable in publication frequency or scale, and therefore it makes little sense to aggregate all publications for a university and then rank it, rather than aggregating rank orderings for each discipline (if even that is meaningful). However, there is little value in ranking universities on the basis of their aggregate rank ordering by discipline since not every university addresses every discipline.
Moreover, this ignores the overall goals of a university, which may include how it fits into the socio-economic structures of its locale, or any number of other aims.
These rankings seem to have the goal of letting spectators know which universities are “the best” in the greatest number of disciplines. However, their variability suggests that there is no agreement on the factors that should contribute to a ranking. If one were to attempt to rank the rankers, one would quickly run across the argument that each ranking is measuring different things and should be taken on its own terms. If the university rankers are each to be taken in their own terms, why not the universities?
October 8, 2013 at 11:49 am
I’m not all that certain what question these lists are the answer. I’ve looked up History and in Ireland TCD is in the first 100 with NUI,Galway in the segment between 100-150 with nary a sign of the rest. So, from these lists there is no point in the study of History Classics & Archaeology at UCD, UCC or NUI,M. If this translates into the rest of them it makes the whole utterly meaningless.
October 8, 2013 at 12:07 pm
“The point is that, like it or not, we are in the age of rankings. People want to have an objective view of quality and merit, and they will go for something that looks as if it offers that. Even when we criticise the league tables, as at some point we all do, we still play the game they set us. And in truth, that’s what we have to do.”
But do we really *have to*? Play the game knowing that behind the patina of rigorous assessment the rules and the outcome are fallacious? Aren’t we complicit (and have been for a long time) in a mechanism of progressive erosion of academic values which has gifted us with the madness of the REF
http://coastsofbohemia.com/2013/10/05/update-from-wonderland-the-lancaster-history-ref-farce-goes-on/
a rampant commodification of education and a devaluation of anything which has to do with pedagogy?
In my childish naivety I cannot accept that, I think that the compromise has gone too far, that there is a need for a radical change of culture, one that does not supinely accept to ‘play the game’ someone else has devised but is actively involved in the production of the game itself..
A small aside on the Quacquarelli Symonds (QS), data provider of the rankings http://en.wikipedia.org/wiki/Quacquarelli_Symonds
The word ‘Quacquarelli’ in Italian conjures up images of silly, useless chatter…
October 8, 2013 at 12:16 pm
At the end of the day whether we decide to use them is besides the point as overseas students will. They count for a lot and Irish universities should be tapping to them more. Foreign academics and business leaders will use them too. So to ignore them will work towards our detriment.
October 8, 2013 at 12:36 pm
As the author of the following piece rightly argues: ‘Predicting the exact impact that rankings have on the international student market is fraught’….in fact “we should be wary of arguing too strongly that a change in rankings alone will dramatically affect international (or domestic) student preferences. Many factors influence why students choose a particular university or country over another.”
http://the-scan.com/2013/10/03/the-ritual-dance-of-university-rankings/
October 8, 2013 at 5:09 pm
Bealoideas, I don’t believe that they do. In my discipline, we have a lot of overseas graduate students this year. None of them looked at rankings..we asked them, they didn’t….I think university academics and administrators like to think that overseas students, foreign academics and business leaders use the rankings, but I bet they don’t, in fact…
October 8, 2013 at 1:15 pm
“That’s what we have to do.”
No, it isn’t: http://www.theatlantic.com/magazine/archive/2005/11/is-there-life-after-rankings/304308/
October 8, 2013 at 4:10 pm
Read Malcolm Gladwell’s article on college rankings – he demonstrates it us all just about greed, mendacity and academic stupidity. At some stage, we’ll all just cop ourselves on…
http://www.newyorker.com/reporting/2011/02/14/110214fa_fact_gladwell?currentPage=all
October 9, 2013 at 10:10 am
Fresh from Twitter the latest frontier in university rankings, the mentions in new media!
http://chronicle.com/article/The-New-Rankings-Frontier-/142197/?key=SGp7KVJvZSoVYC4zYmxBbjYAYHFrZB8qangcansjblBcGQ%3D%3D
We should not be surprised though since this is not only the age of rankings, as mentioned in the post, but the one of the ‘Quantified Self’ as well http://en.wikipedia.org/wiki/Quantified_Self
October 10, 2013 at 9:19 pm
I wonder how long it will be before Channel 4 includes university rankings along with their ‘Top 100 Best Films’ and ‘Top 100 Toys of the 80s’ shows? I can already hear the scathing remarks by B-list comedians. Or maybe, instead of continuing their quest to divide all forms of matter into two groups (those that give you cancer and those that prevent cancer) the Daily Mail should go ahead and start their own ranking of universities. EU students … -10 points. Non-EU foreign students ‘stealing’ UK places … -20 points. Maybe the multinationals will just rate the interviews with graduates and compile a list of ‘universities supplying people we employed’ and ‘universities supplying people we laughed at’. Sure to get some decent debate going …
I wonder what would happen to the THE list if you divided the mark awarded to the institution by the funding. Surely that would give you a real picture of institutional success?
October 11, 2013 at 10:16 am
If I may chip in to the discussion, no matter how confusing the overall rankings may get, there is one that’s strict, highly specialised and reliable.
Moreover, it measures things that actually matter on a global scale – it’s the Green League. Prepared by specialists in People & Planet, published annually by Guardian it’s a serious indicator of UK Universities’ sustainable performance.
What’s worrying, despite topping all other charts, in this one, RGU hits the very bottom of a ranking.
Here is a link to this year’s full account:
http://www.theguardian.com/news/datablog/2013/jun/10/green-league-university-list
http://peopleandplanet.org/green-league-2013/tables
It is not for PR, neither for the number of students recruited – but I would say we really have to pay a closer attention to this report, to be an organisation with a human face, who cares about the environment it operates in because that’s just the right thing to do.
Although I noticed a few areas where we have already improved and will definitely score points in next year’s audit, there is still so many things to work on! And these are definitely worth constructing strategy around.
October 12, 2013 at 10:50 am
Interesting discussion ridiculing ranking from academics, whose universities I guess, find it difficult to be spotted within the top 150 of either rankings. Any university which finds in place in the top 30 of these world rankings is good. That is undeniable.
As for overseas students attending these low ranking UK/Irish universities, they want just to get into the West so that they can escape their country, and work as work visa route has become difficult. Otherwise, if they value rankings, many many students from India and China would stay home and attend their best universities. Many do, and others in the above two countries and from other non-EU countries, who cannot find places even in their country’s low ranking universities because, shall we say of somewhat modest academic background, are welcome in many of the above universities; sure, they will not say they look for rankings, as they want simply to get into the country as non-EU citizens!
“But don’t be mesmerised by it, and for goodness sake don’t construct your strategy around it”
Amusing indeed, considering that my son and his friends did just that they have now very successful careers, and others I know too!
October 12, 2013 at 11:09 am
Wow! Successful careers! All world knowledge has finally culminated in its ultimate outcome and justification: somebody has a career! Bravo!
October 12, 2013 at 1:53 pm
My guess about academics in low ranking universities is justified.after all by this comment!
October 12, 2013 at 1:56 pm
Dunning-Kruger strikes again!
October 12, 2013 at 1:58 pm
As implied above, no use in arguing with you at all.
October 12, 2013 at 12:01 pm
“`But don’t be mesmerised by it, and for goodness sake don’t construct your strategy around it’
Amusing indeed, considering that my son and his friends did just that they have now very successful careers….”
Do you mean to suggest that your son would not have a successful career without having adopted that strategy of university selection?
October 12, 2013 at 1:57 pm
YES, in his, his friends’.cases, in the cases of so many who selected high ranking universities and were successful. High ranking universities hence are in such great demand. That is the truth, you may not agree with it, and that is fine with me.
October 12, 2013 at 2:36 pm
As I gather, the top UK university for graduate employment is Robert Gordon university in Aberdeen, beating all others including the Russell Group members. That seems to run counter to your point.
October 12, 2013 at 2:44 pm
No, in Eddie’s world “employed ignoramus” is an oxymoron.
October 12, 2013 at 2:57 pm
Well, dig deep, you will know the full facts.
This is from THE : “Trinity College Dublin’s drop in the Times Higher Education (THE) World University Rankings 2013-14 should be a wake-up call for Ireland’s higher education sector where more sustained investment is needed to drive societal and economic renewal, according to Trinity’s Dean of Research, Professor Vinny Cahill. Trinity’s ranking dropped from 110 last year to 129 this year, according to Times Higher Education Rankings.
This is from Vinny who I know and respect as a good researcher in computing and some one who pioneered work in Middleware..
October 12, 2013 at 8:20 pm
“YES, in his, his friends’.cases, in the cases of so many who selected high ranking universities and were successful.”
The question was not whether you think your son is successful following a career at a high ranking university; rather, the question was whether you think your son would not have a successful career without having adopted the strategy he did.
Your answer is “YES”: this implies that you think that his genes and upbringing did not equip him for any alternative path to success. This is evidently a low estimation of his native endowment and ability.
Is this really what you mean to imply?
October 13, 2013 at 6:47 am
Best universities bring out the best in students, that was what I implied.
October 13, 2013 at 12:23 pm
I’m guessing you’ve never actually set foot in a university.
October 12, 2013 at 3:52 pm
As expected, my recent post is removed. It does prove the point I made there!!
October 12, 2013 at 4:39 pm