Quality in higher education

It would probably not be hard to get a consensus around the proposition that universities should aim for high quality in both their teaching and their research. But it is much harder to identify what quality actually is, how it can be recognised and how it can be measured. This is illustrated by the fact that some of the key policy documents on quality assurance for universities go into great detail about the process by which quality should be assured without ever once saying what actually constitutes ‘quality’. For example, the Standards and Guidelines for Quality Assurance in the European Higher Education Area issued by the European Association for Quality Assurance in Higher Education in 2005 makes no attempt to define, describe or identify quality criteria. The Irish Universities Quality Board (IUQB) on its website has a page of ‘useful definitions in quality assurance‘, but the term ‘quality’ is not defined there. On its website, the British Quality Assurance Agency (QAA) does purport to define ‘academic quality’, thus:

‘A comprehensive term referring to how, and how well, institutions manage teaching and learning opportunities to help students progress and succeed.’

But is that actually a definition of anything?

It seems strange that a whole industry dedicated to monitoring and improving quality seems to have no advice to offer as to what ‘quality’ actually is. I once sat next to a senior academic at a dinner; he had been involved in the QAA’s quality assurance process for ten years. When I asked him what ‘quality’ meant, he chided me for asking an ‘irrelevant’ question. He suggested that the institutions themselves could determine what constituted quality, and the task for people like him was to see whether they lived up to their aims. That seems sensible at one level, except that if it were that simple then institutions could guarantee superb quality simply by setting themselves very modest ambitions, and then meeting or exceeding them.

However, because nobody has anything much to say as to what quality is, the temptation is to get out of this dilemma by focusing entirely on process: we cannot say whether what you teach is good quality, but we can ask whether you have followed the 20 prescribed steps when you developed the programme and are counting the answers students have given in the feedback questionnaires. And on the whole, that is how a fair amount of quality assurance has been conducted.

The problem with this is that if your excellence is – publicly – going to be measured on the basis of how satisfied people are with your processes, then you had better have comprehensive processes and stringent monitoring; and if that’s your concept of quality, then you had better steer clear of innovation, because innovation (like entrepreneurship) is risky and may sometimes fail or not enthuse the users at first. It is much safer to stay with your existing offerings and just make sure that all the paperwork is in good order.

We have therefore come to accept that quality assurance is about process, whereas it should be obvious that quality is about content and intellectual innovation. If we are serious about having a high quality higher education system, then we have to start asking questions about content, avoiding the risk of suggesting that there is one standard way of measuring this.

Advertisement

How important is teaching to the academy?

As the saying goes, what gets measured gets done. So one of the curious aspects of modern higher education is that what most would still regard as its core activity, teaching, does not find its way into most of the formal metrics used to assess institutional performance. True, things like the student-staff ratio (which, mind you, is not of as much value as you might think) are used, and quality assurance provides some insights. But on the whole the assessment of teaching is fixated on process rather than content or standards.

This gap has all sorts of consequences. League tables and rankings, while suggesting all sorts of other criteria, usually end up assessing institutions on the basis of their research outputs. Career progression, even in research non-intensive institutions, has a tendency to be research-driven. Institutional (and indeed individual) reputations are built on research. You get the picture, there is a pattern.

The question that is sometimes asked in relation to this is whether a focus on research helps or damages the university’s teaching, and in particular the student experience. There are mixed views on this, but a recent Australian survey has suggested that in the most research-intensive institutions students tend to find employment more easily after graduation, but have more negative views of their learning experience.

Of course in a properly ordered system teaching and research should not be seen as rival activities. Excellent staff research provides a more informed environment for students, always provided that the leading researchers are also engaged as teachers. But most universities have not managed to convey this relationship in practice. It is now sometimes suggested that this can only be effectively remedied if teaching is subjected to peer assessment with numerical scores. At any rate unless there is some attempt to rate teaching, it will be seen as the poor relation; and that is a situation that cannot really be allowed to continue.

Qualifying higher education

No educator likes this kind of talk, but if we were to accept for a moment and for the sake of argument that universities are selling something, what is it? Although it is very hard to identify the real nature of the transaction or exchange, we do know that we get money (whether in sufficient quantity or otherwise), and we know that we undertake an activity connected with that payment. But if someone is buying something, who is that someone and what are they buying?

There are several possible answers to this question, but let us now assume that the purchaser is the student (which is increasingly true in a number of countries). Would the student believe that his or her tuition fee is paying for an education, or would they maybe say it is for a qualification? In  other words, if the university invited students to take a programme of study but declined to offer any formal qualification at the end, would the students still come? Or at least, would so many of them?

In some ways the educational bureaucracy has long made the assumption that the qualification is what the bargain is all about, ever since the quality assurance movement got under way. That movement assumes that the ‘quality question’ of higher education is whether the educational process, leading to a degree or diploma, was correctly administered and is consistent across the higher education sector. But that is not a question about pedagogy (or arguably even about educational standards): it is a question about the consistency, transparency and efficiency of outputs.

I am not suggesting that quality assurance mechanisms are bad (though when badly administered, they are). Rather, I am reminding myself (and others if they are interested) that we have become rightly concerned to monitor how educational institutions fulfil their mission, but that we ask surprisingly few questions about the real nature of learning and what it entails. We are sucked into process, but not into experience. As a result the gold standard of higher education is the exit qualification, and graduates can take that to their new employers and often need not worry whether they will be able to explain anything they have learned.

It is time to look again at education in a context other than its formal elements of delivery and assessment. It is time for us to be clearer about what we want education to do. And I really would prefer not to be told that it’s just there for its own sake.

Lighting up the external examiner system

Observers of higher education in these islands may believe that quality assurance processes introduced over the past decade or two are the key guarantors of quality and standards in universities.  Without wanting to get into a debate on whether this is true or not, it is worth observing that a much older and on the whole robust framework for securing standards has been the external examiner system.

Under this system all examination scripts and assessed essays and projects are open to external scrutiny by an examiner appointed from another institution, and often from another country. These individuals consider the overall results, assess borderline performances by individual students and do spot checks across the whole range of results. Their task is to confirm that the overall standards adopted by the internal examiners are in line with the norm across the sector and that individual students have been fairly and appropriately assessed. They are also typically asked to comment on the general approach of the department and the suitability of the syllabus adopted. Generally external examiners receive a small fee, but it would be fair to say that this fee doesn’t even come close to rewarding them for time and effort. External examining is highly demanding and very pressurised, as the tasks set out above typically have to be performed over a very short space of time.

Lecturers and the universities more generally get much advice and support from external examiner reports. To students on the other hand these reports and the associated activities are almost entirely hidden, and therefore they do not have an opportunity to benefit from the confidence building aspects of the system. For this reason it has now been proposed that external examiner reports should be available for students to see. While some fear that this might cause the examiners to be less frank and forthright in their comments, in the end this is an argument against almost all freedom of information. It seems to me that the proposal is right, and that the reports should be openly available (perhaps without those comments that are ad personam evaluations of individual students). Doing so will also help make the case for continuing (and maybe better) support for this vital aspect of higher education.

Measuring higher education quality

Ever since the quality of higher education started to become a matter of concern in society, people have been struggling with the idea of how, if at all, quality could be measured. There has tended to be an assumption that quality assurance could only be real if there were metrics involved, because without them there would be no sense of objectivity, and furthermore there could be no meaningful targets for the achievement of quality. However, the metrics have tended to focus strongly on inputs (the student-to-faculty ratio being a typical example), not least because outputs (principally degree results) have often been questioned in terms of their integrity.

Of course it is not just the quality assurance process that comes up against this, it arises also when various bodies or media attempt to put together league tables (which, to carry weight, have to be based on an assumption of comparable quality). This ambivalence of all this has recently been illustrated by the US journal Chronicle of Higher Education, showing the really wildly different criteria that are used in various rankings.

But in the end rankings are indicative rather than definitive. However, formal quality assurance processes have to convey a sense of confidence in the objectivity of their use of metrics or other information. Can this be done?

The Irish university sector has fared better than most, because the quality processes of the Irish Universities Quality Board have taken these issues into account and have used a negotiated framework aimed at supporting improvement rather than condemning failure. This gain could be easily lost. It must be hoped that the new framework of the proposed unified agency will work constructively with what has been achieved to date. The important thing about higher education quality is not that we measure it, but that we continually enhance it.

Reviewed to death

Eighteen years ago this month I had my first encounter with a quality review. At the time I was Dean of the University of Hull Law School, and the system that became known as ‘quality assessment’ had just been introduced in the UK. This was the framework for teaching and learning reviews, and the Law School in Hull was the first unit to be assessed ever. And did we prepare for this! I set up a working group that included an external expert, we had daily briefing meetings for months, every lecturer was coached in how to teach a class in front of a reviewer, the paperwork we put together probably created serious deafforestation somewhere. In those days there were only three ‘grades’ you could be awarded through the site visit of the reviewers: excellent, satisfactory, and unsatisfactory. You could not be found to be ‘excellent’ unless you had applied to be considered for this up front, which we did. When the review visit began, the chair of the panel told us she did not believe any academic unit – except maybe two or so in the entire world – was excellent, and that she did not approve of any request to be considered for this. In the end we were judged ‘satisfactory’ with ‘some excellent features’. After all that work…

Nowadays I could not see how anyone could afford to put in that amount of time end energy into just one review. And I say that because, at least for some people, reviews are coming out of our ears. Every major research project has initial site visits, mid-term reviews and site visits, end of term reviews and so forth. Every unit gets regular quality reviews to look at teaching and everything else. The university itself gets reviewed overall. Not to mention that need to put together review documentation for our auditors, and for the Comptroller and Auditor General. And then there are audits conducted by this or that agency or funder. In short, the sheer volume of reviews now visited upon us is such that for some people the process never ends.

Of course I would not suggest that we should not be monitored and held accountable for our standards. But we have reached a point where review mania has taken over, and as this is growing just as money and resources are declining, it creates a major problem for us. And it is made much more complex by the array of organisations and agencies that feel they have a right to review us; and also, by the bureaucratic complexity of some of these processes.

Nowadays when a public utility wants to dig up the road in order to put in new cabling or whatever, other utilities are asked too join in and deal with their deeply buried technology also, so as to avoid a multiplicity of such diggings. It may now also be time for a much more coordinated framework to emerge for quality and other reviews. If there is to be a digging around in the university, let it be one coordinated process sharing relevant information as may be useful. It would also be timely to look again at some of the more specific requirements for reviews, to avoid them becoming excessively bureaucratic.

Back when I was in the UK, and a couple of years after the above review of the Law School, we had a university-wide quality review. There was a session with the visiting reviewers open to some senior staff, and I was one of those present. At the end of the meeting the chair asked us ‘what single change would make the most positive difference to your work and your ability to support the objectives of the university and ensure high quality?’ One of our number responded quickly: ‘Stop having these reviews!’ Here in Ireland, it is important that we conduct and organise reviews so that these can be perceived to be constructive and helpful engagements, but for that to work we must not over-load people. We must take care that we don’t drown in paperwork, to the point where we consider the idea of people asking questions about quality to undermine that same quality. Those who feel we need to be more accountable need to watch they their concept of accountability does not in the end destroy the very quality it seeks to protect.

Large classes, quality concerns

The Irish Universities Quality Board (which has been the subject of occasional discussion on this site) last week published the institutional quality review of National University of Ireland Maynooth (NUIM). The review was an overwhelmingly favourable one, and the panel had many positive things to say about the university. But it did have a concern about the growing class sizes, in some subject areas in particular. This is how the issue is addressed in the report:

‘The student numbers at NUI Maynooth have grown rapidly in recent years, resulting in an unfavourable lecturer-student ratio and extremely large student cohorts, especially in first year, in some disciplines (for example, Geography and English). This poses extra challenges in the delivery of effective pedagogy and does not allow personal tutoring of students. The University is clearly aware of this problem and has been addressing it in various ways, such as training lecturers to use new equipment which facilitates teaching in large classes, and using postgraduate students as teaching assistants. While acknowledging the value of the measures taken to date, the Team encourages the University to continue to explore more creative and innovative approaches to the problem of large classes. Much can be learned from the extensive literature that already exists on this subject. Any delays in giving the issue urgent attention risks damaging the student experience at NUIM and, in the end, may be detrimental to the University’s reputation.’

Clearly this is an issue which we are all having to face. Not only is the state funding for universities being reduced, the institutions are also being instructed to reduce staff numbers by 6 per cent over two years; and at the same time student demand for places is up, and universities are being pressed to take on larger numbers.

It is clear that this combination of policies and pressures cannot go on without creating a major quality issue. The whole concept of higher education in this country has been built around teaching students in manageable classes and, regularly, in small groups. Neither the resources nor the staff now exist to allow us to do this effectively. New equipment and technology, or the use of postgraduates to do some teaching, can no doubt make a contribution, but none of that can solve the overall problem.

It seems to me that either Irish universities will have to reduce student numbers decisively, or else accept (and point out to the government) that small group teaching is no longer sustainable. Of course the third option would be to have a new concordat between the universities and the government, under which a financial strategy is agreed – involving state funding that is capable of supporting programmes of study with teaching methods that maximise quality. Unfortunately I don’t think this is available, and so we remain on a path to much larger classes that could, in the end, destroy the international reputation of Irish higher education.

The statutory dimension

The Irish university system as it is currently constituted has its legal basis in the Universities Act 1997. This statute was the outcome of lengthy discussions and deliberations and an in-depth consultation process involving the sector. It created a single legal framework for all the universities (before that different institutions were governed by different Acts), and it set out a number of principles for higher education, including institutional autonomy for universities, protection of academic freedom, allocation of responsibility for quality assurance, and recognition of the distinction between governance and management. The Universities Act in essence produced a settled framework for higher education and research, and allowed Irish higher education institutions to become serious global competitors. Its significance could not easily be over-stated.

In the light of recent developments, and more particularly in the light of government decisions to re-position responsibility for the monitoring of quality assurance and to dissolve the National University of Ireland, it has become necessary to consider legislation to amend the Universities Act. It may seem that such amending legislation will be limited and will not change the nature of Irish higher education. But as we have not seen any draft Bill so far, and indeed don’t even know for sure what issues the Bill will address and in what way, we cannot be sure about its potential impact. For example, we do not know whether the idea of university autonomy will be compromised, nor do we know whether the legislation will impose greater burdens of bureaucratic controls.

We hear about a likely time frame for the legislation – it has been suggested that the Bill will be published before the summer and will be enacted early next year – but such a tight timeline will be easily managed only if the substance is limited to quite specific and narrow changes. But we have also become aware of the complexities of the proposed legislation, particularly  in relation to the intended winding up of the NUI (which has generated some resistance and criticism); in a recent report the Irish Independent has suggested that there may now be ‘major delays’. In the meantime, according to a report in the Sunday Business Post, a spokesperson for the Minister for Education and Science, Batt O’Keeffe TD, has suggested that one of the purposes of the new framework will be to bring to an end the ‘self-regulating’ nature of the current higher education system.

I confess that all of this makes me pretty nervous. I am nervous because I do not know for sure where all this is going: I am not sure whether we will see a limited and essentially non-controversial update, or whether the principles of the 1997 Act set out above are about to be changed. If the latter, then we should really be having a wider debate (or indeed, any kind of debate) about what is proposed. And it would need to be seen in the context of whatever is going to be proposed in the report of the higher education strategic review now nearing completion.

But even if the intentions of the legislation are presented as limited in nature, they may not be that in practice. For example, quality assurance (which will definitely be affected) goes to the heart of the system, and transferring the responsibility for monitoring this from university governing authorities (which they have delegated to the Irish Universities Quality Board) to a state bureaucracy is not a minor step and may have profound implications for the nature of Irish higher education.

Universities cannot insist that a perfect state has been reached under the 1997 Act and that nothing can ever change. But they can and should argue that the 1997 Act represents a major national settlement on what constitutes a high value university system and that it should not be changed lightly or without proper concern for the implications. What we have right now is a move, at least potentially, to change the system on the back of budget considerations and anecdotal comments on university performance. If that happens, it would not be good enough. So it is now time to explain what is intended and to open discussions on the details. Irish universities are key to Ireland’s economic, social and cultural future. They can and should be reviewed critically, but not casually.

Assuring quality and standards in higher education

Anyone reading the Irish newspapers over the weekend will have seen immediately that the ‘grade inflation’ story, first covered in the Irish Times a week ago, continues to rumble on. And as it does so, the substance of the coverage has begun to shift from looking at the basic numbers to considering the implications and possible actions to be taken. One area of interest in all this has emerged, that of quality assurance. In Saturday’s Irish Times, in an extended piece by Education Editor Sean Flynn, there is the following comment:

‘There are close parallels here with the banking crisis. Essentially, the universities and the ITs regulate themselves. Five university presidents or their nominees sit on the IUQB board; four senior IT figures sit on the Hetac board. One IUQB member tells The Irish Times: “The whole thing is a cosy cartel. Each of us has our own agenda and we can pursue it without difficulty. Occasional concerns have been raised about grade inflation but there has been no serious debate, let alone any decent research work.”’

I would really like to know which IUQB board member delivered themselves of this comment, since it does not correspond at all with what has been going on at board meetings. The description of what goes on at the IUQB is totally inaccurate, and frankly rubbish. Nobody that I have encountered there comes in with an agenda; in fact if they did it might be a better experience, as there might then be a more proactive approach to the quality agenda. However, there is no cartel approach, and the external members ensure that this is so. Furthermore, at least in my presence, no board member has ever suggested that there should be debate or research on grade inflation.

The article continues by pointing out that the government intends to control quality and standards in a new unified agency. This is a reference to the merger between the Higher Education and Training Awards Council (HETAC), the Further Education and Training Awards Council (FETAC) and the National Qualifications Authority of Ireland (NQAI), and the acquisition by the merged body of the quality assurance functions of the Irish Universities Quality Board (IUQB).

Controlling quality assurance, in the sense of exercising a coordinating role of the kind currently undertaken by the IUQB to ensure that there is a consistency of approach, is something the new agency can and no doubt will do. Quality in this context is usually taken to mean a consistency and transparency in the educational process. Standards, on the other hand, are quite a different matter. Essentially standards are the product of the curriculum, which in turn determines the content of each course and the learning outcomes expected from it. In the university sector we do not have a national curriculum, and if it were even to be contemplated it would at a stroke end the idea of institutional autonomy, and even more significantly, of academic freedom. Nor is it clear to me that a process of bureaucratisation of higher education learning content and outcomes would raise Ireland’s reputation.

However, we may now need to respond to the ‘grade inflation’ story in a more proactive manner. Whatever I may believe myself – and I am of the view that the accusation of grade inflation identifies quite the wrong problem – it is clear that this view is not shared by many now commenting in public. The university sector may need to address the matter differently, and may need to consider independent reviews to show clearly what the true position is, and how any issues that may exist can be addressed. Waiting for the Minister to establish a national higher education curriculum is not the best possible tactic, I suspect. We need to undertake some speedy and serious confidence building.

Snuffing out academic eccentricity

Today a friend of mine from another university (which I won’t name) told me about an investigation that has just been launched there to determine whether a particular lecturer’s eccentricity is incompatible with quality requirements. The lecturer concerned does not, I gather, find it emotionally right to face his students, and so he lectures with his back to them. It’s really rather a striking image, a kind of pre-Vatican 2 approach to teaching. As I understand it, students have never complained (though it is a matter of some humorous comment), but a visiting quality assurance team found it unacceptable.

I have in a previous post pointed out that a university system should have some eccentrics, not least in order to avoid the potentially boring uniformity that we would otherwise have to endure – a point also made a few years ago in Times Higher Education by a professor from Sheffield University. Conformity in all things, including teaching conduct, is quite likely to breed intellectual conformity and an impoverishment of academic life. I would readily agree that it would not work well if all academics cultivated eccentricity, and I would argue that it would be a different matter if students objected in a particular case or if the eccentricity consisted of a neglect of duties. But on the whole we need to be tolerant of different ways of thinking, and different ways of doing. We need to welcome and celebrate creativity, which often is closely related to non-conformity.

And above all, we need to discourage all those who believe that quality is found in uniformity.