Quality in higher education

It would probably not be hard to get a consensus around the proposition that universities should aim for high quality in both their teaching and their research. But it is much harder to identify what quality actually is, how it can be recognised and how it can be measured. This is illustrated by the fact that some of the key policy documents on quality assurance for universities go into great detail about the process by which quality should be assured without ever once saying what actually constitutes ‘quality’. For example, the Standards and Guidelines for Quality Assurance in the European Higher Education Area issued by the European Association for Quality Assurance in Higher Education in 2005 makes no attempt to define, describe or identify quality criteria. The Irish Universities Quality Board (IUQB) on its website has a page of ‘useful definitions in quality assurance‘, but the term ‘quality’ is not defined there. On its website, the British Quality Assurance Agency (QAA) does purport to define ‘academic quality’, thus:

‘A comprehensive term referring to how, and how well, institutions manage teaching and learning opportunities to help students progress and succeed.’

But is that actually a definition of anything?

It seems strange that a whole industry dedicated to monitoring and improving quality seems to have no advice to offer as to what ‘quality’ actually is. I once sat next to a senior academic at a dinner; he had been involved in the QAA’s quality assurance process for ten years. When I asked him what ‘quality’ meant, he chided me for asking an ‘irrelevant’ question. He suggested that the institutions themselves could determine what constituted quality, and the task for people like him was to see whether they lived up to their aims. That seems sensible at one level, except that if it were that simple then institutions could guarantee superb quality simply by setting themselves very modest ambitions, and then meeting or exceeding them.

However, because nobody has anything much to say as to what quality is, the temptation is to get out of this dilemma by focusing entirely on process: we cannot say whether what you teach is good quality, but we can ask whether you have followed the 20 prescribed steps when you developed the programme and are counting the answers students have given in the feedback questionnaires. And on the whole, that is how a fair amount of quality assurance has been conducted.

The problem with this is that if your excellence is – publicly – going to be measured on the basis of how satisfied people are with your processes, then you had better have comprehensive processes and stringent monitoring; and if that’s your concept of quality, then you had better steer clear of innovation, because innovation (like entrepreneurship) is risky and may sometimes fail or not enthuse the users at first. It is much safer to stay with your existing offerings and just make sure that all the paperwork is in good order.

We have therefore come to accept that quality assurance is about process, whereas it should be obvious that quality is about content and intellectual innovation. If we are serious about having a high quality higher education system, then we have to start asking questions about content, avoiding the risk of suggesting that there is one standard way of measuring this.

Advertisement

Irish higher education: the new quality and standards legislation

The Irish government has now introduced the long-awaited legislation on educational quality and standards. The Qualifications and Quality Assurance (Education and Training) Bill 2011, which has the purpose of amalgamating a number of public agencies and taking over the functions of the Irish Universities Quality Board, has now been published.

The original impetus for this legislation, signalled by the last Fianna Fáil-led government, was a budgetary one, and was part of the decision to rationalise the public service agency landscape. In this particular instance the idea was that all those agencies dealing with quality and standards in post-secondary education should be brought under one umbrella. This intention was first announced in 2008, so the implementation has not proceeded at what one might call a good pace. However, the Fine Gael/Labour coalition has continued with the plan, though one original intention – to dissolve that National University of Ireland – was dropped. So we are now to have a new agency, to be called the Qualifications and Quality Assurance Authority of Ireland (QQAAI, perhaps not the finest acronym one might wish for). As an aside, this continues the practice in Ireland of naming such agencies an ‘Authority’, which suggests a particular, and maybe rather outdated, directive approach to public service.

One concern that might have existed about the proposed legislative framework was that by mixing the quality assurance process of the university sector with quality and standards in other educational bodies, the autonomy of the university system and its particular ethos and mission might be compromised. The Bill may reassure in that respect, at least in so far as there is special provision throughout  for what are described as ‘previously established universities’. These will continue to operate their own quality assurance processes, subject however to obligations of consultation with and information provided to the new Authority (section 28).

So are there still points of concern? Possibly. One would be that the system will be run by an agency which will not primarily be concerned with universities, and which may either not have the resources to focus on them sufficiently, or may find it easier to apply assumptions more appropriate to non-university bodies. The latter risk may be augmented by the uncertainty as to what will happen to the existing staff of the IUQB, which is not technically being merged into the new body. Another concern is that the new Authority, under the proposed statutory provisions, may possibly be under pressure to operate in a rather bureaucratic way.

The proof of this particular pudding will certainly be in the eating. It will be important to ascertain how the QQAAI will work in practice. One promising aspect at least is that its chief executive will be Dr Padraig Walsh, formerly of DCU and most recently the chief executive of the IUQB. His influence on the culture of the new body may ensure that it operates in a way that respects university autonomy and encourages a positive approach to quality assurance and enhancement.

Measuring higher education quality

Ever since the quality of higher education started to become a matter of concern in society, people have been struggling with the idea of how, if at all, quality could be measured. There has tended to be an assumption that quality assurance could only be real if there were metrics involved, because without them there would be no sense of objectivity, and furthermore there could be no meaningful targets for the achievement of quality. However, the metrics have tended to focus strongly on inputs (the student-to-faculty ratio being a typical example), not least because outputs (principally degree results) have often been questioned in terms of their integrity.

Of course it is not just the quality assurance process that comes up against this, it arises also when various bodies or media attempt to put together league tables (which, to carry weight, have to be based on an assumption of comparable quality). This ambivalence of all this has recently been illustrated by the US journal Chronicle of Higher Education, showing the really wildly different criteria that are used in various rankings.

But in the end rankings are indicative rather than definitive. However, formal quality assurance processes have to convey a sense of confidence in the objectivity of their use of metrics or other information. Can this be done?

The Irish university sector has fared better than most, because the quality processes of the Irish Universities Quality Board have taken these issues into account and have used a negotiated framework aimed at supporting improvement rather than condemning failure. This gain could be easily lost. It must be hoped that the new framework of the proposed unified agency will work constructively with what has been achieved to date. The important thing about higher education quality is not that we measure it, but that we continually enhance it.

‘Grade inflation’ and the Irish Universities Quality Board

Yesterday’s Sunday Business Post carried an interview with Padraig Walsh, the chief executive of the Irish Universities Quality Board (IUQB). Over the past fortnight when the ‘grade inflation’ story was doing the rounds, some commentators picked on the IUQB and wondered why a body with a remit to protect quality did not address this alleged decline in standards. In the interview Padraig Walsh makes the point that the IUQB is not a regulatory body and cannot compel the universities to adjust their marking systems.

There may be people who will respond to that by asking what the purpose of the IUQB is if it cannot restrain universities when they are compromising standards. The answer is that there is an important difference between quality and standards: quality assurance is about checking whether decision-making processes are transparent and consistent, whereas standards relate to the substance of the curriculum and its appropriateness. Under law the latter is a matter for each university; changing that would require the state (or some other body designated for this) to implement a centralised curriculum for the entire sector, a move that would be totally incompatible not just with university autonomy but also with international best practice.

Under plans announced some time ago by the government, the IUQB’s functions in relation to quality assurance are to be transferred to a new regulatory agency that will be in charge of quality assurance for the entire post-secondary education sector. There is a risk, I believe, that there will be expectations that this new body will be able to compel individual institutions to revise or change their standards. If this were to happen, the implications will be far more damaging to the reputation of Irish higher education than any perceived grade inflation.

It is worth saying that the IUQB, contrary to the impression given by some commentators, has played an enormously important role in addressing quality issues. Across the university sector it is now accepted without argument that institutions owe a duty to the public to demonstrate that they use methods and apply decisions in a consistent and fair manner, and that publishing quality reviews so that everyone can consult them is in the interests of transparency. These are major advances.

On the other hand, the universities themselves need to explain to the wider public much more convincingly than they have done how they determine standards and how they ensure that these are in line with international benchmarks. Simply declaring that there is no problem will not be sufficiently convincing. There is more work to be done.

Is it the end of the road for the Irish Universities Quality Board?

As I mentioned last week, the Irish Budget and Book of Estimates published on October 14 included, more or less in passing, a commitment by the Government to transfer the functions of the Irish Universities Quality Board (IUQB) to a new state agency to be formed from a merger of three existing ones. I confess that I am deeply uneasy about this move, for a number of reasons.

1. There was no consultation or even advance warning that this step was to be taken. Such a step – and as I shall explain in a moment, it is a major step – should not be taken without proper analysis and discussion.

2. The three existing bodies that are to be merged (and then have the IUQB functions added) are the Higher Education and Training Awards Council (HETAC), the Further Education and Training Awards Council (FETAC), and the National Qualifications Authority of Ireland (NQAI). These bodies, while significant, have no formal role in relation to universities, and it is doubtful whether min merged form they are an appropriate entity to supervise quality assurance for the university sector.

3. The IUQB, while it has its imperfections, has performed its role well on the whole, and is aware of what it needs to do to improve further. More importantly, it (and the processes it coordinates) has the confidence of the wider academic community.

4. It may be feared that what is brewing here is a move towards a higher education quality ‘inspectorate’, which would be a very wrong approach to quality assurance and improvement in Ireland.

But the chief concern is that the Government is adopting a heavy-handed approach to the universities, and appears to be eager to demonstrate that it does not trust them and is not prepared to work with them. This, if it is true, is serious, and will need to be corrected, no doubt by an effort on both sides. On the Government side, one step in the right direction would be to stop publishing new decisions and initiatives in the media and press statements which have not been the subject of any prior discussion with the universities.

For the universities, it may be important to recognise that we do not appear to have the confidence of this key stakeholder; we will need to work constructively on a policy of engagement that helps us to overcome that.

Assessing the Budget and Estimates

Now that the details of the Budget and the Book of Estimates have become clearer, it is possible to say that, on the whole, it could have been worse. Even after all details have been accounted for, higher education will still have been treated less generously than primary or secondary, but that is often the case at times of pressure on public finances. It now appears that the revenues generated from the increased student registration charge will be available to the universities and colleges. This will mean that, overall, the aggregate income between fees, recurrent grant and registration charge will leave the universities with more or less the same income from the State as last year – which, however, means a cut in real terms of over 6 per cent.

There is, as I have noted previously, a fairly generous increase in capital spending, but for the moment I do not know what it is for, and I suspect a lot of it is for building projects already authorised. I also understand that PRTLI Cycle 5 will be announced shortly, although expenditure under it will not be possible for over a year.

The one item about which I confess I am disturbed is the decision to compel the Irish Universities Quality Board (IUQB) to transfer its business to a new State agency created from a merger of three existing ones – HETAC, FETAC and the NQAI. I believe that this move is a mistake, and could lead to an unacceptable bureaucratisation of quality assurance. But I suppose I should withhold judgement until I know more in the way of detail about these plans.

The year ahead – or more likely, the years ahead – will be tricky for universities, and it will be hard to innovate. But we shall address the situation positively in DCU.

Universities have a lot to learn?

In an article in yesterday’s Sunday Independent, we were told that a report commissioned by the Higher Education Authority was ‘damning in its indictment of universities in the Irish Republic’ for failing major quality standards and expectations. The article also claims that the report ‘noted “sustained systematic shortcomings” where universities had unilaterally suspended quality assurance activities and ignored the IUQB [Irish Universities Quality Board]’.

It is probably the case that the author of this article got somewhat carried away, for while the report does indeed point to some issues that need to be addressed, the overall tone was far from ‘damning’. For a start, it needs to be stated that this report was commissioned by the HEA, but at the request of the IUQB itself, with the agreement of the universities. The expert group who conducted the review, in their findings, say at the outset that ‘the work done to date [by the IUQB] is impressive and has given strategic impetus to raising the awareness and activity levels in the universities in respect of quality assurance and improvement.’ The report does also refer to and highlight some shortcomings, some of them quite significant, but that is what a quality review is about. The response by the IUQB has beeen constructive and positive.

It is not my intention here to get into the details of this report and its recommendations – that is perhaps for another time. However, it is important that media coverage of such documents should be balanced and constructive, and should not seek out and highlight only the critical passages. This is important for a number of reasons (including the need for good journalism), but one of them is that quality reviews are most effective, and the follow-up most successful, if the emphasis is on constructive engagement and support rather than judgement and blame.

Here in DCU we have, we believe, a very good track record in quality assurance. All of the university, including my own office and my own role, have been quality reviewed by independent expert panels (with a majority external membership), and my senior management team meets every review panel and is involved in the follow up for every singly review. All the reports are published and are available to the general public. We have also persuaded the academic community that these reviews are not intended to damn anyone, but to support them and help them in bringing about and maintaining the highest levels of quality.

We all have more to learn, and there are always things we could do better. Even in the Sunday Independent, I suspect.

PS. Since I wrote this, the government has announced in the 2009 Budget documentation that the IUQB’s functions are to be transferred to a new government-run agency.

Quality in the universities

One of the key developments in the higher education sector over the past two decades is the arrival of what is sometimes called the ‘quality movement’. In a nutshell, the various stakeholders of the universities have been less and less willing to take on trust that the teaching and research in these institutions is of high quality and have sought ways in which this could be independently verified.

In the UK this led to the quality assessment process overseen by the Quality Assurance Agency (the QAA), and the Research Assessment Exercise. It has been argued that these two mechanisms may have, at least initially, had a very doubtful impact on quality, though their quantitative impact may have been clearer (not to mention the bureaucratic impact). On the other hand, there can be little doubt that these processes made both the institutions and their staff acutely aware of the need to demonstrate the value of delivery of the universities’ core mission.

In Ireland the universities established the Irish Universities Quality Board, to which the governing authorities of all the institutions have ceded certain tasks for maintaining a framework of quality assurance and improvement. It has had an important role in developing and sustaining a framework of regular and public reports for each university and for academic units and programmes.

I propose to look again in a future post at the success of the Irish model, but it seems to me that a question we must ask initially is the key one: what actually constitutes ‘quality’ in a university, and if you want to measure it, what do you measure and how? These questions appear simple enough, but are in fact extremely complex. The British experience may be instructive, as a good deal of what was done initially to introduce quality assurance mechanisms may have been less than ideal; in Ireland we have an opportunity to learn from that.

So in my next post on this blog, I propose to attempt an answer to the question of what constitutes quality. Then after that, I shall look briefly at how our initiative with the IUQB has worked.

In the meantime, I would welcome comments.