Quality in higher education
It would probably not be hard to get a consensus around the proposition that universities should aim for high quality in both their teaching and their research. But it is much harder to identify what quality actually is, how it can be recognised and how it can be measured. This is illustrated by the fact that some of the key policy documents on quality assurance for universities go into great detail about the process by which quality should be assured without ever once saying what actually constitutes ‘quality’. For example, the Standards and Guidelines for Quality Assurance in the European Higher Education Area issued by the European Association for Quality Assurance in Higher Education in 2005 makes no attempt to define, describe or identify quality criteria. The Irish Universities Quality Board (IUQB) on its website has a page of ‘useful definitions in quality assurance‘, but the term ‘quality’ is not defined there. On its website, the British Quality Assurance Agency (QAA) does purport to define ‘academic quality’, thus:
‘A comprehensive term referring to how, and how well, institutions manage teaching and learning opportunities to help students progress and succeed.’
But is that actually a definition of anything?
It seems strange that a whole industry dedicated to monitoring and improving quality seems to have no advice to offer as to what ‘quality’ actually is. I once sat next to a senior academic at a dinner; he had been involved in the QAA’s quality assurance process for ten years. When I asked him what ‘quality’ meant, he chided me for asking an ‘irrelevant’ question. He suggested that the institutions themselves could determine what constituted quality, and the task for people like him was to see whether they lived up to their aims. That seems sensible at one level, except that if it were that simple then institutions could guarantee superb quality simply by setting themselves very modest ambitions, and then meeting or exceeding them.
However, because nobody has anything much to say as to what quality is, the temptation is to get out of this dilemma by focusing entirely on process: we cannot say whether what you teach is good quality, but we can ask whether you have followed the 20 prescribed steps when you developed the programme and are counting the answers students have given in the feedback questionnaires. And on the whole, that is how a fair amount of quality assurance has been conducted.
The problem with this is that if your excellence is – publicly – going to be measured on the basis of how satisfied people are with your processes, then you had better have comprehensive processes and stringent monitoring; and if that’s your concept of quality, then you had better steer clear of innovation, because innovation (like entrepreneurship) is risky and may sometimes fail or not enthuse the users at first. It is much safer to stay with your existing offerings and just make sure that all the paperwork is in good order.
We have therefore come to accept that quality assurance is about process, whereas it should be obvious that quality is about content and intellectual innovation. If we are serious about having a high quality higher education system, then we have to start asking questions about content, avoiding the risk of suggesting that there is one standard way of measuring this.
Explore posts in the same categories: higher educationTags: IUQB, QAA, quality assurance
You can comment below, or link to this permanent URL from your own site.
University Blog 
October 18, 2011 at 7:30 am
One of the best definitions of ‘Quality’:
“Quality … you know what it is, yet you don’t know what it is. But that’s self-contradictory. But some things are better than others, that is, they have more quality. But when you try to say what the quality is, apart from the things that have it, it all goes poof! There’s nothing to talk about. But if you can’t say what Quality is, how do you know what it is, or how do you know that it even exists? If no one knows what it is, then for all practical purposes it doesn’t exist at all. But for all practical purposes it really does exist. What else are the grades based on? Why else would people pay fortunes for some things and throw others in the trash pile. Obviously some things are better than others … but what’s the “betterness”? … So round and round you go, spinning mental wheels and nowhere finding anyplace to get traction. What the hell is Quality? What is it?
Robert Pirsig, Zen and the Art of Motorcycle Maintenance
October 18, 2011 at 7:38 am
Lee Harvey and James Williams are editors of the academic journal, Quality in Higher Education. In 2010, they published two articles that reviewed 15 years of contributions to that journal on the topic of quality in HE. On foot of this comprehensive review, they conclude that: “Ultimately, the review suggests that it is still not clear that, even after 15 years, quality assurance systems have really enhanced higher education” (Harvey and Williams 2010b:81).
Harvey, L., J. Williams. 2010a. Fifteen Years of Quality in Higher Education Quality in Higher Education 16(1) 3–36.
Harvey, L., J. Williams. 2010b. Fifteen Years of Quality in Higher Education (Part Two). Quality in Higher Education 16(2) 81–113.
October 18, 2011 at 8:17 am
Halleluiah! At long last a senior Academic who may have some influence has spotted something which has been blindingly clear to someone who has spent 27 of their 31 year working life within Industry.
Ask any Production, Industrial or Mechanical Engineer what ‘Quality’ is and you will get a proper answer. Not some airy fairy nonsense touted by that guy you sat next to at dinner.
Academia is really only concerned with the appearance of quality and not the reality of it. Making sure that all their boxes have been ticked and that their processes have been followed to the letter. Absolutely bugger all about are the courses they teach any bloody good or not.
Far too much incestuous and naval gazing behaviour within Academia I’m afraid and all the signs are its getting worse and not better.
October 18, 2011 at 8:53 am
I’m with Steve on this. The first courses we put online were in “Quality Management” and I gradually became aware from my exposure to the ideas in those courses that higher education has no clue about quality. Most of the people who are responsible for standards don’t even understand the principles of measurement (concepts like repeatability, reliability, precision, accuracy, sampling etc).
One of the main principles that has really been missed (and alluded to in the blog) is that you cannot guarantee outputs by measuring inputs. No set procedure will guarantee a good outcome, and is unlikely to achieve one if you don’t precisely define the outcomes you want and measure correctly what you are getting. We are now about to tackle quality in education in Ireland by setting up an Academy to improve training and research into teaching methods. This is absolutely hopeless. Teaching is not rocket science. If we measured properly, identified where we have weak teachers, required them to improve, and provide simple training, it would achieve a lot more. (eg put less text on your Powerpoint slides)
Many people who resist the idea that we should control or reduce costs in higher education use the quality argument as a fig leaf and not only do not have the quantitative skills to back up their arguments, they do not even realise that Quality Management is largely a quantitative technique.
The elephant in the room is that academics do not want anyone telling them how to do their jobs (often describing this as “Academic freedom”), their unions’ main reason for existing is to support them on this and to get them the odd pay rise, and governments have never had the courage to take them on.
October 18, 2011 at 9:09 am
Brian,
Thank God! Another free thinking individual who says what needs to be said. Surely there must be more of us out here in the wilderness.
October 18, 2011 at 9:21 am
@steve – but Universities are full of free thinking people. That’s what they were set up to do – free thinking. And that is what the concept of Academic Freedon is there to protect. And if we have to be distracted by the tedious issue of teaching, then just herd a pile of them into a room and we’ll waffle at them in 50 minute slots. Unless of course, you’re a free thinker in the domain of educational methods. In which case you can use the students to experiment on (And not using some well proved technique like “less writing on Powerpoint slides”)
And before someone says to me that there are lots of good lecturers out there, can you just count them, divide that number by the total number of lecturers out there and multiply by 100 and let me know what you get (that’s known as a percentage).
October 18, 2011 at 9:11 am
Are you looking at this with one eye only?
What or how would you measure quality? I’d be more interested in a holistic answer covering expectations of students also.
Behind your first elephant may be an even bigger elephant???
October 18, 2011 at 9:26 am
‘Al. I’d be interested in a holistic measurement also and student expectations would, of course, have to be included. This is a highly technical area. We have to devise methods of measurement that we can verify measure what we want (ie discriminate between students who have achieved outcomes ant those who have not in a graduated way), and are demonstrated to be repeatable. This requires a lot of work that no institution seems to be willing to do.
By the way, I showed you my elephant, will you show me yours?
October 18, 2011 at 9:47 am
My bigger elephant is level of student ability and ability to measure this level.
This elephant has many facets.
What measurement? knowledge, skill, competency?
Method of measurement? in terms of affordability!
Outcomes system: surely worthy of a challenge?
Do you see my elephant?
Or smell him/her?
October 18, 2011 at 9:06 am
Great post!
In the same way that Law does not deliver Justice, Quality Assurance doesn’t necessarily deliver Quality!
Quality Allowance may be a better description, in that a system is set up that allows quality to happen easier.
However, dogmatic adherence to systems leaves the whole enterprise as a box ticking exercise. As are standards that loose present time effectiveness.
October 18, 2011 at 9:30 am
@Al. Quality Assurance done properly DOES deliver quality.
Your perception of what constitutes Quality Assurance may come from some of the poor systems you have encountered. Many people are not a fan of the term Quality Assurance and prefer Quality Improvement. I prefer the term “Continuous Improvement” which suggests that you can only improve by measuring your output, finding out what is wrong with it, and acting to improve it. This may or may not result in mind-numbing box ticking procedures. (usually not required)
October 18, 2011 at 9:52 am
I have to object!
Lets move our conversation into prison education in order to contrast our viewpoints.
Obviously the current system probably could improve, but by my understanding of what you say, can we create a system that reforms the prison population?
October 18, 2011 at 10:51 am
@Al. Good example. Is reforming the prison population an explicit objective. Have we measured how we are doing in that? Have we designed specific activities within the system to improve that outcome and how effective have those activities been? You may well be referring to another public service that is equally dysfunctional.
October 18, 2011 at 11:36 am
1- Yes
2- No
3- Yes
4- TBD!
No, I am referring to people in general, where dysfunctionality exists in uneven distributions. These people are the lecturer, the learner and the manager of the process. We need to design systems that cater for our national dysfunctions, systems that challenge them.
I like the idea of ‘continuous improvement’, but when it comes to humans and our lack of ‘quality’ we need to acknowledge this and pay particular attention to the terrestrial features that we have to navigate rather than over focusing on a map that may not for whatever reasons want to show these features on the ground.
October 18, 2011 at 11:48 am
@Al. I can’t find anything to disagree with there. Might it then be worth suggesting that we get rid of the term “quality assurance” insofar as it seems to suggest that there is a specific goal that we at some point can reach, and instead, admit that whatever our disagreements about the level of quality we are currently at, we should agree that there will always be plenty of room for improvement. Perhaps this may reassure those who feel threatened. However, having done that we still need to do the hard work of determining what our objectives are, figuring out how to measure them (a cop out to say we can’t), measuring how far short we are falling, prioritising areas for improvement, and being ingenious when devising strategies for improvement (throwing money at a problem does not constitute and ingenious solution).
October 18, 2011 at 1:37 pm
All of what you say looks agreeable, but where do you strike a balance here between bread and butter lecturing and improvement.
Is it an 80-20 rule of thumb or something?
October 18, 2011 at 1:44 pm
@Al. I would suggest that because at this point we know so little about how this should be best carried out, we should not be too deterministic about it. However, my gut feeling is that if the effort to improve quality (including taking measurements) constitutes more than 5% of our effort in teaching it would be too much. It really needs to be costed. What is the cost of our efforts vs. the value of the improvements? If we really adopted an effective continuous improvement system we would make great improvements at first and then they would tail off. We would continue to operate a system in order both to make marginal improvements but perhaps more importantly not to let bad practices emerge again. Perhaps, 2% of our effort – we’d have to suck it and see.
October 18, 2011 at 2:03 pm
Ok, put me down as an enthusiastic septic!
October 18, 2011 at 10:11 am
Thanks to Brian & Steve for rehearsing for us all some of the best, blatantely anti-intellectualist stereotypes regarding academia, masquerading the whole lot as an example of free-thinking was sublime! Please do not let anyone else’s comments prevent you from indulging further in such an arrogant, misguided performance…
October 18, 2011 at 10:58 am
@Anna. Comments may well be useful in changing my viewpoint, but to be honest what I really need is evidence. I’m actually pro-intellectual but the trouble is I thought that requiring evidence was one of the foundations of intellectual thinking.
Everyone is entitled to an opinion but we don’t have the time to listen to them all – we need evidence.
October 18, 2011 at 12:50 pm
*It seems strange that a whole industry dedicated to monitoring and improving quality seems to have no advice to offer as to what ‘quality’ actually is….However, because nobody has anything much to say as to what quality is, the temptation is to get out of this dilemma by focusing entirely on process.*
I think there is a difference between asking what quality ‘is’, (i.e. looking for a stable, unequivocal definition) as opposed to asking what quality ‘means’ (what you asked the senior academic at dinner) the latter although focused on content implies the search for understanding, which is in itself a valuable *process* and, most importantly, the possibility that quality might *mean* different things to different people in different contexts. Also it is not accurate that nobody has anything much to say about quality, philosophers from Aristotle onwards have speculated on its ambivalent relationship with subjective feelings and objective facts, the QAA inability to provide a definition only reflects the elusive nature of the concept, particularly when it comes to its application in the field of knowledge acquisition/education.
The current trend to apply instrumentalist notions deriving from the world of business to educational matters should be resisted (as many argue) as entirely inappropriate, what I would instead recommend is a combination of a revisited notion of process (different from the current one which, I agree, is unsatisfactory) that allows intellectual innovation to flourish, this is akin to contemporary notions of creativity which have abandoned the cliche’ notion of Romantic individual genius to focus instead on the collaborative nature of the *creative process*.
October 18, 2011 at 1:00 pm
Sorry, I’m slightly amending the last sentence, my recommendation should read as follows:
‘ what I would instead recommend is a revisited notion of process (different from the current one which, I agree, is unsatisfactory) that allows intellectual innovation to flourish, this is akin to contemporary notions of creativity which have abandoned the cliche’ idea of the Romantic individual genius to focus instead on the collaborative nature of the *creative process* itself’.
October 18, 2011 at 1:06 pm
@Anna. If we were adopt this approach to quality how would we know if it was succeeding?
October 18, 2011 at 1:39 pm
From one of Brian’s earlier notes (rearranged) –
“The elephant in the room is that academics do not want anyone telling them how to do their jobs (often describing this as “Academic freedom”), their unions’ main reason for existing is to support them on this and to get them the odd pay rise, and governments have never had the courage to take them on.”
“….. I gradually became aware from my exposure to the ideas in those courses that higher education has no clue about quality”
“Teaching is not rocket science. If we measured properly, identified where we have weak teachers, required them to improve, and provide simple training, it would achieve a lot more. (eg put less text on your Powerpoint slides)”
Brian,
I entirely agree. This in my view is the core of the whole issue that is being addressed here -(saving the side argument distractions, which are interesting in themselves.)
In my experience rather too many academics moving up the ccareer ladder are more concerned with political maneouvring and saying ‘the right things’ to those whom they see as having power than in ensuring good quality output. The hard questions are not asked, the usual boxes are ticked and any dissenting views are, all too often, explained away as being from non-academics or anti -intellectuals. Or simply not read, ignored. So the system as it currently exists does not have any adequate corrective mechanisms in place, nor is it evident that fresh thinking or retraining may be useful. . The Emperor can continue wearing his new clothes.
And as asked at the head of this blog, it appears to many outsiders (and to rather too many insiders) that ‘University education is losing its value’. It need not. At least this blog asks the question- and I think some worthwhile issues are being raised.
John Bisset
October 18, 2011 at 2:07 pm
May I add a rider –
I also agree with Brian that if quality improvement takes too much of the available time and resource it will be counter-productive.
Somewhere between 2% to 5% continuously is my gut feel for an acceptable level of effort, though the initial work to get a satisfactory process running would temporarily require more overhead. Change is always difficult to kick start.
‘Suck it and see’ indeed.
John Bisset
October 18, 2011 at 3:39 pm
This has been the most interesting post exchange I have read in a long time. Lets hope that we are all not simply wasting our time here and that someone is listening. I fear that the status quo will remain unchanged as the Earth would have to spin in the opposite direction before sanity triumphs.
Those with any get up and go will have got up a left Academia for the sanity which is called Industry.
October 18, 2011 at 4:02 pm
‘the sanity which is called Industry’ What a humorous notion!
Let’s spare a thought for the mad, almost masochistic types who stay behind and still believe that what they are doing has some cultural/social/economic ‘value’, I guess that would be asking too much, such preposterous is the contention!
October 18, 2011 at 4:36 pm
@Anne I think you nailed it there: “Let’s spare a thought for the mad, almost masochistic types who stay behind and still believe that what they are doing has some cultural/social/economic ‘value’,” – the key word there is “believe” – a little like religion where people will either believe what suits them (“a la carte”) or are too timid to question the wisdom that has been passed on to them,and do not consider that they really should be looking for evidence. When it comes to bad teaching it is hard to know which is worse, those who know that they are not good teachers and could not be bothered (I used to be one of them), or those that genuinely believe that they are good but are not. (This is not to deny the existence of those who think that they are good and actually are).
Having said that I always thought that if you were setting up a experiment to falsify the hypothesis that things are better in the private sector than in academia, you might suggest: “If they are there will be a net flow of people from the academia into the public sector”. Now has anyone measured that over the past 15 years? Answers on a postcard…
October 18, 2011 at 6:39 pm
@Brian I agree the key word is ‘believe’ but not for the slightly patronising reasons you identify. Religious metaphors aside having the firm belief that what one is doing has some value, considering one’s job almost a vocation does not automatically entail self delusion or critical blindness. Self-reflexivity, critical thinking, and yes evidence (crucially, the meaning of such terms resonate differently in different disciplinary contexts) is an essential component of academic practice, as any *good* educator knows.
October 19, 2011 at 9:22 am
@Anna OK, so what would you consider to be evidence that Higher education currently gives value for money to those who pay for it?
October 18, 2011 at 4:46 pm
@Steve. I don’t think anyone is listening. I’m not sure why. It is as if we have an unwritten agreement between the management of the cartel of public education and the unions to try to hold on to a good thing for as long as possible. Some politicians may see through this but changing it would be such a long-term initiative that it would never be a vote getter. Even now the signs are that they are looking in the wrong direction. The recent statement by the Irish Higher education Authority on “directed diversity” really indicates is that all we are going to get is some centrally planned modification of the cartel that will yield slight efficiencies in doing all the things we are currently doing wrong. The proposed Higher Education Academy is another good example of where we will try to do what we are doing already more efficiently rather that do something that actually has an impact.
October 19, 2011 at 1:19 pm
@Brian Similarly to the blizzard of peformance indicators Music mentions in her reply below, there are endless reports that confirm the economic value of university degrees in terms of securing higher levels of income in future employment.
Also, to confirm the efficency of the UK university system a Thomson Reuters Study just published states that the country produces most impactful research despite falling research investment, http://www.reuters.com/article/2011/10/19/idUS40187+19-Oct-2011+HUG20111019
As it has often been discussed in this forum Universities should get better at communicating their role and the wider value of education, this article offers a few insights: http://www.guardian.co.uk/higher-education-network/blog/2011/oct/18/communicating-he-value?newsfeed=true
October 18, 2011 at 11:09 pm
John’s experience suggests that “the system as it currently exists does not have any adequate corrective mechanisms in place, nor is it evident that fresh thinking or retraining may be useful.”
Respectfully I’d like to add a different experience. As an academic directly engaged in quality assurance processes, I work in a system that has all sorts of corrective mechanisms in place, and that generates a blizzard of performance indicators in relation to perceptions of quality, relative metrics of quality, graduate employment outcomes as a proxy for educational quality, standards as a proxy for quality and, sometimes, the blunt force of targets met as a measure of quality. None of these are simple, and none of them are stupid (well, the last one is often a bit tedious, but it does allow for sign-off on workplans.)
I do however agree that it remains relatively difficult for anyone to precisely capture the quality of this quality we’re pursuing, which is why other near-but-dfferent terms relating to satisfaction or standards can come to stand in for it.
As an academic, I’m strongly with Anna on all of this. We need to work together within and beyond universities to develop a means of approaching this complicated question. Academics who call this kind of process “critically self-reflexive” mean what they say–it’s genuinely tough to do this kind of thing effectively without descending into name-calling about everyone else’s navel gazing.
October 19, 2011 at 12:47 pm
I have sat in exam board meetings where colleagues bend over backwards to condone module failures in order that a students who may have had multiple resits and still failed a Module can depart with an MSc. No doubt they to an extent feel pressurised to meet so called quality performance targets.
I took a year off work and lost £80k in salary to take my Engineering MSc at the ripe old age of 46 and to see absolutely crap students leaving with the same qualification that I have makes me a tad annoyed to put it mildly.
I have little confidence that the situation will change any time soon BUT by God I will fight my corner to ensure that only students who meet at least the minimum standard (and believe me a 40% pass mark is pretty poor by my standards), leave with an Engineering Masters. Luckily I still have a reputation within Industry and will not compromise my integrity just to tick another quality performance pass rate box.
October 19, 2011 at 1:26 pm
@Steve. And I’m sure you would agree that individual lecturers sticking to high standards or working hard at teaching does not constitute a quality “system”.
October 21, 2011 at 3:48 pm
I agree with the critique and the conclusion that you cannot meaningfully consider quality without considering outcomes.
I find a “journey” model useful in making sense of the concepts of quality and outcomes (or standards).
“Quality” is the journey from a starting point, considering all aspects of learning and the students’ experiences. “Outcomes” are the destination – what the students have gained, achieved and demonstrated when they reach the destination. “Reference points” can be added to the model to help locate the destination (outcomes), and the value of arriving there. These reference points include qualifications frameworks, and requirements or expectations of professions and employers.
The model can be adapted or extended to suit different contexts and purposes.
Another problematic term that needs unpacking is “enhancement”. I understand there were political reasons for its use in current quality policy. These include: reluctance for higher education to be seen to be adopting the business / industry process language of “continuous improvement”, and concerns about deficit views -if you always need to improve, you are never good enough. I do not share these fears. The term is downright unhelpful if it is interpreted as always about “nice to have”, “safe”, incremental additions to existing practice.
Again, I find a need to make sense of the term for operational purposes. Enhancement is about change -about making it, or doing it better, and making the best use of opportunities and resources. This may include doing new things, in new ways, and being prepared to stop doing things.
Major change is a reality of the current higher education environment, including changes in funding, technologies, student populations and opportunities for graduates -many of these have been discussed in recent blog postings. We need to shape our concepts of quality and enhancement, and their implementation, in ways that actively encourage, and indeed expect change in practice.