Posted tagged ‘assessment’

My colleague the computer

April 30, 2012

It’s that time of year when academics all over the place get ready for another avalanche of marking and assessment. In my own case, while I really do miss teaching very much and am looking at ways of returning to it, I don’t miss marking. Not even slightly. And I feel for those who will, over the next couple of months, be inundated with it.

But is there another way? In fact, could we just give the job to computers? And might we find that they can grade essays and assignments and examinations just as effectively as we can? Well perhaps, according to a study conducted by researchers at the University of Akron. They compared grades given to 22,000 short essays written in American schools by live examiners with those recorded by computers running ‘automated essay scoring software’. The differences were, according to the researchers, ‘minute’.

I don’t know what kind of software this is, or how it works, or what its stated limitations might be, but this is a pretty amazing result. We know that computers can easily grade multiple choice examinations, but essays? And can we really imagine that an assignment intended to produce reasoned analysis could be assessed by machine? More generally, how much work has been done in considering the role that computers can play in designing, conducting and assessing teaching?

In fact, this is a subject of some interest in the education world. In July of this year there will be a conference in Southampton in England on computer-assisted assessment, and indeed there is a journal on the subject.

There are probably various contexts in which higher education assessment can be conducted by or with the help of software. But equally there are others where, at least from my perspective, it is unlikely that computers will be able to make robust qualitative judgements that could replicate human marking. Somehow I doubt that, in a few years, lecturers will no longer have to be examiners.

Whose grade is it anyway?

November 23, 2010

One of the key performance indicators of higher education is the grade given to a student as part of the examination or assessment process. In order to ensure that the grade is appropriate and merited and is not influenced by improper considerations, various safeguards are built in. These include the consideration of grading by external examiners and boards of examiners, where marks can be reconsidered and adjusted.

However, such processes can become controversial, and indeed can raise accusations of inappropriate decision-making. In one Irish institute of technology recently some lecturers boycotted graduation ceremonies because they were unhappy about the adjustment of marks by appeal boards. One Canadian university has recently been in the spotlight for allegedly forcing a professor to lower his marks.

As the debate over the past year about ‘grade inflation’ has shown, the way in which student performance is assessed is one of the most critical issues in higher education. In order to ensure that grades are seen as appropriate and are respected, the system used needs to be impartial, transparent and intellectually demanding. In this context however, groups and boards can get it wrong just as easily as an individual, particularly if they pay excessive attention to institutional interests.

Occasionally it is suggested that the answer is to make this an administrative process, subject to bureaucratic procedures that will kick in particularly if the pattern of grades arouses suspicion, and more particularly still if grades are coming out too high. On the other hand, academic achievement is not a matter of administrative judgement, and should not become one. Equally however, the grades awarded are not necessarily an expression of ‘academic freedom’: I cannot insist that my marking standards should be applied even where they deviate from those of others.

There is no perfect way of dealing with this, but the one most likely to address problems is the system of external examiners, under which grades are checked by senior academics from other institutions to ensure that the system has integrity. However, this system, which relies heavily on personal and institutional goodwill, is coming under stress, in part because external examiners (now considered ’employes’ for revenue purposes) cannot be properly rewarded for what they do, and in part because the bureaucracy of assessment is threatening to overwhelm the system. Confidence in higher education depends strongly on assessment working well; we should be aware of that and, therefore, we should be willing to restate support for external examining as a vital element in maintaining a high quality system.

The highs and lows of examinations

September 3, 2008

My own English-language education experience was remarkably consistent for its entire duration. Both at school and at university, I received instruction through face-to-face contact with teachers, and at the end of the course I was tested in a written examination on my retained knowledge, with the exam typically determining how I was deemed to have performed in the subject. At school the exam result was balanced by what we would now call continuous assessment, but mostly the overall result depended on the examination. At university there was, at the time, no continuous assessment at all: the exam result was absolutely everything.

My German language experience was somewhat different. At my secondary school in Germany, I was tested at various intervals during the year through so-called ‘Arbeiten’, which were written assignments, some (but not all) performed under exam conditions. These were staggered through the year, so that I gradually built up my performance profile. The final grade on leaving school was determined by the exam called the Abitur, which had a written and an oral element.

Fast forward to 2000, the last year during which I undertook regular teaching duties at my then university, the University of Hull. My main module – the one I taught by myself – did not have an examination element at all, but was assessed entirely through project work carried out during the year. Other modules to which I contributed had mixed elements of examination and continuous assessment.

In fact, it has for a while been a topic of pedagogical debate as to whether exams are a good way of testing ability and achievement, or a bad way. Opinions are divided. Some believe that as they are conducted in conditions where things such as plagiarism and cheating can be controlled they are a more accurate reflection of a student’s performance; others believe that they encourage memory exercises only and discourage intellectual ingenuity or independent thinking. Others simply don’t know and hedge their bets (and support a mixed mode).

It is perhaps time that this issue was handled more systematically. It is of course likely that not all learning can be tested in the same way. First years students need to be assessed differently from final year students, and those doing a PhD need to be tested in a wholly different way. But we do need to have a clear understanding, on pedagogical grounds, of what is right in each case, and there should be more consistency, even in a system that allows for variety. And we need to come to an understanding of the potential and risks involved in online testing, with multiple choice or other formats.

There has been some research on this – see this book, for example – but in practice there is little sign that an integrated approach based on evidence and analysis is being applied. It is probably time for that now.