My colleague the computer
It’s that time of year when academics all over the place get ready for another avalanche of marking and assessment. In my own case, while I really do miss teaching very much and am looking at ways of returning to it, I don’t miss marking. Not even slightly. And I feel for those who will, over the next couple of months, be inundated with it.
But is there another way? In fact, could we just give the job to computers? And might we find that they can grade essays and assignments and examinations just as effectively as we can? Well perhaps, according to a study conducted by researchers at the University of Akron. They compared grades given to 22,000 short essays written in American schools by live examiners with those recorded by computers running ‘automated essay scoring software’. The differences were, according to the researchers, ‘minute’.
I don’t know what kind of software this is, or how it works, or what its stated limitations might be, but this is a pretty amazing result. We know that computers can easily grade multiple choice examinations, but essays? And can we really imagine that an assignment intended to produce reasoned analysis could be assessed by machine? More generally, how much work has been done in considering the role that computers can play in designing, conducting and assessing teaching?
In fact, this is a subject of some interest in the education world. In July of this year there will be a conference in Southampton in England on computer-assisted assessment, and indeed there is a journal on the subject.
There are probably various contexts in which higher education assessment can be conducted by or with the help of software. But equally there are others where, at least from my perspective, it is unlikely that computers will be able to make robust qualitative judgements that could replicate human marking. Somehow I doubt that, in a few years, lecturers will no longer have to be examiners.
Explore posts in the same categories: education, higher educationTags: assessment, computer-assisted assessment, examinations
You can comment below, or link to this permanent URL from your own site.
University Blog 
May 1, 2012 at 1:57 am
It may be some time before the automatic grading of essays becomes a reality, but issues such as marker reliability ensure that it will always be a popular topic. I just wrote about the William and Flora Hewlett Foundation in the US running a competition, with prizes totalling $100K for designers of software that could reliably automate the grading of essays for state tests in the latest e-Assessment Association’s newsletter (apologies for the shameless plug – but the newsletter, as well as membership to the Association are both free):
http://www.e-assessment.com/resources/eAA_newsletters
Something else that is free is the UK’s largest conference dedicated to exploring the benefits of eAssessment. Now in it’s fourth year, eAssessment Scotland 2012 is being held at the University of Dundee on the 31st of August (with a parallel online event running from the 23rd to the 6th). Last year, the conference attracted around 300 delegates and featured a series of presentations, workshops and posters from around the globe.
Note: Although it’s called ‘eAssessment Scotland’, it’s not limited to those from Scotland! Everyone is welcome!
We’ll also be publishing a special edition of the International Journal of e-Assessment (which you mentioned), featuring papers from the conference:
http://journals.sfu.ca/ijea/
The conference also plays host to the Scottish e-Assessment Awards (also in its fourth year), which recognises the achievements of those working with and developing the technology that extends the practice of assessment. You can submit your entries now, with the winners announced at the conference (I’ll be waiting for the Robert Gordon’s entry with interest):
http://www.e-assessment-scotland.org/awards/
There’s also the chance to get involved – the call for papers, presentations and posters is running until the end of the week. You don’t need to submit a lengthy paper to win a place on our programme – just provide us wth a short description of what you’re doing and we’ll take it from there. We also encourage submissions from every sector, there are no restrictions on who can come along or present. If you an e-Assessment story to tell, then we want to hear from you.
Did I mention it was free to attend? Registration opens later this month, when we’ll be announcing the first stage of the programme – places go fast though, so make sure to sign up early!
Find out more at the conference website: http://www.e-assessment-scotland.org
May 1, 2012 at 5:34 am
How on earth will a computer assess an essay when the student goes outside the walls of the course. I remember a tutor saying to me that he wasn’t going to mark my one since I’d added a 17 century map to the flyleaf. It seemed logical to me that the books I’d read and the lectures were liberally sprinkled with explanatory maps that whacking in one of my own to aid a complex concept was reasonable. Anyway he passed it up the line.
And anyway, why on earth is there all this fuss, aren’t you getting them marked by PhD candidates on the cheap. Exactly how much cheaper do you want it.
You could always close the places to undergrads. 😐
May 1, 2012 at 8:53 am
Ferdinand, you betray the teaching profession. To follow your suggestion of increased computerisation, why not get students to sit in front of computers to be taught in the first place – there! No need now for teachers. Everything to be done by computers. More de-personalisation of education and less human interaction – for what? To give more time to important things like attending conferences, colloquia, writing books – more time for self-actualisation? Marking exam scripts, aka examining students, is as important a tool to educating students as is lecturing – don’t devalue it. Of course, multiple choice examinations are already widely assessed by computer programmes, and anti-pliagiarism software is widely and correctly used. I’m sure you don’t mean to be, but your stated relief at not marking scripts, which I concede isn’t a belly of laughs most of the time, exposes a degree of self-centredness leading me to believe that you’re not a real educator at heart. Such persons care for their students, they care for the knowledge they’ve imparted; they care for the understanding of that knowldge by the student. We’re talking about adult formation here. We all see some of the negative attributes of computers in society – please don’t advocate their use in the precious issue of shaping our next generation – they’re too valuable to sacrifice on the altar of an ideology.
May 1, 2012 at 8:56 am
Munchkinman, I’m not sure you read what I wrote! My point was exactly the same as yours: you cannot replace personal marking in assessing qualitative performance.
May 1, 2012 at 7:33 pm
I like marking, find it a useful feedback excercise!
However, I’m quite naughty because I mark in series (entire script at a time) instead of in parallel (all question 5s at a time). I find that marking the entire script in one go gives me a feel for how the semester has been for a student, and lessens the chance of error in totting up the marks.
This year, I taught a course where maths assessments were marked by computer; I still had to go over all the answers in order to pick out all the nuances the computer missed.
May 6, 2012 at 9:44 am
A useful feedback exercise if the students receiving the feedback actually read it and don’t just stop at the grade being awarded.
Marking in series as you describe seems reasonable in one sense – as you get a feel for the overall performance. However, marking on extended responses is notoriously unreliable (in assessment terms) and reading through the entire script can have a subjective bias on the grading in one section.
And in terms of the maths assessment – it could be the quality of software in that particular instance, but maths is quite advanced in terms of e-Assessment, and automated marking from systems like Numbas, are pretty impressive (though only generally as good as the people who set them up).
(another shameless plug – but for a free event)
I’d recommend going along to eAssessment Scotland 2012 in Dundee on the 31st of August and I’ll introduce to several reliable systems!
May 1, 2012 at 7:37 pm
P.S. I also think it is BONKERS that IoT lecturers get an allowance for correcting exams. If someone told you at interview that thiis was part of the job, you would accept it as fair..
May 2, 2012 at 8:33 am
The practice of awarding allowances to mark exam scripts is also afforded to some grades of academic staff in my university…
May 3, 2012 at 5:21 pm
Back to the future…automation at first level education appears to be well underway according to this report from South Korea with plans to have a teaching robot in every kindergarten by 2013. See: http://bit.ly/h0WGqr
Apparently the robot’s ‘dumpy’ appearance has all kinds of intentions behind it. So I’m sure it’s just a matter of time before it catches on in higher education…and soon we’ll be defining collegiality with an entirely new set of vocabulary.
May 4, 2012 at 7:26 am
Come to think of it, that lecturer I had in first year physics some years ago now did look rather odd…
May 6, 2012 at 9:35 am
Although the conversation has moved on, I think the comment is a little misleading. The ‘robot’ you refer to is principally a telepresence device with a ‘real’ teacher located overseas delivering the lesson; a solution for teaching English to young Korean students – i.e. it’s cheaper than bringing teachers into the country. So the ‘robot’ can be thought of as a very sophisticated webcam/mic.
Also the ‘2013’ statement isn’t quite right either – the project is still in its trial phase (in 30 kindergartens/schools), with the headline originating from a statement in 2010 made by one of the robot’s developers, who thought the technology would advance in 3-5 years to the point when native English teachers could be replaced. Obviously media outlets pounced on that and went with ‘robot in every kindergarten by 2013’.
Can’t help but thinking what’s needed here is a ‘robot’ to check the veracity of the comments.
(Just to be clear – that last comment was a joke.)
😉