Dr Brendan Guilfoyle, Institute of Technology, Tralee
Almost two years ago a post on this blog led to the following question: what do we mean by educational standards? While matters have moved on somewhat since that initial debate on grade inflation, it is still worthwhile to attempt to answer this question.
In a recent paper ’New Metrics for Detecting Changes in Educational Standards‘, I have considered it from a theoretical point of view and applied the conclusions to updated data from the Irish educational system at second and third level. This is paper 9 of a series published by the Network for Irish Educational Standards investigating this issue in a broader context.
I hope the methods and conclusions are of interest to the academic community and I would like to thank Professor von Prondzynski for affording me the opportunity of this guest post to summarize the results.
The theoretical framework adopted has three fundamental assumptions. These are that:
- we are dealing with a mass education system,
- the results of assessment provides sufficient grade differentiation,
- assessment measures student performance against some hierarchical taxonomy of activity.
The first assumption is to allow for robust statistical analysis, the second means we do not consider pass/fail systems, and the final assumption means that assessment seeks to measure student performance against some framework of activity defining the educational standard. It can be as abstract as Bloom’s famous taxonomy of cognitive activity or it can be more specifically articulated to a particular set of learning objectives.
For our purposes the details of the taxonomy are not important. The key feature is that it is hierarchical: the scale of activity has an ordering that is cumulative. Thus a lower activity must be mastered in order to advance to the next level of activity. Conversely, those who have mastered the higher levels find it easier to perform lower activities. Only a belief in a radical dissociativity of cognitive activity would lead one to reject this assumption for second or third level education, and such a Fordist belief does not appear to be widespread among contemporary educationalists.
Consider then an educational system satisfying our three assumptions: that is, an educational system which produces a grade for a large number of students which measures their performance against a hierarchical taxonomy. This grade could be arrived at in any number of ways and could be, for example, an aggregation of a number of measurements. The resulting grade distribution reflects the attainment of the student cohort against the standard.
From a formal point of view, assessment is then a mapping from an ordered set of abstract attributes to the grade distribution of the student population. Such a grade mapping is determined by numerous interlinked factors, including the nature of the material being assessed, the mode of assessment, the selection of students, as well as historical and institutional factors.
We define declining standards to be a change over time in a given educational system where the grade mapping gives higher grades to those at fixed levels in the taxonomy. Equivalently, decline means that a lower level in the taxonomy is required to attain a fixed grade. In such a situation, the mean of the grade distribution would naturally increase. Reversing the argument does not work directly, however, since grade increase on its own does not necessarily imply declining standards. It could result from a variety of factors, for example better teaching, higher selectivity of students etc.
Indeed, it is precisely this issue that is the main point of contention, if not controversy, in the debate about declining educational standards. When is grade increase a symptom of grade inflation (i.e. declining standards) and when is it a sign of higher student attainment? In the absence of other comparative metrics of performance, how can the former be distinguished from the latter?
In the paper I come to the conclusion that if increasing mean grade is accompanied by decreasing skewness, then we have strong evidence of declining standards. In particular, I consider second and third order effects in the grade distribution and what one expects to see during times of declining standards in mass education systems. It is argued that the hierarchical nature of the taxonomy implies that, during times of declining standards, in general those students operating at a higher level in the taxonomy benefit more. That is, one expects to see a non-linear effect in which the grade distribution, aside from having an increasing mean, becomes more negatively skewed. We should witness a depletion of lower grades and an increase in higher grades as grades migrate across the increasing mean.
Furthermore, an advanced decline in standards leads to a second order effect in the form of decreasing standard deviation. This is an artifact of the ceiling effect whereby the top grades cannot increase any further. Such a decrease in standard deviation undermines the whole ethos of assessment as a measure of achievement in educational settings.
Let us for a moment, consider a simple example that illustrates these concepts. If a third level lecturer gives a hint that a particular topic will appear on the end-of-term examination, this information will benefit a student only to the degree to which they are able to take advantage of it. That is, the best students will pick up on it immediately and make a note, the average student may know that a hint has been given but be unclear as to exactly what it refers to, while the weak student, if they are even present, will have little awareness of what has transpired. Thus the better students are more advantaged than their weaker colleagues and, aside from increasing the mean grade in the class, negative skewness will be introduced into the examination grades.
Should the errant lecturer go so far as to show the students the test beforehand, not only will the mean jump, but most of the students will be squeezed into the top grades and become well-nigh indistinguishable. Except, of course, for the poor student who didn’t turn up that day. In any event, the standard deviation will have decreased.
We then turn to grades for the Leaving Certificate Examination from 1992 to 2009 and undergraduate university awards from 1998 to 2008. From the grade data at both levels one finds that the grade distributions display the characteristic pattern of declining standards: increasing grade mean and decreasing grade skewness.
This is a feature of almost all of the most popular subjects of the Leaving Certificate Examination over the period. Interestingly, mathematics has managed to maintain its standard relative to other subjects by these metrics. Perhaps this is a missing argument about reform at second level: the problem is not that mathematics is too hard, but rather that other subjects have become too easy!
The standard deviation of Leaving Certificate awards has remained more or less stable. Thus while it has certainly slipped down the taxonomy, the examination is still able to distinguish between students in a relative sense.
For Honours Bachelor degrees in all seven universities the mean has increased and the skewness has decreased, again the fingerprint of falling standards. In fact, the universities have seen a dramatic period of inflation (namely 1998-2005) at which point in time the mean has stabilized. Could this be the ceiling effect?
Perhaps more worryingly, university grades are found to have decreasing standard deviation – a hallmark of advanced decline in educational standards whereby assessment fails to distinguish between students. Thus, Irish universities appear to be slouching towards a pass/fail system with relative merit residing with institutional reputation rather than award level.
It is hoped that these findings will move the discussion of educational standards forward. We have presented a theoretical model of educational settings which has predictive qualities, as well as tools for a detailed analysis of grade trends against which to test predictions. These tests track changes over time for a given institution and therefore are not subject to the usual difficulties of comparing across institutions. The available data can be analysed from a number of other interesting perspectives within this framework.
For the Irish education system as a whole, at second and third level in particular, the message is clear: there has been a decline in educational standards over the past 15 years. Those who argue that this is not so – that increasing award levels are attributable to student motivation, improvements in teaching or whatever – must now present their case with both a coherent theoretical framework and empirical data to back up their claims. Otherwise, the debate must move on to how best to halt the decline.