Performance measures

Tony Neal lays out the Law of Ineffectual Tampering - those attempts by government to improve flawed policies with minor amendments invariably making them worse.

For 14 years, since the introduction of school league tables, the percentage of 15 year olds attaining five A*-C GCSE grades has been embraced by governments as the critical measure of effectiveness of secondary schools.

This measure has become so ingrained in the national psyche that we have to remind ourselves of its obvious defects. Because it treats schools grossly unequally, it is not only unfit for purpose but quite futile as a comparative measure.

Yet it was not until the current generation of politicians suddenly realised that schools were finding ways of achieving this benchmark which did not appear to match up to their 'five O level' expectations (like vocational courses) that any need was seen to amend it.

Moreover the argument that any exclusive measure of success promotes a focus on children on the borderline at last began to hold sway with ministers committed to the twin mantras of 'personalised learning' and 'every child matters'.

So, two new measures have been introduced. Both of these measures come adorned, like the road to hell, with good intentions. And, while celebrating the demise of five A*-Cs, we need simultaneously to observe how the Law of Ineffectual Tampering affects the new measures.

Firstly, 'the percentage attaining five or more A*-C grades including English and maths' (that'll put the vocational courses in their place). On the face of it, this seems like a credible adaptation. After all, some aspects of English and maths can plausibly be described as essential life skills.

It wasn't until I talked recently with a head about her school's GCSE results that the complete inadequacy of this as a measure was brought home to me. Results overall were good, and attainment in both maths and English were high.

"You must have done well on the new measure", said I, innocently. "Well, no", said she. "You see, they're not the same students."

To illustrate, let us consider two hypothetical schools. In both, attainment is good and 70 per cent of students have five or more A*-C grades. In both, 65 per cent of students have an A*-C grade in maths and 65 per cent have an A*-C grade in English.

Creditable results, you might think. Yet on the new measure, one school scores 65 per cent and the other scores 0. How can this be? Diagram A will illustrate.

In School B, every one of its students has two of these three very desirable qualifications. In School A, 30 per cent have none of them. Yet on this measure School A is feted by Ofsted and School B is deemed fit for closure.

Now you may justifiably claim that neither of these extremes is likely in reality. But they illustrate a more general truth. For any two schools whose attainment is roughly the same, the school which sends the larger number of students away empty handed may well achieve a higher percentage.

At one brilliant stroke, we have rewarded those schools in which the results are most polarised.

Progression by levels

The second measure is now in pilot stage: 'the percentage of pupils who progress by two levels during a key stage'. Inherent in this measure are some obvious concerns about the nature of the levels themselves. It is difficult to perceive how a level 5 at Key Stage 2, a level 5 at Key Stage 3 and a Grade E at GCSE can possibly be regarded as showing the same level of attainment.

So the whole concept of 'progressing by two levels' has a dubious provenance. And any measure which depends on the reliability of results in Key Stage 3 English has a long uphill road to climb.

But, setting these anxieties aside, this new measure is still an 'exclusive' measure. A little more sophisticated, perhaps, but schools' success or failure this time is likely to be determined by those students on the borderline of a benchmark.

In diagram B, the arrows show the distribution of levels achieved nationally by pupils at Key Stage 2 in 2003 compared with their achievement three years later at Key Stage 3 in 2006. The thicker the arrows, the larger the percentage of pupils.

From this we can see with hindsight what each pupil might reasonably have been expected to achieve at Key Stage 3. Those who might reasonably have been expected to advance by two levels are shown in red on the diagram.

As the diagram shows, the top pupils could all have been expected to advance by two levels. (In English the best 10 per cent and in maths, quite inexplicably, the best 60 per cent.) So, those schools which succeed in attracting the most able pupils are favoured.

But in each subject, we have two other randomly placed bands of pupils unexpectedly well armed for this quest. Pupils just failing to achieve level 5 at Key Stage 2 in English (a group somewhere between the 30th and 40th percentile) are very likely to achieve a level 6 at Key Stage 3.

In maths, those who just missed a level 4 at Key Stage 2 (a group around the 75th percentile) are very likely to attain a level 5 at Key Stage 3. Others who have progressed equally well have moved by just one level - and some in English by no levels at all.

Inspired by the new measure, schools will no doubt be seeking to recruit pupils in the favoured groups and focusing resources on their continued well-being. But, quite clearly, the use of this measure as a comparator between schools is not only unfair, it is capriciously arbitrary.

So schools are left in their habitual position of trying to derive some benefit from ill-considered legislation. In setting school targets for current Year 8s and Year 10s we are forced by these changes to do what most of us do already - to work with each pupil to set properly aspirational and motivational targets for what s/he might achieve in two years.

But how we can validly translate these into motivational school targets remains a mystery. And the notion that in two years time these measures will tell us anything useful about the comparative performance of schools is doomed by the Law of Ineffectual Tampering.

Tony Neal is a former headteacher and ASCL's representative on the GTC(E).