ALL London branch
Last update: 27/05/2012
Click here to download the 4 page position statement + appendices
version as at 9/4/06/ including references to CEM information highlighted in TES article 31/3/06
The "severe grading" of MFL grades at GCSE and A level.
Who has put together this paper and why?
This paper has been put together by Helen Myers, with reference to members of the ALL National Executive Committee, and in response to members’ concerns expressed over a long period of time about the apparent severe grading of MFL external examinations. This has included more recently a significant number of expressions of concern from managers in Language Colleges (in response to a presentation of the data at the Annual Language Conference organised by SST), and from contributors to the ALL net, CILT hosted Linguanet forum and teacher-run mflresources forum.
At a time when professionals are judged by the performance of their pupils in external examinations, the committee believes that it is very important that managers who judge professionals are crystal clear about the nature of this data, as misunderstanding / false conclusions can lead to unfair criticism of professional standards in teaching and learning, and to wasted time and energy in the wrong area.
The paper was presented at the SST Language Conference and made available through Linguanet and ALL net. There was a significantly high level of positive response to the presentation.
Pupils from similar prior attainments should gain similar grades in subjects at GCSE and A level, but if this is not the case (and it is not) then there needs to be widespread awareness of the differences in order to avoid misleading judgements.
We use the phrase "severe grading" to put the emphasis onto the grading issue. Terminology is important. Words like "hard"and "standards" carry multiple connotations. Our principle is that pupils of comparable prior attainment should attain comparable grades across a range of subjects whether Business Studies (which will be only started at GCSE), Maths, Drama, etc all of which have very varying situations for different pupils.
This imbalance goes back a long time including for example the issue of comparability of grading of the old O level maths and O level English; readers of the Cockcroft report in 1982 will remember the stark statistic that 33% gained an O level English, 25% gained an O level maths. The relative gap between those subjects has been pretty consistently maintained over the last 20 years - perhaps as a consequence of the relentless media focus on 'standards' versus 'dumbing down'.
Peter Downes raised MFL issue when he was President of ALL.
When comparing the grades gained by pupils in different subjects, you can either do the comparisons on the basis of
i) PANDA Relative Performance Index (RPI) - for each pupil, compare the difference between the grade (score) in the particular subject, and the average grade (score) of that pupil in all their other subjects.
ii) Durham CEM Rasch model (publ. Mar 06) - This looks at the probabilities (odds) of a pupil who gets a certain grade in one subject getting a certain grade in another. The model then calculates "difficulties" for subjects and grades that best match all the combinations of pupils, subjects and grades. "The grade difficulty depends on the relative probabilities of that grade being achieved by candidates of different ability, as determined by their performance in all their subjects and taking into account the different difficulties of all the grades they have gained." These "difficulties" can then be converted back into "grade" units for comparison and dicussion. Note that the higher the number the more "difficult" the subject, which is the opposite way round to the prior attainment analysis (where comparable pupils will be gaining lower numbers in the more "difficult" subjects).
Data supporting the position statement
The data presented in Appendix 1from DfES data and Ofsted / PANDA data shows that:
By the principle which we have suggested above (that pupils of similar prior ability should expect to get similar grades in all subjects) you would expect the differences to be negligible. They are not.
PANDA: These figures show that French and German are both about half a grade BELOW the average of other subjects whereas Art, Drama, English and PE and about half a grade above, thus confirming that between Art/Drama/English/PE and French/German there is a difference of one full grade for pupils looking at their grades in different subjects
DfES value-added: these figures compare the average grade obtained by students with the same KS3 score in a range of different subjects. French and German are consistently amongst the lowest, especially in the mid-range of candidates generally gaining C and B grades
In practice, all three methods give similar outcomes. In particular, for MFL at GCSE, the CEM Rasch model gives a very similar outcome to the other two methods. For grade C, French, Spanish and German are half a grade more "difficult" than subjects such as Science (dual), Geography, History, Maths, and a grade more "difficult" than English, PE, Drama, DT and Art.
Decline in numbers: the data shows that over the last three years:
Issues of grading, grade boundaries etc become very complex during such a transitional period
This discrepancy in grades in likely to lead pupils to regard French and German as less successful subjects and therefore they are less likely to choose them at post 16.
The data presented in Appendix 2 from DfES data and Ofsted / PANDA data shows that:
at the end of 2 years of study, there is nearly half a grade ON AVERAGE between grading in MFL and the other subjects taken by those pupils, and nearly a whole grade between MFL and some other common subjects
Note that ...because by and large the most able are getting A grades in each of their subjects .... so the disparity will be much greater for the lower grades - Cs in one subject, E in French for example will be a typical result.
Note that French & German have high "severe grading factor" at A/L.
Candidates for French & German have the highest average grades in OTHER subjects
so will suffer in comparison with other subjects. French & German are only left with the most able.
Decline in numbers and consequent issues for grading : From 2004 to 2005 there has been a drop in numbers taking French and German.
The question is,
has the ability profile stayed the same in the 2004 and 2005 cohorts
have the more able continued
The German output profile tends to indicate that the less able have given up
The French output profile indicates that the ability profile has remained the same
In German, the numbers of those achieving A grades has stayed the same
In French, the percentage of those achieving A grades has remained the same.
When numbers in subjects are declining, this puts great pressure on the examiners and grade-awarding bodies because however sophisticated the statement-related criteria may be, inevitably the actual questions will vary in difficulty from year to year, so it becomes very difficult for examiners to make an accurate consistent judgement from one year to the next.
Two different approaches to this are highlighted above.
French appears to have adopted a policy of keeping the grade profile constant, whereas in German the absolute numbers of candidates getting A grades has remained constant and the profile has been adjusted, with more getting higher grades and fewer getting lower grades.
Outside the severe grading issue, this different approach can therefore cause great problems even when comparing results within a department, let alone for SMT and governors in individual schools who will be judging professionals by the results obtained by their students.
The research from CEM was reported on in the TES on 31 Mar 2006. Some extracts from the TES article and the CEM website follow:
In the context of MFL, it is vital that people know about the national pictures so that they do not leap to false conclusions which could severely affect the feeling of success and the desire to go on….whether for pupils or teachers!!
Simplistic, poor use of data will lead to depressing the sense of achievement .. or even actual achievement
Our ideal would be that "Language grading is fair, so...pupils choose MFL at KS4, feel good after their exams and get grades in line with other subjects so...more choose at AS and get grades in line with other subjects so...more continue to A2 and so..."