ALL London branch   

 

Last update: 27/05/2012

Home
Up
Contact

Introduction
Site Map
Diary
Events-News
People
Subcommittees
Web Links
ICT
FAQ
National ALL
Webinars

Statement

Click here to download the 4 page position statement + appendices 

to severe grading home page

version as at 9/4/06/ including references to CEM information highlighted in TES article 31/3/06

 

The "severe grading" of MFL grades at GCSE and A level.

Background

Who has put together this paper and why?

This paper has been put together by Helen Myers, with reference to members of the ALL National Executive Committee, and in response to members’ concerns expressed over a long period of time about the apparent severe grading of MFL external examinations. This has included more recently a significant number of expressions of concern from managers in Language Colleges (in response to a presentation of the data at the Annual Language Conference organised by SST), and from contributors to the ALL net, CILT hosted Linguanet forum and teacher-run mflresources forum.

At a time when professionals are judged by the performance of their pupils in external examinations, the committee believes that it is very important that managers who judge professionals are crystal clear about the nature of this data, as misunderstanding / false conclusions can lead to unfair criticism of professional standards in teaching and learning, and to wasted time and energy in the wrong area.

The paper was presented at the SST Language Conference and made available through Linguanet and ALL net. There was a significantly high level of positive response to the presentation.

Principle

Pupils from similar prior attainments should gain similar grades in subjects at GCSE and A level, but if this is not the case (and it is not) then there needs to be widespread awareness of the differences in order to avoid misleading judgements.

We use the phrase "severe grading" to put the emphasis onto the grading issue. Terminology is important. Words like "hard"and "standards" carry multiple connotations. Our principle is that pupils of comparable prior attainment should attain comparable grades across a range of subjects whether Business Studies (which will be only started at GCSE), Maths, Drama, etc all of which have very varying situations for different pupils.

Historical context

This imbalance goes back a long time including for example the issue of comparability of grading of the old O level maths and O level English; readers of the Cockcroft report in 1982 will remember the stark statistic that 33% gained an O level English, 25% gained an O level maths. The relative gap between those subjects has been pretty consistently maintained over the last 20 years - perhaps as a consequence of the relentless media focus on 'standards' versus 'dumbing down'.

Peter Downes raised MFL issue when he was President of ALL.

Methodology

When comparing the grades gained by pupils in different subjects, you can either do the comparisons on the basis of

  1. prior attainment - i.e look at pupils with the same results at an earlier stage (typically Key Stage 3 test results) and then compare the grades gained by those pupils in the different subjects (at GCSE). This is the basis of value-added analysis e.g. as used by DfES
  2. concurrent attainment - i.e. look at the grades gained by individual pupils in the different subjects, and then compare the accumulation of outcomes. There are two ways of doing this:
    i) PANDA Relative Performance Index (RPI) - for each pupil, compare the difference between the grade (score) in the particular subject, and the average grade (score) of that pupil in all their other subjects.
    ii) Durham CEM Rasch model (publ. Mar 06) - This looks at the probabilities (odds) of a pupil who gets a certain grade in one subject getting a certain grade in another. The model then calculates "difficulties" for subjects and grades that best match all the combinations of pupils, subjects and grades. "The grade difficulty depends on the relative probabilities of that grade being achieved by candidates of different ability, as determined by their performance in all their subjects and taking into account the different difficulties of all the grades they have gained." These "difficulties" can then be converted back into "grade" units for comparison and dicussion. Note that the higher the number the more "difficult" the subject, which is the opposite way round to the prior attainment analysis (where comparable pupils will be gaining lower numbers in the more "difficult" subjects).

Data supporting the position statement

  1. GCSE

The data presented in Appendix 1from DfES data and Ofsted / PANDA data shows that:

By the principle which we have suggested above (that pupils of similar prior ability should expect to get similar grades in all subjects) you would expect the differences to be negligible. They are not.

PANDA: These figures show that French and German are both about half a grade BELOW the average of other subjects whereas Art, Drama, English and PE and about half a grade above, thus confirming that between Art/Drama/English/PE and French/German there is a difference of one full grade for pupils looking at their grades in different subjects

DfES value-added: these figures compare the average grade obtained by students with the same KS3 score in a range of different subjects. French and German are consistently amongst the lowest, especially in the mid-range of candidates generally gaining C and B grades

Outcomes

In practice, all three methods give similar outcomes. In particular, for MFL at GCSE, the CEM Rasch model gives a very similar outcome to the other two methods. For grade C, French, Spanish and German are half a grade more "difficult" than subjects such as Science (dual), Geography, History, Maths, and a grade more "difficult" than English, PE, Drama, DT and Art.

Decline in numbers: the data shows that over the last three years:

the overall number taking any modern language has declined significantly
the overall numbers getting A*-C grades has remained constant
the percentage of candidates getting A*-C grades has increased significantly

Issues of grading, grade boundaries etc become very complex during such a transitional period

 

% of ALL 15 year old pupils taking any modern language

% of ALL 15 year old pupils gaining grade A*-C in any modern language

% of CANDIDATES in any modern language gaining grade A*-C

2003

73%

36%

49%

2004

68%

36%

53%

2005

59%

35%

60%

This discrepancy in grades in likely to lead pupils to regard French and German as less successful subjects and therefore they are less likely to choose them at post 16.

  1. A level (including AS)

The data presented in Appendix 2 from DfES data and Ofsted / PANDA data shows that:

at the end of 2 years of study, there is nearly half a grade ON AVERAGE between grading in MFL and the other subjects taken by those pupils, and nearly a whole grade between MFL and some other common subjects

Note that ...because by and large the most able are getting A grades in each of their subjects .... so the disparity will be much greater for the lower grades - Cs in one subject, E in French for example will be a typical result.

Note that French & German have high "severe grading factor" at A/L.

Candidates for French & German have the highest average grades in OTHER subjects

so will suffer in comparison with other subjects. French & German are only left with the most able.

Decline in numbers and consequent issues for grading : From 2004 to 2005 there has been a drop in numbers taking French and German.

The question is,

has the ability profile stayed the same in the 2004 and 2005 cohorts

OR

have the more able continued

The German output profile tends to indicate that the less able have given up

The French output profile indicates that the ability profile has remained the same

In German, the numbers of those achieving A grades has stayed the same

In French, the percentage of those achieving A grades has remained the same.

When numbers in subjects are declining, this puts great pressure on the examiners and grade-awarding bodies because however sophisticated the statement-related criteria may be, inevitably the actual questions will vary in difficulty from year to year, so it becomes very difficult for examiners to make an accurate consistent judgement from one year to the next.

Two different approaches to this are highlighted above.

French appears to have adopted a policy of keeping the grade profile constant, whereas in German the absolute numbers of candidates getting A grades has remained constant and the profile has been adjusted, with more getting higher grades and fewer getting lower grades.

Outside the severe grading issue, this different approach can therefore cause great problems even when comparing results within a department, let alone for SMT and governors in individual schools who will be judging professionals by the results obtained by their students.

Commentary

The research from CEM was reported on in the TES on 31 Mar 2006. Some extracts from the TES article and the CEM website follow:

Dr Coe analysed a national dataset of over 600,000 students to compare the grades achieved in each subject with the grades achieved by the same student in their other subjects. Some subjects do come out as apparently 'harder' than others, though it makes a difference which grade you look at. (CEM)
The figures will alarm the exam regulator, however. It has argued that all GCSEs are equally difficult to do well in. (TES)
They concluded that, students, whatever their ability, would be more likely to achieve good grades in subjects such as drama and PE, than in science and languages. (TES)
Robert Coe, director of secondary projects at Durham university’s Curriculum, Evaluation and Management Centre, who led the research, said: "It’s hard to deny that there are differences in the difficulty levels of particular subjects." (TES)
Dr Coe said it was unlikely that this [better teaching, motivation, time allocation] explained all of the differences. The analysis is powerful in that it offers more information than can be gleaned by analysing "raw" results in GCSE subjects each year. (TES)

Summary

In the context of MFL, it is vital that people know about the national pictures so that they do not leap to false conclusions which could severely affect the feeling of success and the desire to go on….whether for pupils or teachers!!

Simplistic, poor use of data will lead to depressing the sense of achievement .. or even actual achievement

Our ideal would be that "Language grading is fair, so...pupils choose MFL at KS4, feel good after their exams and get grades in line with other subjects so...more choose at AS and get grades in line with other subjects so...more continue to A2 and so..."

Recommendations

  1. The issue of "severe grading" in French and German at both GCSE and A-level is recognised as fact, and is not confused or obfuscated by issues such as "standards", etc.
  2. Publicity is given to this so that school managers and others automatically accept that there is an issue when judging the results of teachers of French and German, and make due allowance for "severe grading"
  3. At a time of substantial decline in numbers taking French and German at GCSE and A-level, that grading issues are examined carefully and a more level playing field is established, and certainly that a steeper one is not created!
 

to severe grading home page