NCEA Moderation 2007 and 2008
New Zealand secondary schools deliver the New Zealand Curriculum and assess it using a mixture of internally and externally assessed standards. These standards contribute to students' achievement of National Certificates of Educational Achievement (NCEAs) at levels 1, 2 and 3.
During 2007 a programme of NCEA enhancements was instigated. As part of this programme (starting from 2008) the number of internally assessed standards moderated was substantially increased (see Note 1) with a goal of 10% being set. To reach this goal, and to achieve greater consistency of moderation, a number of full time moderators were employed (see Note 2).
Moderation, in general, serves two main purposes; firstly it ensures consistency and accuracy in the marking of student work. Secondly, through the use of post-assessment feedback, moderation provides professional guidance to teachers on the suitability of assessment materials that they have used. NZQA aims, through secondary school moderation, to both gauge the reliability of assessment taking place while also providing guidance on how to improve assessment materials and make assessor judgements at the national standard.
Random sampling is used to reduce any bias that could arise if samples were selected in a subjective manner. While the selection of internally assessed standards for moderation by NZQA is not random, from July 2008 the samples of student work selected for moderation were randomly selected. Due to the lag that occurs with moderation many of these randomly selected standards from 2008 will not be moderated until 2009. Lag occurs in a number of areas in the moderation process. The first lag is the time between assessment of a standard and its moderation which can be as much as 12 months. The second lag occurs between the teacher receiving feedback for a standard and the resulting change in their teaching practice and subsequent moderation occurring.
The results of moderation show the level of agreement between the moderator and the teacher on what the result for a particular piece of student work should be as well as feedback on the suitability of the assessment materials.
Overall moderator-teacher agreement rates in both 2007 and 2008 were around 70 percent. This agreement rate is similar to the rates found previous to 2007.
2008 was the first year of an increase in the amount of moderation undertaken and of the employment of full-time moderators. NZQA did not expect to see immediate changes to moderation results given the time lags inherent in the process.
It should also be noted that the above results are largely based on teacher selected work at grade boundaries (see Note 3). Results from the next rounds will be from randomly selected standards and for 2010 will not include bias for grade boundary samples of work. Early indications of 2009 moderation rates are looking promising and it is likely that upcoming moderation initiatives (see below) will contribute to an improvement of agreement rates, over time.
For assessment materials moderators record whether the task is approved as being suitable for measuring student achievement against the standard, whether it requires minor modification or whether it is not suitable to measure the particular standard. They also record where tasks are not supplied for the standard being moderated. It is possible to obtain a valid result for a student even where the assessment materials are judged to not be at the national standard as students can exceed the requirements of the task and reach the required standard.
In 2007 the percentage of assessment tasks either approved by moderators , or requiring only minor modification, was around 80 percent. In 2008, as well as an increase in tasks judged valid there was a significant decease in the proportion of standards for which no assessment materials were submitted.
Upcoming moderation initiatives
Beginning late in 2008, and continuing into 2009, NZQA has been running moderation best-practice workshops . In these workshops student work is moderated by teachers using a cluster moderation approach. These workshops have proved very popular and are expected to lead to improvement in teacher marking practice. Having moderators able to facilitate and lead the workshops and to directly engage in developing a shared understanding of the standards strongly benefits teachers and enables this understanding to be passed on to students.
Starting later in 2009 NZQA will make available to commercial enterprises, as well as the Ministry of Education, the option to quality assure assessment materials before they are used in schools. This will mean teachers can be more confident when making assessment judgements of student work by using valid, quality assured assessment materials. This will also allow moderators to concentrate more of their resources on giving detailed feedback on teacher judgments. As well, NZQA is making changes to the selection process used to determine what standards are to be moderated each year. These are not currently random and it will improve comparison across years when a more consistent sample is moderated and reported on.
Over 2008 a significant number of assessment activities available to teachers through the Ministry of Education website Te Kete Ipurangi (TKI) have also been up-dated by the moderators so that they at the current national standard.
From 2011, revised Level 1 standards will be in place with supporting assessment material and exemplars. Level 2 and 3 standards will follow in 2012 and 2013 respectively. These revised standards with supporting assessment tasks and annotated exemplars (see Note 4) should also have a positive impact on teacher marking practice.
Continuing initiatives include:
- increased detail in moderation reports
- subject specific web pages for teachers
- moderator newsletters and clarifications documents (see Note 5) for standards
- teachers sending additional student work to moderators to respond to specific questions
- more moderation best practice workshops
- moderators liaising with subject advisers and presenting at national conferences
- extending the length of the moderation cycle
- annotated exemplar development and provision
- ongoing review of internal moderation processes taking place in schools
Random selection trial
Random sampling is used to reduce any bias that could arise if samples were selected in a subjective manner.
The random selection trial undertaken in September 2007 was run alongside the usual collection of student work selected by teachers. The intention of this trial was to provide a quality check on the system of moderation and to pilot a method for obtaining random samples from schools in a manageable and effective manner.
The trial looked at Level 3 standards only. A representative sample (approximately one sixth) of schools was asked to participate. Instructions as to how many pieces of student work and how to undertake the random selection process were supplied. The method used meant that the total number of pieces of student work should have been the same (at the least) or more than the number collected through the usual process.
The trial returned substantially fewer pieces of student work than expected. It is possible that teachers misunderstood the instructions, did not indicate when samples overlapped or just did not comply with the instructions.
As a result, the data collected was not suitable to provide useful quality assurance checks on the national system of moderation as a whole.
The trial clearly showed a number of areas needed further work before all student work could be randomly selected. These areas included the instructions supplied to teachers about the quantity of work to be selected and the random selection process itself. Both areas were further refined and random selection of student work for moderation of internally assessed standards in secondary schools was introduced in July 2008.
Note 1: Previously about 4 percent of standards per annum were moderated.
Note 2: Previously some 380 contract-for-service moderators were employed. 31 full-time, 4 half-time, and 165 contract-for-service moderators were employed in 2008.
Note 3: Grade boundaries are at the top or lower end of a grade range.
Note 4: Exemplars are samples of marked student work showing results. Usually these are provided to illustrate differences at the grade boundaries to assist teachers with their marking decisions.
Note 5: Documents produced for specific standards providing extra support through exemplars of student work.
Page updated: 16 June 2009