Secondary school statistics
Getting the most out of NQF statistics: A guide for users
Part 2 – Planning an Analysis |
|
This section provides guidance on
- the stages of planning, conducting and interpreting an analysis of NCEA/NQF data
- Some specific questions, analyses, and interpretive notes addressing the broad question 'How can educational processes in a department be improved?'
Some users, faced with the quantity of data available on the statistics pages of the NZQA website may feel overwhelmed. However, this is less likely to happen if one begins with an appropriate, well-framed question. The starting point for any meaningful data analysis and the important question to ask is ‘what do I want to know?’ Starting with this question will help to determine which data need to be used, and to organise the analyses that need to be carried out. Table 2.1 below outlines the main steps involved in planning, conducting and interpreting data analysis with additional explanatory text.
Table 2.1 below outlines the main steps involved in planning, conducting and interpreting data analysis with additional explanatory text for those who require it.
Table 2.1 Stages of planning, conducting and interpreting an analysis of NCEA/NQF data
Stages of data analysis |
Further explanation |
|---|---|
High-level question/s: This is the starting point of a good analysis. It should be a succinct summary of the reason for conducting the analysis (i.e., what is the main knowledge that will be gained?). |
It is often useful to begin with a broad question, perhaps one that does not directly refer to data or statistics. For example, a head of department might ask, ‘how can I improve the educational processes in my department’? While a question like this cannot be fully answered from assessment data (because there is much more to good educational outcomes than success in assessment), assessment data can provide the best available quantitative measure of educational success. |
Specific questions: The high-level question/s need to be translated into specific questions that can be directly answered by the data analysis. The specific questions then determine which analyses will be used |
Having posed a broad question, the next step is to break it down into parts that can be directly answered by analysing specific data. |
Analyses : At this stage the data are examined, organised and analysed in order to answer the specific question/s framed in the previous stage. |
The next step is to plan actual data comparisons based on the more refined questions. Examples of this are shown in Table 2.2 below. . |
Interpretation: How do the results of the analyses reflect upon the specific question/s? How do these in turn reflect upon the high-level questions? Are there any aspects of the analyses that make interpretation problematic? |
While statistics can be useful in determining patterns and trends in assessment performance within schools and departments, a statistical analysis in isolation is not very useful. An analysis needs to be interpreted in conjunction with the professional judgment of teachers, Heads of Department, Principals and Principals’ Nominees. While assessment data provide a quantitative measure of educational outcomes, there are many more subjective measures that need to be taken into account, such as the extent to which students are engaged in a subject by a particular teacher, and the fact that the first job of a teacher is to deliver a curriculum rather than to teach to assessment. |
Table 2.2 outlines specific questions that may be used for data analysis. Notes are provided for each analysis, including potential pitfalls and potential ways of using the results to improve educational processes.
Please note that Table 2.2 provides an illustration of an approach to data analysis that can be used for a topic of interest. It does not provide an exhaustive list of specific questions or analyses that could be conducted, or address all interpretative issues that might arise.
Table 2.2 Some specific questions, analyses, and interpretive notes addressing the broad question 'How can educational processes in a department be improved?'
Specific questions |
Analyses and data comparisons |
Notes |
|---|---|---|
Does my department have specific areas of strength and weakness in assessment outcomes? |
Compare success rates in individual achievement standards with aggregated figures from similar schools (e.g., decile band, single-sex boys/girls or co-educational). |
Standards that have either a positive or negative difference from comparison data, which are larger or in a different direction to those for most of the standards for the subject, then the material taught for those standards may indicate departmental areas of strength or weakness. It is necessary to take into account any differences in the average ability levels of students undertaking particular standards that may not be evident at other schools. |
Are assessment outcomes in my school / department improving over time, declining over time or remaining stable? |
Monitor success rates in a set of standards over successive years. |
Several years’ data are required to identify trends. Even quite substantial changes from one year to the next may not indicate a trend with any particular underlying cause. It is also necessary to take into account any revisions of the standards of interest, and any change in the way in which a standard is applied in assessment. |
Do some teachers (in the department) have successful teaching strategies which enable their classes to achieve consistently better assessment outcomes? |
Over a five-year period, compare results from each teacher to determine whether any teachers’ classes show consistently better assessment performance. |
If particularly effective teachers can be identified, it might be possible to share their successful teaching strategies with other teachers. It is necessary to monitor trends over time as a single year’s data is insufficient to form an accurate picture of an individual teacher. |
Are some teachers better with some groups of students, and others with other groups of students (e.g., male and female students)? |
Similar analysis can be conducted, breaking down the results by any student demographics of interest (e.g., gender). |
Any systematic differences between the abilities of students typically taught by particular teachers would clearly need to be taken into account (e.g., one teacher might be responsible for students who are struggling with a subject). |
| Index | | | Part 1 | | | Part 2 | | | Part 3 | | | Part 4 | | | Part 5 |
Page updated: 08 May 2009