Analysis of the Predictive Power of PISA Test Items

Abstract

The predictive power of the PISA test items for future student success is examined based on data from the Longitudinal Surveys of Australian Youth (LSAY) for the PISA 2003 cohort. This working paper analyses how students’ responses to mathematics and problem-solving items in PISA 2003 are related to the students’ qualifications in education in 2007 and 2010. The results show that items do differ in their predictive power, depending on some of their deep qualities. PISA mathematics and problem-solving items are grouped into various classifications according to their qualities. This paper proposes 16 new classifications of items. Among mathematics-specific item classifications, two are found to be significantly related to future student success: those that assess knowledge, understanding, and application of statistics; and those related to rates, ratios, proportions, and/or percent. These items frequently require students to apply common mathematical concepts to solve multi-step, non-routine problems, think flexibly, and understand and interpret information presented in an unfamiliar format or context. Among classifications that are not specific to mathematics, items that were classified as using reverse or flexible thinking are found to be related to student qualifications in both mathematics and problem solving. These items require students to be able to think through a solution at various points during the problem-solving process, not just at the start.